Understanding The HTTP Plugin

I just found two great resources on the WebSphere plugin

• Demystifying the IBM® Web Server Plug-in 
• by Naveen Shetty AND Steve Reid
• Understanding the WebSphere Application Server Web server plug-in
• by Sharad Cocasse And Makarand Kulkarni

Apache The Definitive Guide - Book Review

I just finished Apache The Definitive Guide by Ben Laurie & Peter Laurie and would highly recommend this book to anybody who is administrating an Apache or IBM HTTP Server. Apache The Definitive Guide is a ground up approach to teaching you about the Apache web server platform. After spending the last year moving away from a developer role and into a infrastructure role I learned a lot about the IBM HTTP Server  but still had a lot of questions. The Definitive guide tied a lot of concepts together and removed the once dense fog around the web server component of our infrastructure.

Growing access.logs for IBM HTTP Server

While working on some performance issues in Prd we discovered that our web server access.logs were in excess of 2GB and still growing strong. After some research we decided it was time to implement piped logs . We first tried to edit the httpd.conf file for the web server through the WebSphere admin console. We replaced the following 

CustomLog logs/access.log common

with

CustomLog "|bin/rotatelogs logs/access.%Y.%m.%d 86400" common 

Stopped the web server , tried to start it and after a 30 sec pause we got a msg back saying that the web server could not be started. Further research into the log files for the deployment manager and web server did not shed any light on why. After some digging on IBM's support site I found this technote explaining that the full path must be specified. I replaced the above with this, (adjusting the formatting to better fit our needs) and the server started without issue and our log files were now rolling on a 24hr bases.

CustomLog "|C:/IBM/HTTPServer/bin/rotatelogs.exe -l C:/IBM/HTTPServer/logs/access-%a-%m-%d-%Y.log 86400" common 

The Pain of Windows and WebSphere 6.1

This is a repost of a blog entry I did on the Webshereusergroup.org site. Since this blog is about me getting orgnized and keeping track of things I run into I am reposting it here.

Last Friday we started to experience slowness in the response from our production web servers. The CPU and memory on each server was not above normal. The application servers were responding as expected it was only traffic directed to or through the web servers that slowed. At first the report seemed to be only a nuisance and nothing more, but as time went on the web servers went from a 30 second response time to four or five minutes
In the web servers logs I found the following message sometimes several times a second

[Fri Apr 06 07:54:07 2012] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

The frequency of the messages had increased and on each server we were getting as many as 6 or 7 a second. The slowness of the servers got progressively worse until our entire collection of web servers had crashed. After the crash of the first web server I tried to restart it using the console. The web server would not start. After the other two web servers crashed production was no longer available so  we restarted the OS on the first web server that crashed, it was already broken what more could happen. After the reboot the web server started back up normally and the error messages were no longer being generated.

We then logged a PMR with IBM for more information about this issue. Within an hour I received a response that indicated that this is a known issue in a Windows environment where "other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions" http://publib.boulder.ibm.com/httpserv/ihsdiag/errorlog.html#LSP                                                                   

We had read online that other vendors software could include “anti-virus, firewall, virtualization, or vpn” http://rob.brooks-bilson.com/index.cfm/2008/1/4/Intermittent-Apache-Problems-and-winntaccept-Asynchronous-AcceptEx-failed  Post outage we returned to each server and verified in the add remove software that not updates or new software had been installed in the last day. Anti Virus updates had run but several hours before the first record of the error in the log file. IBM did let us know that a fix does exists for this error however the version of Apache that WebSphere 6.1 runs does not support the fix.

WebSphere Proxy Servers

Cool older education assistant video on the use of proxy servers from IBM. Some interesting points were

• They replace web servers
• Are the direction for performance
• Can be configured across cells
• Require a unique profile

 http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.was_v6/was/6.0/SystemManagement/WASv6_WebServer/player.html

IBM HTTP Server Documentation

I was watching the IBM WebSphere Application Server V6 Web Server Definition and Plug-in Education Assistant video and ran across this link that provides the download info and support documentation for the last 5 versions of IBM HTTP Server.

http://www-01.ibm.com/software/webservers/httpservers/library/v70.html