When in Dought Encrypt It !

Recently I ran into trouble getting two Maximo environments up one 7.1 and the other 7.5.  We got several different errors and none pointed us in a single direction. After working with IBM we identified one important step that we were missing, running the encryptproperties.bat . This is required with any change to the Maximo property file on all 7.x versions of Maximo.

 http://www-01.ibm.com/support/docview.wss?uid=swg21314942 

Here are some of the errors we got

• BMXAA6539E - Failed to initialize the MAXIMOStartupServlet 
• WSVR0209E: Unable to start EJB jar mboejb.jar
• java.lang.NoClassDefFoundError: psdi.iface.jms.JMSListenerBean
• WSVR0100W: An error occurred initializing, MAXIMO
• WAE0008E An error occurred reading mbojava.jar

Another anomaly that will be researched further had to do with the sequence required to get past the errors above.

1. Deploy and Start the app to the App Server/ Cluster - Expect Class loading WSVVR like error 
2. Stop App Server / Cluster
3. Start App Server / Cluster  - App should load pass the error in step 1.  

manageprofiles.bat doc

Great link that outlines the use of the manageprofiles command.

The Pain of Windows and WebSphere 6.1

This is a repost of a blog entry I did on the Webshereusergroup.org site. Since this blog is about me getting orgnized and keeping track of things I run into I am reposting it here.

Last Friday we started to experience slowness in the response from our production web servers. The CPU and memory on each server was not above normal. The application servers were responding as expected it was only traffic directed to or through the web servers that slowed. At first the report seemed to be only a nuisance and nothing more, but as time went on the web servers went from a 30 second response time to four or five minutes
In the web servers logs I found the following message sometimes several times a second

[Fri Apr 06 07:54:07 2012] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

The frequency of the messages had increased and on each server we were getting as many as 6 or 7 a second. The slowness of the servers got progressively worse until our entire collection of web servers had crashed. After the crash of the first web server I tried to restart it using the console. The web server would not start. After the other two web servers crashed production was no longer available so  we restarted the OS on the first web server that crashed, it was already broken what more could happen. After the reboot the web server started back up normally and the error messages were no longer being generated.

We then logged a PMR with IBM for more information about this issue. Within an hour I received a response that indicated that this is a known issue in a Windows environment where "other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions" http://publib.boulder.ibm.com/httpserv/ihsdiag/errorlog.html#LSP                                                                   

We had read online that other vendors software could include “anti-virus, firewall, virtualization, or vpn” http://rob.brooks-bilson.com/index.cfm/2008/1/4/Intermittent-Apache-Problems-and-winntaccept-Asynchronous-AcceptEx-failed  Post outage we returned to each server and verified in the add remove software that not updates or new software had been installed in the last day. Anti Virus updates had run but several hours before the first record of the error in the log file. IBM did let us know that a fix does exists for this error however the version of Apache that WebSphere 6.1 runs does not support the fix.

MXServerRemoteImpl (Incompatible magic value 169877536)

Recently several of our servers have been complaining about an “Incompatible magic value” when we try to connect with the TRM Rules Manager IDE, Eclipse. We narrowed it down to an application issue and specifically the way that the ear file is built. Everything worked as expected when we do not use Application server security but when we enable Application Server security we get the incompatible magic value error.

After contacting TRM it turns out that this is related to the use of the FORM auth-method instead of the BASIC auth-method. The different between FORM and BAISC is that instead of getting the clean looking IBM Maixmo login page you get a generic looking browser prompt to enter your user name and password, but Eclipse can connect. 

Troubleshooting this issue required me to grow my ANT knowledge so I wanted to include some of the links I found along the way.

http://en.wikibooks.org/wiki/Apache_Ant

http://ant.apache.org/manual/

This link is a hack of sorts in that it is a directory and not a html page that represents all of the ANT commands.

http://ant.apache.org/manual/Tasks/