Maximo Cron Tasks not running... ahhh just google it

After a large update to the infrastructure to support our Maximo DR project the first clone of the server environment left our cron tasks stalled after one run. We tried to reload them, then stopped and restarted them and just before I turned server logging on I thought "I should google this.." Five minutes later I discovered that our servers where in admin mode thanks to Chon at Maximotimes

Our environment  took me to Go To -> Configuration -> Database Configuration Then Select Action -> Manage Admin Mode 

BMXAA6464I and BMXAA6473E Error Msgs

After restarting one of my fail over nodes I notice no Maximo when I tried to log in and the following in msg in the log file.

[INFO] BMXAA6464I - A registry was not found. The system is creating a registry.
BMXAA6473E - Maximo failed to start.

Quick check of the RMI server showed they were down so I checked the MonitoringPolicy on the RMI server and it was set to STOPPED not PREVIOUS. I started the RMI server update the MonitoringPolicy  to PREVIOUS and restarted the node to test the change. The system started as expected.

My BMXAA1347E was really a CWSIK0024E ??? You don't say

I have been working the last few months on a Maximo disaster recovery config for our servers and during QA testing I ran into a strange issue with one of our MIF integrations. Every time I tried to send over a transaction I got a "BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination".

Overview
The integration in question is a sequential inbound (or Enterprise Service ) that uses a interface table as the entry point. Records are inserted into the interface table were a database trigger inserts a record into the MXIN_INTER_TRANS table. The JMSQSEQCONSUMER SEQQIN cron task then picks the records up and processes it using a WebSphere SIB running in it's own Maximo cluster.

The Error

After I inserted a record into the interface table I quickly queried the MXIN_INTER_TRANS table and returned was one row confirming that the trigger worked as expected. After the cron task ran I noticed the one row was updated with an error msg. I quickly checked the standard out and found the following.


7/10/13 23:00:19:512 MDT] 00000050 SystemOut     O 10 Jul 2013 23:00:19:512 [ERROR] BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination jms/maximo/int/queues/sqin.psdi.util.MXApplicationException: BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination jms/maximo/int/queues/sqin. at psdi.iface.jms.JMSProducer.handleError(JMSProducer.java:275) at psdi.iface.jms.JMSProducer.sendMessage(JMSProducer.java:216) at psdi.iface.jms.MEAQueueProcessor.writeToQueue(MEAQueueProcessor.java:263) at psdi.iface.jms.MEAQueueProcessor.writeDataToQueueIn(MEAQueueProcessor.java:225) at psdi.iface.intertables.IfaceTbCronTask.sendMessage(IfaceTbCronTask.java:929) at psdi.iface.intertables.IfaceTbCronTask.processData(IfaceTbCronTask.java:1008) at psdi.iface.intertables.IfaceTbCronTask.processIfaceData(IfaceTbCronTask.java:835) at psdi.iface.intertables.IfaceTbCronTask.cronAction(IfaceTbCronTask.java:519) at psdi.server.CronTaskManager.callCronMethod(CronTaskManager.java:1556) at psdi.server.CronTaskManager.access$400(CronTaskManager.java:84) at psdi.server.CronTaskManager$CronThread.run(CronTaskManager.java:2074)

Trouble Shooting 
I began comparing the JMS config from our PTCH and DEV environments to QA because the same test had passed in each of the weeks prior.

After many hours nothing... I noticed that the MIF global dir was filling up with XML records from the cron task retrying the interrogation so I deleted all but the most recent xml files.

I then decided to try and upload a XML record that the cron task had already created (study item from
Test 000-501: IBM Maximo Asset Management V7.5 Infrastructure Implementation). From the start center I went to Integration -> External Systems -> <The System in Question and > and the Enterprise Services tab. From here I selected the service in question and clicked the Data Import button. I then browsed to the MIF global dir and selected the last record processed. It did not work but I got much more detail from the log of the server I was logged onto.


[7/11/13 9:02:42:417 MDT] 00000081 SystemOut     O DEBUG: Control uploadfile could not be found in order to process event loadData :: 
[7/11/13 9:02:42:516 MDT] 00000081 SystemOut     O 11 Jul 2013 09:02:42:516 [INFO] wrote data to Queue jms/maximo/int/queues/sqin
[7/11/13 9:02:42:803 MDT] 00000081 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file C:\IBM\WebSphere\AppServer\profiles\Custom01\logs\ffdc\maxqa_ui08_00000081_13.07.11_09.02.42_0.txt
[7/11/13 9:02:42:815 MDT] 00000081 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\IBM\WebSphere\AppServer\profiles\Custom01\logs\ffdc\maxqa_ui08_00000081_13.07.11_09.02.42_0.txt
[7/11/13 9:02:42:829 MDT] 00000081 SystemOut     O 11 Jul 2013 09:02:42:821 [ERROR]
CWSIA0053E: An exception was received during the call to the method JmsSessionImpl.commitTransaction (#3): com.ibm.wsspi.sib.core.exception.SIRollbackException: 

CWSIC2008E: This transaction cannot commit as an operation that was performed within the transaction boundary failed. The first operation that failed generated the following exception: com.ibm.ws.sib.processor.exceptions.SIMPSendAllowedException: CWSIK0024E: The destination sqinbd is send disallowed for messaging engine maximo_if.000-ifjmsbus...

All right ! 

A quick look up of  CWSIK0024E took me here  where I learned that

CWSIK0024E: The destination {0} is send disallowed for messaging engine {1}.

Explanation An attempt has been made to send a message to a destination and a check has been made to ensure that a queue point is available for this destination. This check failed as the only queue point for this destination is send disallowed.

Action Update the queue point for the destination so it is send allowed and retry the send operation.

Solution 
I found that under  Buses > ifjmsbus > Destinations > sqinbd > Queue points > sqinbd@maximo_if.000-ifjmsbus the "Send allowed" check box was not checked in QA and every other environment has this checked. The XML file that stores this info is cells / Cell01 / clusters / maximo_if / sib-engines.xml

WebSphere docs say this about "Send allowed"

Send allowed

Clear this option (setting it to false) to stop messages from being put onto this message point.

Required	No
Data type	Check box

Conclusion 

Check the "Send Allowed" on queue points because without this checked messages will not be processed.

Turning Logging on in Maximo

Turning Logging on in Maximo

Steps:

Go to the Logging application in Maximo 7

Go to Select Action menu

Select Manage Maximo Root Logger

Click on the little book icon on the right besides the Appenders field

Select Rolling

Click OK
Go to Select Action menu, Apply Settings

To log all SQL statements the SQL logger in the logging application should be set to INFO

Go to Select Actions menu, Apply Settings

The info above is from the following link.
  http://www-01.ibm.com/support/docview.wss?uid=swg21385917

Understanding The HTTP Plugin

I just found two great resources on the WebSphere plugin

• Demystifying the IBM® Web Server Plug-in 
• by Naveen Shetty AND Steve Reid
• Understanding the WebSphere Application Server Web server plug-in
• by Sharad Cocasse And Makarand Kulkarni

manageprofiles.bat and the zipProfile.ant

I recently backed up all of our WebSphere profiles using a home grown PowerShell script and in the process learned a few things about the manageprofiles tool.

Check the profile root for extras ! 

The first time I ran the process on the deployment manager profile it took around 10 to 20 min. When I moved to the profile that the application ran in I was surprised to see the process run for over an hour ! Turns out we had a bunch of .PHD files totaling a few Gigabytes. After I removed the .PHD files the process ran in 20 min.

zipProfile.ant is a trouble maker !

I got two errors that complained about the zipProfile.ant:13: saying  "The process cannot access the file because another process has locked a portion of the file." Turns out that this can be the cause of a few different things I experienced two.  The first was in one of  our testing environments. The appServerRoot>/properties/profileRegistry.xml had a profile listed isAReservationTicket="true". I set this to false (see the IBM notes below ) and the process ran as expected.

Zombies are never a good thing 

The second error was during the PRD release. After checking the profileRegistry.xml to make sure the isAReservationTicket value was not the issue, I checked my processes to see if I had a zombie to kill. Turns out my zombie stuck out like a sore thumb because we use service accounts to run our WebSphere services. 

Finale Thought

Lastly, I wanted to share how fast these error occur. If I had issues they always happened during the first few minutes of the process running.

Content and URL from the IBM KA.
Problem(Abstract)
manageprofiles.sh -backupProfile -profileName <your_profile> -backupFile <your_backup>
Symptom
The command fails with the following message:

Profile <your_profile> is currently in use: Retry the command later.  If there are no other processes are operating on the profile, then the profile might be corrupt.  Run the validateAndUpdateRegistry command and create the profile again.  INSTCONFFAILED: Cannot backup profile: For more information, consult /WebSphere/AppServer/logs/manageprofiles/your_profile_backupProfile.log.


Cause
It is possible a started process locked an XML file or if the process was killed, a zombie process was created that is holding a file lock.
Resolving the problem
Try the following 3 suggestions to help resolve the problem:
You can try rebooting the server. This will clear any zombie processes and any file locks they are holding.

If you cannot reboot the server, then you can try using the "lsof" command to list the open files on the server. Then look for any IBM® WebSphere® Application Server XML file for which there is an open file descriptor. You should be able to identify the process that is referencing that file descriptor and kill/close that process.

The original process may have create a lock file in the OS /tmp directory. Look for any files in the /tmp directory that were created around the time of the first manageprofiles session that you killed. Then remove that file.

If the preceeding 3 suggestions do not resolve the problem, try modifying the profileRegisry.xml file located in the following default path: 

<appServerRoot>/properties/profileRegistry.xml

Note: This will backup the file prior to making any manual changes. 

In this file there will be an entry corresponding to each profile. For example: 

<profile isAReservationTicket="true" isDefault="false" name="your_profile"
path="/WebSphere/AppServer/profiles/<your_profile>"
template="/WebSphere/AppServer/profileTemplates/default"/>

Change the value of isAReservationTicket="true" to isAReservationTicket="false". For example: 

<profile isAReservationTicket="false" isDefault="false" name="your_profile"
path="/WebSphere/AppServer/profiles/<your_profile>"
template="/WebSphere/AppServer/profileTemplates/default"/>


http://www-01.ibm.com/support/docview.wss?uid=swg21320418  

BMXAA6518E - Service RULESMANAGER is not loaded

I got the following error after I rebuilt and deployed the maximo ear file. 

[ERROR] BMXAA6518E - Service RULESMANAGER is not loaded. See the enclosed exception for detail.

psdi.util.MXSystemException: BMXAA3757E - The ServiceStorage: RULESMANAGER service could not be configured.

Turns out I was rushing a bit too much  and ran the buildmaximoear.cmd instead of the 

rm-buildmaximoear.cmd. The rm-buildmaximoear.cmd updates key xml files in the build, merging the Rules Manager product into the Maximo ear.

Server Names in MAXSESSION table

We found this great blog post  that shows how to set up the server names for cluster members. But that's not the real gem in my eyes, it how to use a variable in the JVM args ${WAS_SERVER_NAME} so that you don't have to hard code each JVM name in every app server config.

Apache The Definitive Guide - Book Review

I just finished Apache The Definitive Guide by Ben Laurie & Peter Laurie and would highly recommend this book to anybody who is administrating an Apache or IBM HTTP Server. Apache The Definitive Guide is a ground up approach to teaching you about the Apache web server platform. After spending the last year moving away from a developer role and into a infrastructure role I learned a lot about the IBM HTTP Server  but still had a lot of questions. The Definitive guide tied a lot of concepts together and removed the once dense fog around the web server component of our infrastructure.

Fixing an invalid Bind account

We have been going through a bunch of password changes and mistakenly change the password on a account in one environment that broke another environment. One of the things I learned (other then never reuse accounts ! ) was that when the bind account is invalid, even the wasadmin account will not be able to access the console. The environment we were working in was development and needing a fix I tried to edit the profiles\Dmgr01\config\cells\Cell01\wim\config\wimconfig.xml and updated the value for bindPassword and the bind account to a value from another instance .  What made this interesting was the fact that I was able to use a encrypted password from a WebSphere 6.1 environment in a WebSphere 7.0 environment. I was also able to update the bind password value with a clear text copy, allowing me access to the console. After I entered the console I went to
Secure administration, applications, and infrastructure > Federated repositories > repository
 and was then able to re encrypt the password by reentering it in the bind password box, clicking apply and then save.

WebSphere System Management & Config

My study notes from the following redbook

System Management Overview
    Terminology

• Stand-alone server environment refers to a single server environment that is not managed as part of a cell. This is your only option for base and express version of WebSphere but Network Deployment can be configured as a stand-alone server. 
• Distributed server environment refers to a environment that has multiple servers that are managed by a deployment manger in the cell. 
• Manage Process refers to deployment manager nodes (and node agents) and application servers. 

Stand-alone Single Server

 Distributed Server Environment 

Java Management Extensions (JMX)

   WebSphere system management functions are based on JMX a standard way to expose management objects. JMX  

  

SECJ0305I Error, Node Agent not Running

This morning I got to work and learned that one of our node agents was not running. What made this unique was that the node agent in question was running on a virtual machine that had recently been moved from one data center to another, across town changing the IP address. After checking the node agent's System.out  I determined that the last time the server been restated the node agent had started successfully.

WSVR0001I: Server nodeagent open for e-business

Puzzled at this point I pinged all three servers from each server and got a reply every time. I then restarted the windows service that was running the node agent and it failed to restart. Looking into the log I discovered a new error stack that started with a

CWWIM4512E The password match failed.

and ended with

SECJ0305I: The role-based authorization check failed for admin-authz operation

I did some goggling and found this tech note from IBM that recommends "restart(ing) the deployment manager, node agents, and servers." This resolved the original issue where the node agent showed as not running but when I tried to restart the node agent service to show what I had done to a peer I got the same error but the node agent status never changed from running in the Deployment manager  : (  I will post a follow up to this issue at a later date (when it is worked out).

Growing access.logs for IBM HTTP Server

While working on some performance issues in Prd we discovered that our web server access.logs were in excess of 2GB and still growing strong. After some research we decided it was time to implement piped logs . We first tried to edit the httpd.conf file for the web server through the WebSphere admin console. We replaced the following 

CustomLog logs/access.log common

with

CustomLog "|bin/rotatelogs logs/access.%Y.%m.%d 86400" common 

Stopped the web server , tried to start it and after a 30 sec pause we got a msg back saying that the web server could not be started. Further research into the log files for the deployment manager and web server did not shed any light on why. After some digging on IBM's support site I found this technote explaining that the full path must be specified. I replaced the above with this, (adjusting the formatting to better fit our needs) and the server started without issue and our log files were now rolling on a 24hr bases.

CustomLog "|C:/IBM/HTTPServer/bin/rotatelogs.exe -l C:/IBM/HTTPServer/logs/access-%a-%m-%d-%Y.log 86400" common 

Maximo 7.5.0.2 and WebSphere 8

We are in the process of setting up Maximo 7.5.0.2 and recently my lead ran across this link from IBM that outlines how to do just that but with WebSphere 8 as the middleware. Currently we have Maximo 7.5 set up with WebSphere 7 but are evaluating upgrading WebSphere to 8. This tech journal from IBM outlines some interesting points on why this might be a good idea. The one that I found the most interesting was the modification to the addnode command providing an add existing option.

Node recovery

A new optional argument for the addNode command, asExisitingNode, makes it easier to move or recover nodes. When this command argument is used, the node is added using the configuration from the Deployment Manager for that node. As a result, a node can be easily moved to a new server and, if need be, there are provisions for changing the host name during this process. In the case of a hardware failure, the node can be easily recovered back to the last configuration reflected in the cell configuration maintained by the Deployment Manager. These two scenarios are depicted in Figure 5.

When in Dought Encrypt It !

Recently I ran into trouble getting two Maximo environments up one 7.1 and the other 7.5.  We got several different errors and none pointed us in a single direction. After working with IBM we identified one important step that we were missing, running the encryptproperties.bat . This is required with any change to the Maximo property file on all 7.x versions of Maximo.

 http://www-01.ibm.com/support/docview.wss?uid=swg21314942 

Here are some of the errors we got

• BMXAA6539E - Failed to initialize the MAXIMOStartupServlet 
• WSVR0209E: Unable to start EJB jar mboejb.jar
• java.lang.NoClassDefFoundError: psdi.iface.jms.JMSListenerBean
• WSVR0100W: An error occurred initializing, MAXIMO
• WAE0008E An error occurred reading mbojava.jar

Another anomaly that will be researched further had to do with the sequence required to get past the errors above.

1. Deploy and Start the app to the App Server/ Cluster - Expect Class loading WSVVR like error 
2. Stop App Server / Cluster
3. Start App Server / Cluster  - App should load pass the error in step 1.  

manageprofiles.bat doc

Great link that outlines the use of the manageprofiles command.

The Pain of Windows and WebSphere 6.1

This is a repost of a blog entry I did on the Webshereusergroup.org site. Since this blog is about me getting orgnized and keeping track of things I run into I am reposting it here.

Last Friday we started to experience slowness in the response from our production web servers. The CPU and memory on each server was not above normal. The application servers were responding as expected it was only traffic directed to or through the web servers that slowed. At first the report seemed to be only a nuisance and nothing more, but as time went on the web servers went from a 30 second response time to four or five minutes
In the web servers logs I found the following message sometimes several times a second

[Fri Apr 06 07:54:07 2012] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

The frequency of the messages had increased and on each server we were getting as many as 6 or 7 a second. The slowness of the servers got progressively worse until our entire collection of web servers had crashed. After the crash of the first web server I tried to restart it using the console. The web server would not start. After the other two web servers crashed production was no longer available so  we restarted the OS on the first web server that crashed, it was already broken what more could happen. After the reboot the web server started back up normally and the error messages were no longer being generated.

We then logged a PMR with IBM for more information about this issue. Within an hour I received a response that indicated that this is a known issue in a Windows environment where "other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions" http://publib.boulder.ibm.com/httpserv/ihsdiag/errorlog.html#LSP                                                                   

We had read online that other vendors software could include “anti-virus, firewall, virtualization, or vpn” http://rob.brooks-bilson.com/index.cfm/2008/1/4/Intermittent-Apache-Problems-and-winntaccept-Asynchronous-AcceptEx-failed  Post outage we returned to each server and verified in the add remove software that not updates or new software had been installed in the last day. Anti Virus updates had run but several hours before the first record of the error in the log file. IBM did let us know that a fix does exists for this error however the version of Apache that WebSphere 6.1 runs does not support the fix.

MXServerRemoteImpl (Incompatible magic value 169877536)

Recently several of our servers have been complaining about an “Incompatible magic value” when we try to connect with the TRM Rules Manager IDE, Eclipse. We narrowed it down to an application issue and specifically the way that the ear file is built. Everything worked as expected when we do not use Application server security but when we enable Application Server security we get the incompatible magic value error.

After contacting TRM it turns out that this is related to the use of the FORM auth-method instead of the BASIC auth-method. The different between FORM and BAISC is that instead of getting the clean looking IBM Maixmo login page you get a generic looking browser prompt to enter your user name and password, but Eclipse can connect. 

Troubleshooting this issue required me to grow my ANT knowledge so I wanted to include some of the links I found along the way.

http://en.wikibooks.org/wiki/Apache_Ant

http://ant.apache.org/manual/

This link is a hack of sorts in that it is a directory and not a html page that represents all of the ANT commands.

http://ant.apache.org/manual/Tasks/

WebSphere host name changes

Recently we went from Physical to Virtual on all of our Nodes within our WebSphere environment. Recent networking issues made me a tad bit paranoid about this so I cut a PMR with IBM to find out how this could affect us. What I learned was that this would not be an issue because WebSphere only stores hostname and  IP address and does so in the serverindex.xml.

Performance Monitoring Request Metrics (PMRM)

Notes from the following link  by Ken Gottry and IBM support page  about request metrics.

• PMRM are transaction based recording  unlike Performance Monitoring Infrastructure (PMI) that provides information about average system resource usage statistics with no correlation across WebSphere Components. 
• Records are written to the system.out log of the app server that the request is made on. 
• The webserver has it's own log to write transaction (http-plugin.log)
• The last two records record the servlet and response time and in the webserver log the size of the request and size of response are also recorded.  

• Why use request metrics ? 
• Request metrics allow you you to track individual transactions and the time in each WebSphere component. 






• Request Metrics Filters 
• Filters exists for 
• EJB
• URI
• Source IP
• Web Services 
• JMS Filters
• Allow you to focus on a specific area.