Maximo Cron Tasks not running... ahhh just google it

After a large update to the infrastructure to support our Maximo DR project the first clone of the server environment left our cron tasks stalled after one run. We tried to reload them, then stopped and restarted them and just before I turned server logging on I thought "I should google this.." Five minutes later I discovered that our servers where in admin mode thanks to Chon at Maximotimes

Our environment  took me to Go To -> Configuration -> Database Configuration Then Select Action -> Manage Admin Mode 

BMXAA6464I and BMXAA6473E Error Msgs

After restarting one of my fail over nodes I notice no Maximo when I tried to log in and the following in msg in the log file.

[INFO] BMXAA6464I - A registry was not found. The system is creating a registry.
BMXAA6473E - Maximo failed to start.

Quick check of the RMI server showed they were down so I checked the MonitoringPolicy on the RMI server and it was set to STOPPED not PREVIOUS. I started the RMI server update the MonitoringPolicy  to PREVIOUS and restarted the node to test the change. The system started as expected.

My BMXAA1347E was really a CWSIK0024E ??? You don't say

I have been working the last few months on a Maximo disaster recovery config for our servers and during QA testing I ran into a strange issue with one of our MIF integrations. Every time I tried to send over a transaction I got a "BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination".

Overview
The integration in question is a sequential inbound (or Enterprise Service ) that uses a interface table as the entry point. Records are inserted into the interface table were a database trigger inserts a record into the MXIN_INTER_TRANS table. The JMSQSEQCONSUMER SEQQIN cron task then picks the records up and processes it using a WebSphere SIB running in it's own Maximo cluster.

The Error

After I inserted a record into the interface table I quickly queried the MXIN_INTER_TRANS table and returned was one row confirming that the trigger worked as expected. After the cron task ran I noticed the one row was updated with an error msg. I quickly checked the standard out and found the following.


7/10/13 23:00:19:512 MDT] 00000050 SystemOut     O 10 Jul 2013 23:00:19:512 [ERROR] BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination jms/maximo/int/queues/sqin.psdi.util.MXApplicationException: BMXAA1347E - A message could not be sent to the Java Message Service (JMS) destination jms/maximo/int/queues/sqin. at psdi.iface.jms.JMSProducer.handleError(JMSProducer.java:275) at psdi.iface.jms.JMSProducer.sendMessage(JMSProducer.java:216) at psdi.iface.jms.MEAQueueProcessor.writeToQueue(MEAQueueProcessor.java:263) at psdi.iface.jms.MEAQueueProcessor.writeDataToQueueIn(MEAQueueProcessor.java:225) at psdi.iface.intertables.IfaceTbCronTask.sendMessage(IfaceTbCronTask.java:929) at psdi.iface.intertables.IfaceTbCronTask.processData(IfaceTbCronTask.java:1008) at psdi.iface.intertables.IfaceTbCronTask.processIfaceData(IfaceTbCronTask.java:835) at psdi.iface.intertables.IfaceTbCronTask.cronAction(IfaceTbCronTask.java:519) at psdi.server.CronTaskManager.callCronMethod(CronTaskManager.java:1556) at psdi.server.CronTaskManager.access$400(CronTaskManager.java:84) at psdi.server.CronTaskManager$CronThread.run(CronTaskManager.java:2074)

Trouble Shooting 
I began comparing the JMS config from our PTCH and DEV environments to QA because the same test had passed in each of the weeks prior.

After many hours nothing... I noticed that the MIF global dir was filling up with XML records from the cron task retrying the interrogation so I deleted all but the most recent xml files.

I then decided to try and upload a XML record that the cron task had already created (study item from
Test 000-501: IBM Maximo Asset Management V7.5 Infrastructure Implementation). From the start center I went to Integration -> External Systems -> <The System in Question and > and the Enterprise Services tab. From here I selected the service in question and clicked the Data Import button. I then browsed to the MIF global dir and selected the last record processed. It did not work but I got much more detail from the log of the server I was logged onto.


[7/11/13 9:02:42:417 MDT] 00000081 SystemOut     O DEBUG: Control uploadfile could not be found in order to process event loadData :: 
[7/11/13 9:02:42:516 MDT] 00000081 SystemOut     O 11 Jul 2013 09:02:42:516 [INFO] wrote data to Queue jms/maximo/int/queues/sqin
[7/11/13 9:02:42:803 MDT] 00000081 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file C:\IBM\WebSphere\AppServer\profiles\Custom01\logs\ffdc\maxqa_ui08_00000081_13.07.11_09.02.42_0.txt
[7/11/13 9:02:42:815 MDT] 00000081 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\IBM\WebSphere\AppServer\profiles\Custom01\logs\ffdc\maxqa_ui08_00000081_13.07.11_09.02.42_0.txt
[7/11/13 9:02:42:829 MDT] 00000081 SystemOut     O 11 Jul 2013 09:02:42:821 [ERROR]
CWSIA0053E: An exception was received during the call to the method JmsSessionImpl.commitTransaction (#3): com.ibm.wsspi.sib.core.exception.SIRollbackException: 

CWSIC2008E: This transaction cannot commit as an operation that was performed within the transaction boundary failed. The first operation that failed generated the following exception: com.ibm.ws.sib.processor.exceptions.SIMPSendAllowedException: CWSIK0024E: The destination sqinbd is send disallowed for messaging engine maximo_if.000-ifjmsbus...

All right ! 

A quick look up of  CWSIK0024E took me here  where I learned that

CWSIK0024E: The destination {0} is send disallowed for messaging engine {1}.

Explanation An attempt has been made to send a message to a destination and a check has been made to ensure that a queue point is available for this destination. This check failed as the only queue point for this destination is send disallowed.

Action Update the queue point for the destination so it is send allowed and retry the send operation.

Solution 
I found that under  Buses > ifjmsbus > Destinations > sqinbd > Queue points > sqinbd@maximo_if.000-ifjmsbus the "Send allowed" check box was not checked in QA and every other environment has this checked. The XML file that stores this info is cells / Cell01 / clusters / maximo_if / sib-engines.xml

WebSphere docs say this about "Send allowed"

Send allowed

Clear this option (setting it to false) to stop messages from being put onto this message point.

Required	No
Data type	Check box

Conclusion 

Check the "Send Allowed" on queue points because without this checked messages will not be processed.

Turning Logging on in Maximo

Turning Logging on in Maximo

Steps:

Go to the Logging application in Maximo 7

Go to Select Action menu

Select Manage Maximo Root Logger

Click on the little book icon on the right besides the Appenders field

Select Rolling

Click OK
Go to Select Action menu, Apply Settings

To log all SQL statements the SQL logger in the logging application should be set to INFO

Go to Select Actions menu, Apply Settings

The info above is from the following link.
  http://www-01.ibm.com/support/docview.wss?uid=swg21385917

Understanding The HTTP Plugin

I just found two great resources on the WebSphere plugin

• Demystifying the IBM® Web Server Plug-in 
• by Naveen Shetty AND Steve Reid
• Understanding the WebSphere Application Server Web server plug-in
• by Sharad Cocasse And Makarand Kulkarni

manageprofiles.bat and the zipProfile.ant

I recently backed up all of our WebSphere profiles using a home grown PowerShell script and in the process learned a few things about the manageprofiles tool.

Check the profile root for extras ! 

The first time I ran the process on the deployment manager profile it took around 10 to 20 min. When I moved to the profile that the application ran in I was surprised to see the process run for over an hour ! Turns out we had a bunch of .PHD files totaling a few Gigabytes. After I removed the .PHD files the process ran in 20 min.

zipProfile.ant is a trouble maker !

I got two errors that complained about the zipProfile.ant:13: saying  "The process cannot access the file because another process has locked a portion of the file." Turns out that this can be the cause of a few different things I experienced two.  The first was in one of  our testing environments. The appServerRoot>/properties/profileRegistry.xml had a profile listed isAReservationTicket="true". I set this to false (see the IBM notes below ) and the process ran as expected.

Zombies are never a good thing 

The second error was during the PRD release. After checking the profileRegistry.xml to make sure the isAReservationTicket value was not the issue, I checked my processes to see if I had a zombie to kill. Turns out my zombie stuck out like a sore thumb because we use service accounts to run our WebSphere services. 

Finale Thought

Lastly, I wanted to share how fast these error occur. If I had issues they always happened during the first few minutes of the process running.

Content and URL from the IBM KA.
Problem(Abstract)
manageprofiles.sh -backupProfile -profileName <your_profile> -backupFile <your_backup>
Symptom
The command fails with the following message:

Profile <your_profile> is currently in use: Retry the command later.  If there are no other processes are operating on the profile, then the profile might be corrupt.  Run the validateAndUpdateRegistry command and create the profile again.  INSTCONFFAILED: Cannot backup profile: For more information, consult /WebSphere/AppServer/logs/manageprofiles/your_profile_backupProfile.log.


Cause
It is possible a started process locked an XML file or if the process was killed, a zombie process was created that is holding a file lock.
Resolving the problem
Try the following 3 suggestions to help resolve the problem:
You can try rebooting the server. This will clear any zombie processes and any file locks they are holding.

If you cannot reboot the server, then you can try using the "lsof" command to list the open files on the server. Then look for any IBM® WebSphere® Application Server XML file for which there is an open file descriptor. You should be able to identify the process that is referencing that file descriptor and kill/close that process.

The original process may have create a lock file in the OS /tmp directory. Look for any files in the /tmp directory that were created around the time of the first manageprofiles session that you killed. Then remove that file.

If the preceeding 3 suggestions do not resolve the problem, try modifying the profileRegisry.xml file located in the following default path: 

<appServerRoot>/properties/profileRegistry.xml

Note: This will backup the file prior to making any manual changes. 

In this file there will be an entry corresponding to each profile. For example: 

<profile isAReservationTicket="true" isDefault="false" name="your_profile"
path="/WebSphere/AppServer/profiles/<your_profile>"
template="/WebSphere/AppServer/profileTemplates/default"/>

Change the value of isAReservationTicket="true" to isAReservationTicket="false". For example: 

<profile isAReservationTicket="false" isDefault="false" name="your_profile"
path="/WebSphere/AppServer/profiles/<your_profile>"
template="/WebSphere/AppServer/profileTemplates/default"/>


http://www-01.ibm.com/support/docview.wss?uid=swg21320418  

BMXAA6518E - Service RULESMANAGER is not loaded

I got the following error after I rebuilt and deployed the maximo ear file. 

[ERROR] BMXAA6518E - Service RULESMANAGER is not loaded. See the enclosed exception for detail.

psdi.util.MXSystemException: BMXAA3757E - The ServiceStorage: RULESMANAGER service could not be configured.

Turns out I was rushing a bit too much  and ran the buildmaximoear.cmd instead of the 

rm-buildmaximoear.cmd. The rm-buildmaximoear.cmd updates key xml files in the build, merging the Rules Manager product into the Maximo ear.

Server Names in MAXSESSION table

We found this great blog post  that shows how to set up the server names for cluster members. But that's not the real gem in my eyes, it how to use a variable in the JVM args ${WAS_SERVER_NAME} so that you don't have to hard code each JVM name in every app server config.

Apache The Definitive Guide - Book Review

I just finished Apache The Definitive Guide by Ben Laurie & Peter Laurie and would highly recommend this book to anybody who is administrating an Apache or IBM HTTP Server. Apache The Definitive Guide is a ground up approach to teaching you about the Apache web server platform. After spending the last year moving away from a developer role and into a infrastructure role I learned a lot about the IBM HTTP Server  but still had a lot of questions. The Definitive guide tied a lot of concepts together and removed the once dense fog around the web server component of our infrastructure.

Fixing an invalid Bind account

We have been going through a bunch of password changes and mistakenly change the password on a account in one environment that broke another environment. One of the things I learned (other then never reuse accounts ! ) was that when the bind account is invalid, even the wasadmin account will not be able to access the console. The environment we were working in was development and needing a fix I tried to edit the profiles\Dmgr01\config\cells\Cell01\wim\config\wimconfig.xml and updated the value for bindPassword and the bind account to a value from another instance .  What made this interesting was the fact that I was able to use a encrypted password from a WebSphere 6.1 environment in a WebSphere 7.0 environment. I was also able to update the bind password value with a clear text copy, allowing me access to the console. After I entered the console I went to
Secure administration, applications, and infrastructure > Federated repositories > repository
 and was then able to re encrypt the password by reentering it in the bind password box, clicking apply and then save.

WebSphere System Management & Config

My study notes from the following redbook

System Management Overview
    Terminology

• Stand-alone server environment refers to a single server environment that is not managed as part of a cell. This is your only option for base and express version of WebSphere but Network Deployment can be configured as a stand-alone server. 
• Distributed server environment refers to a environment that has multiple servers that are managed by a deployment manger in the cell. 
• Manage Process refers to deployment manager nodes (and node agents) and application servers. 

Stand-alone Single Server

 Distributed Server Environment 

Java Management Extensions (JMX)

   WebSphere system management functions are based on JMX a standard way to expose management objects. JMX