The Pain of Windows and WebSphere 6.1

This is a repost of a blog entry I did on the Webshereusergroup.org site. Since this blog is about me getting orgnized and keeping track of things I run into I am reposting it here.

Last Friday we started to experience slowness in the response from our production web servers. The CPU and memory on each server was not above normal. The application servers were responding as expected it was only traffic directed to or through the web servers that slowed. At first the report seemed to be only a nuisance and nothing more, but as time went on the web servers went from a 30 second response time to four or five minutes
In the web servers logs I found the following message sometimes several times a second

[Fri Apr 06 07:54:07 2012] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

The frequency of the messages had increased and on each server we were getting as many as 6 or 7 a second. The slowness of the servers got progressively worse until our entire collection of web servers had crashed. After the crash of the first web server I tried to restart it using the console. The web server would not start. After the other two web servers crashed production was no longer available so  we restarted the OS on the first web server that crashed, it was already broken what more could happen. After the reboot the web server started back up normally and the error messages were no longer being generated.

We then logged a PMR with IBM for more information about this issue. Within an hour I received a response that indicated that this is a known issue in a Windows environment where "other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions" http://publib.boulder.ibm.com/httpserv/ihsdiag/errorlog.html#LSP                                                                   

We had read online that other vendors software could include “anti-virus, firewall, virtualization, or vpn” http://rob.brooks-bilson.com/index.cfm/2008/1/4/Intermittent-Apache-Problems-and-winntaccept-Asynchronous-AcceptEx-failed  Post outage we returned to each server and verified in the add remove software that not updates or new software had been installed in the last day. Anti Virus updates had run but several hours before the first record of the error in the log file. IBM did let us know that a fix does exists for this error however the version of Apache that WebSphere 6.1 runs does not support the fix.

MXServerRemoteImpl (Incompatible magic value 169877536)

Recently several of our servers have been complaining about an “Incompatible magic value” when we try to connect with the TRM Rules Manager IDE, Eclipse. We narrowed it down to an application issue and specifically the way that the ear file is built. Everything worked as expected when we do not use Application server security but when we enable Application Server security we get the incompatible magic value error.

After contacting TRM it turns out that this is related to the use of the FORM auth-method instead of the BASIC auth-method. The different between FORM and BAISC is that instead of getting the clean looking IBM Maixmo login page you get a generic looking browser prompt to enter your user name and password, but Eclipse can connect. 

Troubleshooting this issue required me to grow my ANT knowledge so I wanted to include some of the links I found along the way.

http://en.wikibooks.org/wiki/Apache_Ant

http://ant.apache.org/manual/

This link is a hack of sorts in that it is a directory and not a html page that represents all of the ANT commands.

http://ant.apache.org/manual/Tasks/

WebSphere host name changes

Recently we went from Physical to Virtual on all of our Nodes within our WebSphere environment. Recent networking issues made me a tad bit paranoid about this so I cut a PMR with IBM to find out how this could affect us. What I learned was that this would not be an issue because WebSphere only stores hostname and  IP address and does so in the serverindex.xml.

Performance Monitoring Request Metrics (PMRM)

Notes from the following link  by Ken Gottry and IBM support page  about request metrics.

• PMRM are transaction based recording  unlike Performance Monitoring Infrastructure (PMI) that provides information about average system resource usage statistics with no correlation across WebSphere Components. 
• Records are written to the system.out log of the app server that the request is made on. 
• The webserver has it's own log to write transaction (http-plugin.log)
• The last two records record the servlet and response time and in the webserver log the size of the request and size of response are also recorded.  

• Why use request metrics ? 
• Request metrics allow you you to track individual transactions and the time in each WebSphere component. 






• Request Metrics Filters 
• Filters exists for 
• EJB
• URI
• Source IP
• Web Services 
• JMS Filters
• Allow you to focus on a specific area. 

Dynacache in WebSphere 7

Notes on Dynacache post and WebSphere doc on setting

Caching of too much data can cause performance issues and this makes sizing of the cache very important. Selecting a correct size of cache can be difficult and because of this an underutilized cache can occur. This problem is not easily solved  because Java does not have a size of operator that will tell us the size of a object on the heap.

Dynacache allows the administrator the ability to control the cache by setting high and low water marks on the heap size.

• Dynamic Cache service setting can be found Servers - Server Types - WebSphere application servers - server name> Container services - Dynamic cache service
• Service starts with caching is enabled in Web Container panel.
• Cache size 
• Positive integer  represents the maximum number of entries the cache can hold.
• Default priority 
• How long an entry stays in a full cache   
• Limit memory cache size
• Sets the size of the memory cache. Allows you to control the size of cache in terms of the JVM heap. 
• The least recently used algorithm is used to remove items from cache. 
• Memory Cache size
• Allows you to set the cache in MB. 
• High threshold and low threshold represent the high and low watermarks. 
• expressed in terms of percentage of the memory cache

• Enable disk offload

• Allows items removed from memory to be moved to disk if needed later 
• You CANNOT specify the number of items moved to disk
• You CANNOT specify the amount of disk space to use.  
• Offload location 
• Location on disk to save entry's
• Default ${WAS_TEMP_DIR}/node/server name/_dynacache/cache JNDI name  
• ${WAS_TEMP_DIR} is install/temp dir
• If location is specified then node, server name and cache instance name is appended
• If you use the default dir and the server fills up WebSphere could stall
• Depending on OS you may see disk full messages in the console. 
• Flush to disk 
• Indicates if in memory cache should be written to disk in the event that the app server is shutdown. 
• Limit Disk size in GB
• leaving blank indicates unlimited.
• Limit disk size in entries
• leaving blank indicates unlimited. 
• Disk Cache Performance Settings (how memory resources should be used on background activity such as cache cleanup, expiration, garbage collection, and so on)
• High - all metadata kept in memory
• Balanced - some metadata kept, balance of performance and memory usage found
• Low - limited metatdata is kept
• Custom - Admin will explicitly configure memory setting 
• Set with the  DiskCacheCustomPerformanceSettings  object
• Disk Cache cleanup frequency 
• set in minutes is set to 0 only happens at midnight. 
• Only applies with Per Setting are low, balanced or custom
• High does not require disk cleanup
• Maximum buffer for cached identifiers per metaentry
• Sets number of maximum number of cache identifiers that are stored for an individual dependency 
• If limit exceeded data is off loaded to disk 
• Only Applies to custom Per Setting 
• Maximum buffer for dependency identifiers 
• Sets the number of dependency identifier buckets in the disk cache metadata in memory.
• Only Applies to custom Per Setting 
• Maximum buffer for templates 
• Only Applies to custom Per Setting 
• Sets the max number of temp buckets
•  Maximum buffer for templates 
• sets the max  number of template buckets
• Only Applies to custom Per Setting
• Disk Cache eviction algorithm 
• Only applies if disk offload is specified
• None - once this disk cache reaches the disk size the service stops writing to disk
• Random - 
• Size - Largest are removed first
• High Threshold 
• Sets when the eviction policy runs
• Percent of disk space
• Low Threshold
• Sets when the eviction policy ends
• Percent of disk space
• Enable Cache replication 
• Uses cache replication to have cache entries copied to member of a replication domain
• Full group replication domain
• sets the replication domain
• Replication type (Direct from IBM doc)
• Specifies the global sharing policy for this application server.
• The following settings are available:
• Both push and pull sends the cache ID of newly updated content to other servers in the replication domain. Then, if one of the other servers requests the content, and that server has the ID of the cache entry for the previously updated content, it will retrieve the content from the publishing server. On the other hand, if a request is made for an ID which has not been previously published, the server assumes it does not exist in the cluster and creates a new entry.
• Push only sends the cache ID and cache content of new content to all other servers in the replication domain.
• When you use the Not Shared setting, as cache entries are created, neither the cache content nor the cache IDs are propagated to other servants or servers in the replication domain. However, invalidations are propagated to other servants or servers. You can set the sharing policy at different levels. A global sharing policy, which is the default policy for all caches, is defined when you configure the dynamic cache service. You can overwrite this sharing policy by modifying the cachespec.xml file. For more information on thecachespec.xml file, see the cachespec.xml file topic. Additionally, you can overwrite the sharing policy at the application programming interface (API) level when cache entries are being created.
• The default is Not Shared.
• Push frequency 
• Time in seconds before new or modified cache entries are pushed to other servers

Notes on WebSphere Data Replication Services (DRS)

My Notes from the IBM Education Assistant  on DRS

DRS is a internal component of WebSphere that is used to move data within the app server process. Examples of this include

• HTTP Session replication 
• Dynamic Cache Replication 
• EJB state replication (New at WebSphere 6)

To insure that data and request wind up in the same place DRS coordinates with workload management. 

DRS provides services in two scenarios

• Fail over
• Ensure HTTP session and EJBs can be move to another server transparent to the user.  
• Caching 
• When a servlet or JSP has been configured to have it's output cached repeated request are handled faster. This cached output is what DRS syncs. 

DRS in V5 WebSphere requires the administrator to configure the replicators, replication domain and partition.

• Replicators
• Producer and Consumer responsible for moving data
• Data moves as JMS messages
• Replication Domain
• A set of one or more replicators
• Partition
• a group of replicators configured to communicate with each other.   

DRS is V6 WebSphere requires the administrator to only configure a Replication Domain as configuring replicators is no longer necessary and partition are masked from the user. 

• Replication Domain
• Consists of server or cluster members that have the capability of sharing HTTP Session or Caching data within the domain 
• New in V6 is the coordination with Workload Management (WLM) to coordinate which members serve as backups for other members.   
• Ideally session fail over data stateful session bean data should end up in the same place and somewhere other then where the data originated.  
• Rewritten in V6 using a IBM propitiatory mechanization  to transport data. 
• Only configuration option now is that the number of replications can be set (default is 1)

377 Question area 

• Creating a replication domain can be done at cluster creation. 
• 
• Or can be done manually

• Cache replication can be configured under Server - Container Services  - Dynamic cache Service

Best Practices 

• Create distinct domain for HTTP and EJB data and one for caching 
• Put EJB data and HTTP data in the same domain. 
• Use smallest number of replicas as possible 1, 2, or 3 should work in most cases
• Congestion messages can be resolved by increasing the transport buffer size to 50 MB 10 MB is the default. (App Server - <server name > Core Group Service) 

Mobile What !

I have been sitting on the fence about a tablet just for me for a while now. My wife and I got a iPad a while back and it have proven to be very popular with the both of us. Looking for better personal computing experence I am thinking about going all out for a new Nexus 7   or adding a Unbuntu OS to my Droid X2. Too bad I am so cheep otherwise I would just get both !