d8888 888 888      88888888888 888      d8b                                 888       888          888       .d8888b.           888                               
      d88888 888 888          888     888      Y8P                                 888   o   888          888      d88P  Y88b          888                               
     d88P888 888 888          888     888                                          888  d8b  888          888      Y88b.               888                               
    d88P 888 888 888          888     88888b.  888 88888b.   .d88b.  .d8888b       888 d888b 888  .d88b.  88888b.   "Y888b.   88888b.  88888b.   .d88b.  888d888 .d88b.  
   d88P  888 888 888          888     888 "88b 888 888 "88b d88P"88b 88K           888d88888b888 d8P  Y8b 888 "88b     "Y88b. 888 "88b 888 "88b d8P  Y8b 888P"  d8P  Y8b 
  d88P   888 888 888          888     888  888 888 888  888 888  888 "Y8888b.      88888P Y88888 88888888 888  888       "888 888  888 888  888 88888888 888    88888888 
 d8888888888 888 888          888     888  888 888 888  888 Y88b 888      X88      8888P   Y8888 Y8b.     888 d88P Y88b  d88P 888 d88P 888  888 Y8b.     888    Y8b.     
d88P     888 888 888          888     888  888 888 888  888  "Y88888  88888P'      888P     Y888  "Y8888  88888P"   "Y8888P"  88888P"  888  888  "Y8888  888     "Y8888  
                                                                 888                                                          888                                        
                                                            Y8b d88P                                                          888                                        
                                                             "Y88P"                                                           888   

All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Thursday, April 26, 2012


What you can learn from the WebSphere Application Server Performance 2012 SPECjEnterprise 2010 Results

IBM did a world leadership publish on SPECjEnt  benchmark that was accepted by the SPEC organization.  The result are available here:  http://www.spec.org/jEnterprise2010/results/res2012q2/.

If you are a WAS customer that needs to squeeze the last bit of performance out of WAS on a performance mission critical application then you should study and emulate in DETAIL all the custom parameters and tuning done by IBM on the application server to get this world class performance of 524 enterprise job transactions per second. See the entire publish here

This benchmark heavily stresses the JPA, web container and web services containers in WAS. After going through the results I discuss the relevant server side tuning below -

Application Server & JVM Tuning:  (*Your mileage may vary)

APAR Fixes 

Hardware Tuning for maximum Application Server performance  (*Your mileage may vary)

fs.file-max = 1048576
kernel.sem = 250 32000 100 128
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
net.core.netdev_max_backlog = 250000
net.core.optmem_max = 30000000
net.core.rmem_default = 30000000
net.core.wmem_default = 30000000
net.core.rmem_max = 32554432
net.core.wmem_max = 32554432
net.core.somaxconn = 32767
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.ip_forward = 0
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem  = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 20480 174760 32554432
net.ipv4.tcp_wmem = 20480 174760 32554432
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
vm.nr_hugepages = 8192
ulimit -n 1048576

Network Tuning:
  set txqueuelen 60000
  Stop irqbalance.
  Network interrupts bound to cores corresponding to the server instances.

Server instances were started using taskset, 2 instances per chip.

Wednesday, April 25, 2012


PMI Statistics - Key modules in WebSphere Application Server

The WebSphere PMI  infrastructure provides a window (metrics) into the runtime state of the WebSphere Application Server. General overhead of basic PMI is roughly 2%. Your mileage may wary depending on which modules you enable when using Custom PMI. Please see attached document for the critical PMI counters in WAS. Key PMI modules in WAS

Sunday, April 22, 2012


IMPROVED Session replication and persistence using the WebSphere Data Replication Service

Recently we have improved the performance and footprint of session serialization and deserialization code when failover occurs in WAS.  Issues like session replication causing  OOM have been fixed by APAR 


When the fix is enabled by setting the JVM custom property DRS_BATCH_INTERVAL_SIZE to 50.  The DRS service will replicate HTTP session data in multiple batches of 50 objects in each batch. 

This fix basically chunks the updates sent by one WAS server to another thereby leading to a reduced heap footprint as serialized byte[] are not buffered till the entire object is in memory.

I suggest enabling this custom property for faster and more efficient serialization of your HTTP sessions  with WebSphere Application Server

Wednesday, April 18, 2012


New Flash: Possible security exposure on April 26, 2012, if using the Web Server Plug-in for WebSphere Application Server

AbstractSSL connections between the plug-in and WebSphere Application Server might fail or revert to non-SSL after the shipped version of the plugin-key.kdb password expires April 26, 2012 US EDT.  Note:  If you are using the WebSphere Key and Certificate Management generated plug-in key store you are NOT affected.   

CVE ID: CVE-2012-2162
CVSS Base Score: 6.8
CVSS Temporal Score: See http://xforce.iss.net/xforce/xfdb/74900 for the current score
CVSS Environmental Score*: Undefined
CVSS String: (AV:N/AC:M/Au:N/C:P/I:P/A:P)
NEW FLASH can viewed at: hhttp://www-01.ibm.com/support/docview.wss?uid=swg21591172

Original Flash titled "Password to the plugin-key.kdb file expires on April 26, 2012 US EDT" may be viewed at:   http://www-01.ibm.com/support/docview.wss?uid=swg21588312


Monday, April 16, 2012


Struts 2 support for Dynacache

Basic Dynacache servlet caching will work with Struts 2; however Dynacache support for struts 2 is tricky since Struts2 is not backwards compatible with Struts1 and there is no ActionServlet in Struts 2.

Please note
- Dynamic caching may NOT always work with ALL the cachespec.xml combinations offered by Dynacache.
- Customer will need to set the  com.ibm.ws.webcontainer.invokefilterscompatibility  webcontainer custom property to true to get strut2 fragment caching to work.
- When caching Struts 2 output, the cache entries will need to be written  for the appropriate JSPs and servlet to be served by the application. 


Friday, April 13, 2012


IBM J9 JIT - Inlining of Virtual Methods - Invariant Argument Preexistence Optimization

With Java 6 the the IBM JIT  is aggressive in optimization leading to world class performance; however sometimes the JIT gets tripped up on itself doing certain optimizations.  If you ran into http://www-01.ibm.com/support/docview.wss?uid=swg1IZ58251 or http://www-01.ibm.com/support/docview.wss?uid=swg1IV15424 or http://www-01.ibm.com/support/docview.wss?uid=swg1IV16044 you may be wondering what is meant by JIT invariant argument optimization and if it makes sense to disable it ?

How does pre-existence work?
For a full and rigorous explanation see Chapter 5 from the paper that introduced the concept:

Assume the following scenario:

void m1(Foo arg)
   arg.m2(); // virtual call or interface call

If at the time of compilation for m1 there is only one implementation for callee m2, then the JIT knows exactly what target to expect and can safely inline the code for it with no guard.
If a class load event extends the hierarchy such that a new implementation for m2 is introduced into the system, the code for m1 may call the wrong target for some type of arguments. The JIT invalidates the old body for m1 and immediately schedules a recompilation for it. While the recompilation is in progress application threads are blocked from entering the old body of caller m1. The new body of m1 will not inline m2, but will actually execute the virtual/interface call (note that the different targets for m2 may already be compiled/optimized on their own). Pre-existence is not going to help in this case.

In a nutshell, pre-existence helps the JIT inline some targets when the class hierarchy indicates it is safe to do so.
1) If the class hierarchy is such that the JIT doesn't know what to inline, then pre-existence does nothing. With or without it it's the same thing.
2) If pre-existence was applied, but later-on the class hierarchy changed in an incompatible way, then the JIT basically needs to undo what pre-existence did. Again, disabling pre-existence from the beginning is not going to hurt this case.

Why have pre-existence enabled?
Some applications, while having deep class hierarchies, load only one implementer for various virtual calls. Experiments have shown improvements from pre-existence  Even when the class hierarchy is extended, we do not see any visible decrease in the performance of  the system due to method invalidation.

When does it make sense to disable pre-existence ?
When an application does a a lot of Dynamic class reloading and deep class hierarchies with interfaces having multiple implementations in mainline path, then it may make sense to disable pre-existence with
-Xjit:disableInvariantArgumentPreexistence generic JVM argument. 


Wednesday, April 4, 2012


Opening large heap or system dumps with Eclipse Memory Analyzer

Sometimes Eclipse Memory Analyzer (MA) our favorite tool for analyzing heap and system dumps dies to an OOM when opening LARGE heaps. When  opening these  large heapdumps ( system dumps or phd's > 4GB)  run Memory Analyzer with the following JVM options


- These arguments are to be put in the eclipse.ini 
- Needs recent version of the JDK preferably the one under WAS (/opt/IBM80/java/bin)
- RAM on the machine needs to be at least the size of the JVM max heap on which the dump was taken.
- MA is NOT CPU bound. So any reasonably multi-core modern processor will do.
- We have seen times of up to 3 hours to parse phd heaps of around 4GB.
- For best performance heap analysis should be done as soon as it is opened otherwise the MA indexes which are maintained as soft references will be GCed and will need to be read back from the disk


Tuesday, April 3, 2012


WebSphere Dynacache L2 Cache for Hibernate

I stumbled upon this google code project Dynacache L2 cache provider for Hibernate

Gaurav has got most of the code right although there are some issues with the code which I list below -

 1. In org.hibernate.cache.dynacache.WASDynaCacheProvider.buildCache(String, Properties) you can create the cache programatically using com.ibm.wsspi.cache.DistributedObjectCacheFactory.getMap(String, Properties)
One of the properties passed in should have been the passed in regionName which directly maps to a cache instance in Dynacache. No need to do a JNDI lookup - 

_cacheInstanceProperties = new Properties();
_cacheInstanceProperties.put(DistributedObjectCacheFactory.KEY_CACHE_SIZE.CACHE_SIZE, String.valueOf(_cacheSize));
_cacheInstanceProperties.put(com.ibm.wsspi.cache.DistributedObjectCacheFactory.KEY_DISKCACHE_SIZE, String.valueOf(_diskCacheSize));
DistributedObjectCacheFactory.createDistributedObjectCache(_cacheName, _cacheName,  _cacheInstanceProperties);

2. org.hibernate.cache.dynacache.WASDynaCache.asWASKey(Object) is unnecessary since Dynacache DistributedMap allows keys to be POJO and not just strings. Dynacache provides multiple cache regions which map 1:1 to the cache instance. There is no need to append the cache region to the cache key. Cache keys in two regions will never collide as they are separate cache instances.

3.  Dynacache memory cache instance size can be specified in # of entries as well as # of bytes on a 32 bit heap. The properties below can be used to control the cache instance size in bytes. Therefore org.hibernate.cache.dynacache.WASDynaCache.getSizeInMemory() could return the max cache size in MB. Unfortunately the external interfaces dont expose the current memory in bytes that the Dynacache implementation computes

4. Apply the following JVM custom properties to tune dynacache

* The code was ASL v2 licensed which is why I was able to read and give feedback.


Monday, April 2, 2012


Java 7 Utility programs for watching directories and finding strings

I have just pushed two very useful java programs/utilities that we have used in the past month for debugging file issues and searching for certain string patterns recursively in a directory.

This program finds strings of an input  pattern recursively through a specified list of pattern matched directories.
Sort of like a java grep. We used this program to find untranslated messages that should have been internationalized.

This  program will let you know if a particular directory is being written too. If any file in the directory is created, modified delated. This is very useful when you have to watch a particular directory structure for any changes.

Both these programs make extensive use of Java7  java.nio.file APIs for listing directories, files and for monitoring directories. Please modify and use for your needs.



Ideal WAS configuration for production , performance test and development




Where to download the latest versions of the IBM Health Center

The IBM Health Center is the built in low cost , overhead and footprint profiler. The IBM Monitoring and Diagnostic Tools for Java - Health Center is a lightweight tool that monitors active IBM Virtual Machines for Java with minimal performance overhead. The Health Center suggests live tuning recommendations for garbage collection, profiles methods including call stacks, and highlights contended locks.

You should always upgrade to the latest version of health center server agent on your WAS JVM following this link -

For more on monitoring a running Java applicaiton on the IBM JDK see

To load the health agent data you will need the IBM Health Center client bundled with the IBM Support Assistant .
Health Center agent tool in IBM Support Assistant



Hacking the JVM inside WebSphere Application Server

Sometimes there arise situations where certain JVM classes inside WebSphere Application Server like java.lang.Thread or java.util.Timer need to be instrumented to add some debug information. In such scnearios locate the src.zip file under WAS_HOME/java/src.zip. This file contains all the JDK source code. Extract the file you want to the eclipse workspace and make the necessary modifications like calling Thread.dumpStack() etc.,

Thereafter package the modified class file in its own jar say patch.jar and add the following property to the JVM generic argument

Note that the "/p" is important.  The bootclasspath is where the JDK loads "core" classes from, and the "/p" says to prepend the value you are supplying, thereby overriding the default location of the core classes.


You can  configure the following WAS JVM generic argument to print  the stacktrace when a thread is started or stopped...

Sample output

JVMDUMP006I Processing Dump Event "thrstart", detail "" - Please Wait.
-------- Console dump --------

Stack Traces of Threads:

ThreadName=Signal Dispatcher(0805BFFC)

Monitor=0805ADE0 (Thread public flags mutex)
        In com/ibm/oti/vm/BootstrapClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;
        In com/ibm/misc/SystemIntialization.lastChanceHook()V
        In java/lang/System.completeInitialization()V
        In java/lang/Thread.(Ljava/lang/String;Ljava/lang/Object;IZ)V

^^^^^^^^ Console dump ^^^^^^^^
JVMDUMP013I Processed Dump Event "thrstart", detail "".



December 2006   September 2008   January 2009   February 2009   March 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   October 2010   January 2011   February 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   September 2012   October 2012   November 2012   January 2013   May 2013   June 2013   July 2013   September 2013   October 2013   June 2014   August 2014  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]