d8888 888 888      88888888888 888      d8b                                 888       888          888       .d8888b.           888                               
      d88888 888 888          888     888      Y8P                                 888   o   888          888      d88P  Y88b          888                               
     d88P888 888 888          888     888                                          888  d8b  888          888      Y88b.               888                               
    d88P 888 888 888          888     88888b.  888 88888b.   .d88b.  .d8888b       888 d888b 888  .d88b.  88888b.   "Y888b.   88888b.  88888b.   .d88b.  888d888 .d88b.  
   d88P  888 888 888          888     888 "88b 888 888 "88b d88P"88b 88K           888d88888b888 d8P  Y8b 888 "88b     "Y88b. 888 "88b 888 "88b d8P  Y8b 888P"  d8P  Y8b 
  d88P   888 888 888          888     888  888 888 888  888 888  888 "Y8888b.      88888P Y88888 88888888 888  888       "888 888  888 888  888 88888888 888    88888888 
 d8888888888 888 888          888     888  888 888 888  888 Y88b 888      X88      8888P   Y8888 Y8b.     888 d88P Y88b  d88P 888 d88P 888  888 Y8b.     888    Y8b.     
d88P     888 888 888          888     888  888 888 888  888  "Y88888  88888P'      888P     Y888  "Y8888  88888P"   "Y8888P"  88888P"  888  888  "Y8888  888     "Y8888  
                                                                 888                                                          888                                        
                                                            Y8b d88P                                                          888                                        
                                                             "Y88P"                                                           888   

All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Monday, July 25, 2011


Java Hung Thread Detection & Handling

Have you ever wondered how hung thread detection works ?
See this article  for an excellent approach for hung thread detection in a Java application -

WebSphere Application Server is the ONLY application server that offers hung thread detection (since version The hang detection option for WebSphere Application Server is turned on by default. You can configure a hang detection policy to accommodate your applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. When a hung thread is detected, WebSphere Application Server notifies you so that you can troubleshoot the problem. For details see Infocenter.

WSVR0605W is the message when there is a hung thread.

In WAS 7, WSVR0605W  is accompanied by a thread dump of hung thread:
[11/16/09 12:41:03:296 PST] 00000020 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (00000021) has been active for 655546 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
  at java.lang.Thread.sleep(Native Method)
  at java.lang.Thread.sleep(Thread.java:851)
  at com.ibm.Sleep.doSleep(Sleep.java:55)
  at com.ibm.Sleep.service(Sleep.java:35)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

WSVR0606W is the message when a thread that was seen as hung is no longer hung.

Hang detection only monitors managed threads, such as the WebContainer thread pool. 
Any native threads, or threads spawned by an application are not monitored.

Happy Debugging!

Friday, July 22, 2011


Java Serialization performance improvement

If your application or application server spends a lot of time during serialization and deserialization of java objects say due to disk caching or replication then improving the performance of java serialization pays a lot of dividend in terms of reduced CPU% and improved performance. It is worth spending the time optimizing serialization performance.

The best performance in serializing or deserializing data comes by owning the process by implementing Externalizable. Take control over writing and reading your own state. That will give the best performance.

See these articles that expound on the topic:

Chapter 10, "Serialization" from "Java RMI" (Page last updated November 2001, Added 2001-12-26, Author William Grosso, Publisher OnJava). Tips:
  1. Use transient to avoid sending data that doesn't need to be serialized.
  2. Serialization is a generic marshalling mechanism, and generic mechanisms tend to be slow.
  3. Serialization uses reflection extensively, and this also makes it slow.
  4. Serialization tends to generate many bytes even for small amounts of data.
  5. The Externalizable interface is provided to solve Serialization's performance problems.
  6. Externalizable objects do not have their superclass state serialized, even if the superclass is Serializable. This can be used to reduce the data written out during serialization.
  7. Use Serializable by default, then make classes Externalizable on a case-by-case basis to improve performance.



Record Data Replication statistics in WebSphere Application Server

If any point you need to record the DRS activity in your WebSphere Applications Server cluster you can set the JVM system property ENABLE_DRS_STATS_LOGGING  to 2 to record Data Replication Service (DRS) traffic statistics
See technote http://www-01.ibm.com/support/docview.wss?uid=swg1PK23862&wv=1

Please note, as the technote says Stats logging is enhanced by providing summary and detailed statistics on outgoing message transmission. These statistics are used for servicability and are not generally collected unless instructed by IBM service.


Migrating WebSphere Application Server

If you are considering migrating to WebSphere Application Server or upgrading your current version of WAS please take a look at the IBM® WebSphere® Application Server Migration Toolkit v3.0
  • Quickly and cost-effectively migrate to WebSphere Application Server V8, whether from a previous version of WebSphere Application Server (5.1, 6.0, 6.1, 7.0) or competitive application servers including Oracle® Weblogic Server, Oracle Application Server, and JBoss Server.
  • Simple remediation tools utilize Rational® source code scanning technology to identify and resolve known application migration issues.  All while providing developers complete control over the process.
  • Simplified developer migration experience using Application Migration Tools
  • Support for migrating JEE applications to WebSphere Application Server Version 8.0
  • Improved rule sets which scan application source code for known migration problems.  New depreciated and removed APIs, Behavior changes and   enforcements for JRE and Java EE specifications
  • Enhanced platform support – Supports the latest versions of Rational Application Developer and Eclipse.
Detailed information PDF Doc
This toolkit will reduce the traditional cost associated with migration by 40%.


Thursday, July 21, 2011


Flushing classloader leaks in your application in WebSphere Application Server

WebSphere Application Server has a hidden serviceability gem in JVM custom property com.ibm.ws.runtime.detectAppCLLeaks described in this technote

So what does setting this custom property do ?
This custom property enables the classloader leak detection feature in the application server.When the server is shutting down WAS determines the classloaders that have leaked and thereby held references to all their loaded classes and objects. If a classloader leak is detected a heapdump is triggered. You will also see a warning message in the SystemOut.log like so

[7/21/11 20:50:24:917 EDT] 00000017 ApplicationMg W   Application class loader leak:
   Local ClassPath:
   Parent: com.ibm.ws.classloader.CompoundClassLoader@218fc48[app:leak_war]
   Delegation Mode: PARENT_FIRST

This provides an excellent way to flush out all the classloader leaks in your application. I advise setting this property in development environments. Please note that not all suspect leaks are of concern particularly the ones that belong to WebSphere Application Server system applications.

Happy Debugging!

Classloader leak reference
  1. http://wasdynacache.blogspot.com/2011/05/how-to-detect-java-classloader-memory.html
  2. http://www.zeroturnaround.com/blog/rjc301/
  3. http://www.zeroturnaround.com/blog/rjc201/
  4. http://blog.dynatrace.com/2009/08/13/java-memory-problems/
  5. http://stackoverflow.com/questions/670791/common-java-memory-reference-leak-patterns
  6. http://dennis-zane.iteye.com/blog/57512
  7. http://www.mhaller.de/archives/140-Memory-leaks-et-alii.html

Sunday, July 17, 2011


Windows style Task Manager for WebSphere Application Server

Ever wondered what your application server was up to ? If you want to know what all the threads of the application server are doing or what an individual thread is upto take a look at the two JSPs provided below which simulate a windows task manager for all the threads of the Application Server

These JSPs leverge the ThreadMXBean API in the JDK to display thread information. Caution: This may not work on zOS. These JSPs allow a user to see all threads  in a table view, and then click on one of the threads to get a detailed view.

All Thread Views

Thread Detail View

Tuesday, July 12, 2011


JVM -Xtrace the holy grail of WebSphere Application Server tracing

Today's objective is to generate -Xtrace for ALL methods on the WebSphere PMI interfaces that our PMI modules implement. These PMI interfaces are called by the WAS container to compute PMI statistics. To do this we use IBM JVM's method trace facility.

Method trace is a powerful tool for tracing methods in any Java code. Method trace provides a comprehensive and detailed diagnosis of code paths inside your application, and also inside the system classes. You do not have to add any hooks or calls to existing code. You can focus on interesting code by using wild cards and filtering to control method trace. Method trace can trace method entry, exit and parameters.

Use method trace to debug and trace application code and the system classes provided with the JVM.
While method trace is powerful, it also has a cost. Application throughput is affected by method trace. The impact is proportion to the number of methods traced. Additionally, trace output is reasonably large and might require a large amount of drive space. For instance, a full method trace of a "Hello World" application is over 10 MB
You can control method trace by using the command-line option -Xtrace:. This -Xtrace is set as a JVM generic argument in the WebSphere Application Server admin console.

Please see below for the JVM generic argument for capturing all PMI Event points as JVM trace.

This will put the output in native_stderr.log.
You can then write post processors to process the trace or alternately you may be able to suck the output in IBM Health Center to visualize it. I have not tried this yet.

More details on -Xtrace here

Unfortunately this will generate the JVM trace for ALL methods of the PMI modules not just the ones on the PMI interface. This limitation we will need to live with for now.Thanks to WAS SWAT Team guru Kevin Grigorenko for sounding me out as we discussed options.

Happy Debugging!


Saturday, July 9, 2011


Uneven WebSphere Application Server Plugin load balancing and routing

If your WebSphere Application Server deployment is experiencing an imbalance in HTTP traffic routing from the WebSphere Plugin please take the following configuration and recommendations into consideration
  1. RANDOM load balancing is better than ROUND_ROBIN for load balancing. With RANDOM for all non-affinity requests, the server is “picked” randomly.The affinity requests are sent to the matching clone id. LoadBalance “weight” is not used at all in the Random case.
  2.  An affinity request is one where session affinity is established by the first request. All subsequent requests with the same sesssion ID are termed as "affinity requests". Plugin ALWAYS obeys session affinity when routing requests and routes to the appserver targeted on the first request that established affnity.
  3. In Round Robin load balancing, the first request is routed randomly. After that, the plug-in will use the next server in the list and so on… (member1, member2, member3, member4). Each time a server is “picked”, the load balance weight is decremented by one. Any server with a weight below 0 will NO longer be used for new requests. But, affinity requests will still go through.When all servers have a weight of less than 0, the plug-in will “reset” the weights to the starting values. By default affinity requests do NOT cause the plugin to decrement the weight of the server. This can lead to an uneven distribution when there are many affinity requests.
    • One options you have is to set the property IgnoreAffinityRequests="false". If IgnoreAffinityRequests="false" then the weight IS lowered by each affinity request, leading to a more balanced routing in a Round Robin environment. This can have some adverse consequences if you get a burst of non-affinity requests, they could all be targeted to the servers with the lowest weight and can cause issues with that server. To protect against this, you can defined the maxconnection setting in conjunction, which would avoid having too many non-affinity requests sent to a single server at any one time.
If you already have plugin traces or are willing to enable debug to triage this issue further WAS IBM HTTP Server guru and WebSphere plugin expert Eric Covener has the following scripts that will help analyze your plugin trace and provide a summary of your plugin HTTP routing statistics https://github.com/covener/plugin-tools

For recommended values of the plugin configuration please look at  http://www-01.ibm.com/support/docview.wss?uid=swg21318463
To understand HTTP Plugin load balancing  see http://www-01.ibm.com/support/docview.wss?uid=swg21219567
Demystifying the WebSphere Plugin http://www-01.ibm.com/support/docview.wss?uid=swg27014498&aid=1

How do you determine if this is an "affinity" request
 ) Check for SSLJSESSION cookie
2) Check for JSESSIONID cookie
3) Check for URL rewrite (jsessionid) in the URL
4) Check partitionID against partition table if any
TRACE: ws_common: websphereHandleSessionAffinity: Checking for session affinity
TRACE: ws_common: websphereHandleSessionAffinity: Checking the SSL cookie affinity: SSLJSESSION
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'SSLJSESSION'
TRACE: lib_htrequest: htrequestGetCookieValue: No cookie found for: 'SSLJSESSION'
TRACE: ws_common: websphereHandleSessionAffinity: Checking the cookie affinity: JSESSIONID
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'JSESSIONID'
TRACE: lib_htrequest: htrequestGetCookieValue: name='JSESSIONID',

TRACE: ws_common: websphereHandleSessionAffinity: Checking the JSESSIONID in cookie:
DEBUG: ws_common: websphereParseCloneID: Parsing clone ids from
TRACE: ws_common: websphereParseCloneID: Adding clone id '12n7511dh'
TRACE: ws_common: websphereParseCloneID: Returning list of clone ids

How do you know which server was picked by the WebSphere Plugin ?
The trace shows which server was “picked”. Search the trace file (http_plugin.log) for the string “picked, weight” to quickly see which servers are being chosen in what order.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
lockedServerGroupUseServer: Server GAME3Node01_member4 picked, weight 6.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
serverGroupIncrementConnectionCount: Server GAME3Node01_member4 picked,
pendingConnectionCount 1 totalConnectionsCount 1.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - DEBUG: ws_server_group:
serverGroupNextRoundRobinServer: use server GAME3Node01_member4


Friday, July 8, 2011


First Failure Data Capture for your enterprise application with WebSphere Application Server

Since WebSphere Application Server version 7, we have exposed the First Failure Data Capture (FFDC) API to customer applications. Unfortunately we have not publicized this much in the Infocenter.

One of the key points for FFDC that differentiates it from log and trace is that it is a snapshot and not  history.  Log and Trace tends to give a narrow view, but a history that can be highly valuable.  FFDC tries to provide a wider view at a point in time. 

See Javadoc 

What is FFDC and Why do I care ?
FFDC is capturing the state of the application server and/or the application when an unexpected error or exception occurs the first time. All subsequent iterations of the same error/exception are ignored. This lightweight diagnostic collector of information is designed to be ALWAYS on in a production environment. WAS code leverages FFDC extensively to collect information and state for a component when an exception or catastrophic event occurs. You can now do the same for your enterprise application.

What differentiates it from logging ?
  1. An exception has already occurred. This means that performance has been impacted and the code is in a failure path
  2. FFDC is in the category of dump or snapshot type tools. These tools aim to provide a broad view of the system at a particular point in time. Logging keeps a narrow view, over a period of time
  3. Each FFDC statement executes only once (some rare caveats). If the statement is executed again, it will know that it has already been executed, and it will simply update summary information. This means that FFDC processing can focus more on capturing all needed information, and less on performance.
  4. The FFDC infrastructure provides many points where developers can plug in code that will be called when appropriate. The class or method experiencing a failure rarely knows all of the needed context to resolve the problem. FFDC provides extensions so that more focused serviceability code can take a more holistic view and gather a much broader context.

How do I use it ?
try { 

// Application code here 

} catch (Exception e) { 

    Ffdc.log(e, myClass, myClassName+myMethodName, “lineNumber”, cde1, cde2, …) ; 

// Args:  Exception, reporting class, “sourceId”, “probeId”, context data elements
// where sourceId and probeId are any strings, but this pattern is common 
package howto_ffdc._1_simple;
import static com.ibm.ffdc.Manager.Ffdc;
// import com.ibm.ffdc.Ffdc; // Used if alternate call is done below
public class SimpleTest extends TestCase {
   protected void setUp() throws Exception {
     System.setProperty("com.ibm.ffdc.log", “/opt/IBM/WebSphere/logs/ffdc/");
    public void testWithoutFormatter(){
  try {
         // ... do work
  } catch (Exception e) {
         Ffdc.log(e, this, getClass().getName(),"24", customer);
         /* alternate if generating the parms for the call is expensive */
         // Ffdc ffdc = Ffdc.getFfdc(e, this, getClass().getName(),"24") ;
         //if (ffdc.isLoggable())
         //   MyData myData = expensiveCallToGetData() ;
         //   ffdc.log(customer, myData) ;

An excellent deep dive into FFDC by Michael Casile, WebSphere Serviceability Team Lead shows how you can use FFDC in your Java Enterprise application with WebSphere Application Server. http://slidesha.re/pWODOZ

In summary FFDC is a simple WebSphere application server facility to improve the serviceability of you enterprise software. It is low cost to implement, extensible, can grow with you (Data Collectors, Formatters, Providers, and Incident Forwarders) and the extensions do not impact core code i.e. no change needed to Ffdc.log statements to affect improved information.

... credits to Mike Casile and the WAS Serviceability team


On an intel linux platform, what are the default java parameters that are used when starting WebSphere Application Server ? java -Xmx… etc ?

You will find below ALL the user arguments used with the WAS 8 JVM. 
This was taken for a javacore triggered on WAS8 on a Windows 2003 Server 64 bit WAS install.

2CIUSERARG               -Xms50m
2CIUSERARG               -Xmx256m

How to take a javacore
wsadmin>set objectName [$AdminControl queryNames WebSphere:type=JVM,process=server1,*]

wsadmin>$AdminControl invoke $objectName dumpThreads

Look for Javacore filename:   C:\WebSphere8\m1116.12\profiles\AppSrv01\javacore.20110708.115112.4824.0001.txt

Complete List
1CIUSERARGS    UserArgs:
2CIUSERARG               -Xoptionsfile=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs\options.default
2CIUSERARG               -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry
2CIUSERARG               -XXgc:numaCommonThreadClass=java/lang/UNIXProcess$*
2CIUSERARG               -Xjcl:jclscar_26
2CIUSERARG               -Dcom.ibm.oti.vm.bootstrap.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin
2CIUSERARG               -Dsun.boot.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin
2CIUSERARG               -Djava.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin;.;C:\WebSphere8\m1116.12\lib\native\win\x86_64;C:\WebSphere8\m1116.12\bin;C:\WebSphere8\m1116.12\java\bin;C:\WebSphere8\m1116.12\java\jre\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\development\tools\bin;C:\development\nt_java\java\bin;C:\development\tools\apache-maven-3.0.3\bin;C:\development\tools\SlikSvn\bin\;%M2%;%JAVA_HOME%\bin;C:\cygwin\bin;c:\vslick\win;C:\development\tools\apache-maven-3.0.3\bin;C:\cygwin\bin;C:\development\tools\bin;
2CIUSERARG               -Djava.home=C:\WebSphere8\m1116.12\java\jre
2CIUSERARG               -Djava.ext.dirs=C:\WebSphere8\m1116.12\java\jre\lib\ext
2CIUSERARG               -Duser.dir=C:\WebSphere8\m1116.12\profiles\AppSrv01
2CIUSERARG               _j2se_j9=1119744 0x000007FFFF237160
2CIUSERARG               -Dconsole.encoding=Cp850
2CIUSERARG               -Djava.class.path=C:\WebSphere8\m1116.12\profiles\AppSrv01\properties;C:\WebSphere8\m1116.12\properties;C:\WebSphere8\m1116.12\lib\startup.jar;C:\WebSphere8\m1116.12\lib\bootstrap.jar;C:\WebSphere8\m1116.12/lib/lmproxy.jar;C:\WebSphere8\m1116.12/lib/urlprotocols.jar;C:\WebSphere8\m1116.12\java\lib\tools.jar
2CIUSERARG               -Declipse.security
2CIUSERARG               -Dwas.status.socket=2121
2CIUSERARG               -Dosgi.install.area=C:\WebSphere8\m1116.12
2CIUSERARG               -Dosgi.configuration.area=C:\WebSphere8\m1116.12\profiles\AppSrv01/servers/server1/configuration
2CIUSERARG               -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors
2CIUSERARG               -Xshareclasses:name=webspherev80,nonFatal
2CIUSERARG               -Dsun.reflect.inflationThreshold=250
2CIUSERARG               -Xbootclasspath/p:C:\WebSphere8\m1116.12/java/jre/lib/ext/ibmorb.jar;C:\WebSphere8\m1116.12/java/jre/lib/ext/ibmext.jar
2CIUSERARG               -Djava.class.path=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties;C:\WebSphere8\m1116.12/properties;C:\WebSphere8\m1116.12/lib/startup.jar;C:\WebSphere8\m1116.12/lib/bootstrap.jar;C:\WebSphere8\m1116.12/lib/jsf-nls.jar;C:\WebSphere8\m1116.12/lib/lmproxy.jar;C:\WebSphere8\m1116.12/lib/urlprotocols.jar;C:\WebSphere8\m1116.12/deploytool/itp/batchboot.jar;C:\WebSphere8\m1116.12/deploytool/itp/batch2.jar;C:\WebSphere8\m1116.12/java/lib/tools.jar
2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
2CIUSERARG               -Xms50m
2CIUSERARG               -Xmx256m
2CIUSERARG               -Xcompressedrefs
2CIUSERARG               -Xscmaxaot4M
2CIUSERARG               -Xscmx60M
2CIUSERARG               -Dws.ext.dirs=C:\WebSphere8\m1116.12/java/lib;C:\WebSphere8\m1116.12\profiles\AppSrv01/classes;C:\WebSphere8\m1116.12/classes;C:\WebSphere8\m1116.12/lib;C:\WebSphere8\m1116.12/installedChannels;C:\WebSphere8\m1116.12/lib/ext;C:\WebSphere8\m1116.12/web/help;C:\WebSphere8\m1116.12/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
2CIUSERARG               -Dderby.system.home=C:\WebSphere8\m1116.12/derby
2CIUSERARG               -Dcom.ibm.itp.location=C:\WebSphere8\m1116.12/bin
2CIUSERARG               -Djava.util.logging.configureByServer=true
2CIUSERARG               -Duser.install.root=C:\WebSphere8\m1116.12\profiles\AppSrv01
2CIUSERARG               -Djava.ext.dirs=C:\WebSphere8\m1116.12/tivoli/tam;C:\WebSphere8\m1116.12/java/jre/lib/ext
2CIUSERARG               -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder
2CIUSERARG               -Dpython.cachedir=C:\WebSphere8\m1116.12\profiles\AppSrv01/temp/cachedir
2CIUSERARG               -Dwas.install.root=C:\WebSphere8\m1116.12
2CIUSERARG               -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
2CIUSERARG               -Dserver.root=C:\WebSphere8\m1116.12\profiles\AppSrv01
2CIUSERARG               -Dcom.ibm.security.jgss.debug=off
2CIUSERARG               -Dcom.ibm.security.krb5.Krb5Debug=off
2CIUSERARG               -Djava.library.path=C:\WebSphere8\m1116.12/lib/native/win/x86_64/;C:\WebSphere8\m1116.12\java\jre\bin\default;C:\WebSphere8\m1116.12\java\jre\bin;.;C:\WebSphere8\m1116.12\lib\native\win\x86_64;C:\WebSphere8\m1116.12\bin;C:\WebSphere8\m1116.12\java\bin;C:\WebSphere8\m1116.12\java\jre\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\development\tools\bin;C:\development\nt_java\java\bin;C:\development\tools\apache-maven-3.0.3\bin;C:\development\tools\SlikSvn\bin\;%M2%;%JAVA_HOME%\bin;C:\cygwin\bin;c:\vslick\win;C:\development\tools\apache-maven-3.0.3\bin;C:\cygwin\bin;C:\development\tools\bin;
2CIUSERARG               -Djava.endorsed.dirs=C:\WebSphere8\m1116.12/endorsed_apis;C:\WebSphere8\m1116.12/java/jre/lib/endorsed;C:\WebSphere8\m1116.12\endorsed_apis;C:\WebSphere8\m1116.12\java\jre\lib\endorsed
2CIUSERARG               -Djava.security.auth.login.config=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties/wsjaas.conf
2CIUSERARG               -Djava.security.policy=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties/server.policy
2CIUSERARG               -Dsun.java.command=com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer C:\WebSphere8\m1116.12\profiles\AppSrv01\config smitaNode03Cell smitaNode03 server1
2CIUSERARG               -Dsun.java.launcher=SUN_STANDARD
2CIUSERARG               _port_library 0x000007FFFF235F20
2CIUSERARG               _bfu_java 0x000007FFFF238230
2CIUSERARG               _org.apache.harmony.vmi.portlib 0x0000000000151140

Wednesday, July 6, 2011


Tuning WebSphere Application Server Dynamic Cache Service (Dynacache) and Data Replication Service (DRS) for replication

If you want to improve the consistency & performance of your WebSphere Application Server DistributedMap replicated cache please follow these best practices

Dynacache Tuning
1. Change replication type of the replication domain to either NOT_SHARED or PUSH depending on your replication needs. NOT_SHARED scales better than PUSH because it only replicates invalidations not cache data.
2. Ensure that the API calls that are putting objects in the cache are NOT overriding the sharing type of the replication domain
3. Set the following as JVM custom properties or as properties on your servlet/object cache instance to reduce redundant DRS traffic, improve performance & force creation of cache instances
  • com.ibm.ws.cache.CacheConfig.filterLRUInvalidation = true
  • com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation = true   
  • com.ibm.ws.cache.CacheConfig.filterInactivityInvalidation= true                  * requires a relatively recent APAR. check with support
  • com.ibm.ws.cache.CacheConfig.ignoreValueInInvalidationEvent = true
  • com.ibm.ws.cache.CacheConfig.useServerClassLoader = true
  • com.ibm.ws.cache.CacheConfig.createCacheAtServerStartup = true            * since WAS v7
4. [Optional] In the PUSH replication mode to eliminate the bootstrap of cache data,  the application code that gets an instance of distributedMap should call distributedMap.setDRSBootstrap(false). Please note this will prevent the synchronization of cache instance across JVMs in the replication domain at server startup.

Data Replication Service (DRS) Tuning
Configure DRS to use its own threadpool     *default since

High Availability Manager (HAManager) & Data Consistency Services (DCS) Tuning 

HAManager Protocol Versions
Bump up the lower level wire format (IBM_CS_WIRE_FORMAT_VERSION core group custom property) and the high availability manager (IBM_CS_HAM_PROTOCOL_VERSION core group custom property)  coregroup protocol versions to the latest one supported. See the following article in the Infocenter. New protocol versions are added to the product if new messages, or new algorithms are required to support new product features, or to improve core group performance. Please note that because the new messages, or new algorithm might not be compatible with the existing messages or algorithm, a new protocol might not be able to interoperate with the old version of the protocol.

Please also revisit my earlier FAQ  and technote on cache replication 
Please also read this olden goldie on Dynacache and DRS Best Practices.


Tuesday, July 5, 2011


Network issues

If you are having consistent network disruption issues with the JVM like UnknownHostExceptions, hostname DNS lookup failures etc you can try setting the following JVM option on the IBM JREs
to print all the activities happening over the net package giving you sufficient information to find out what is happening.

JVM networking APIs are a thin veneer over the OS APIs. There are a couple of flags that control network JVM settings  i.e. preferring IPV4 or IPV6, caching the result of a local host lookup to avoid trips to the DNS & caching of lookup results. Generally speaking, all the interesting stuff happens in the OS.

If you are having network issues with HAManager and DCS WebSphere components you can configure the  IBM_CS_IP_REFRESH_MINUTES custom property that controls how frequently the core group IP cache is cleared. By default the value is 60 seconds.

Java Networking Custom Properties
java.net.preferIPv4Stack (default: false)
If IPv6 is available on the operating system the underlying native socket will be an IPv6 socket. This allows Java(tm) applications to connect too, and accept connections from, both IPv4 and IPv6 hosts.If an application has a preference to only use IPv4 sockets then this property can be set to true. The implication is that the application will not be able to communicate with IPv6 hosts.

java.net.preferIPv6Addresses (default: false)
If IPv6 is available on the operating system the default preference is to prefer an IPv4-mapped address over an IPv6 address. This is for backward compatibility reasons - for example applications that depend on access to an IPv4 only service or applications that depend on the %d.%d.%d.%d representation of an IP address. This property can be set to try to change the preferences to use IPv6 addresses over IPv4 addresses. This allows applications to be tested and deployed in environments where the application is expected to connect to IPv6 services.

networkaddress.cache.ttl (default: -1)
Specified in java.security to indicate the caching policy for successful name lookups from the name service.. The value is specified as as integer to indicate the number of seconds to cache the successful lookup.A value of -1 indicates "cache forever".

networkaddress.cache.negative.ttl (default: 10)
Specified in java.security to indicate the caching policy for un-successful name lookups from the name service.. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups.A value of 0 indicates "never cache". A value of -1 indicates "cache forever".

Friday, July 1, 2011


WebSphere Application Server stopped automatically after network cable was unplugged

If you find that your application server stopped automatically after a network cable was unplugged or some sort of adverse network event, please look for the following stack trace in the SystemOut.log

[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O Panic:component requested panic from isAlive
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O java.lang.RuntimeException: emergencyShutdown called:
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.runtime.component.ServerImpl.emergencyShutdown(ServerImpl.java:633)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.runtime.RuntimeProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.coordinator.impl.JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.HAGroupImpl.doIsAlive(HAGroupImpl.java:866)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.HAGroupImpl$HAGroupUserCallback.doCallback(HAGroupImpl.java:1364)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.Worker.run(Worker.java:64)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)

Look for any  HMGR informational messages like HMGR0130I that indicate why the JVM was terminated.
If you using SIBus read this technote http://www-01.ibm.com/support/docview.wss?uid=swg21496900

From technote
"WebSphere Application Server uses database tables to store Java Message Service (JMS) messages. Anytime the messaging engine loses its connection to the data store, the JVM shuts down in an orderly fashion to prevent data loss. If the messaging engine were to continue to run and accept work, results might be unpredictable and your messaging engine might be in an inconsistent state when the data store connection is restored.

If you continually see this problem, consider modifying the Retry Interval for your data source. This value specifies the length of time, in seconds, that the application server waits before retrying to make a connection if the initial attempt fails. By default, this value is 0. If you set the value to 3, the connection is retried instead of holding onto a bad connection. You can find more information on datasource properties in the WebSphere Application Server data source properties topic within the WebSphere Application Server Information Center. "

Happy Debugging :)



Need a cheatsheat or guide to setting up ITCAM with WebSphere Application Server (WAS) ?

If you wanted one place to review ALL easy to read documentation for setting up ITCAM with WebSphere Application Server please go to -

The ITCAM for AD Deployment Guide is a wiki available at  https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/ITCAM+for+Application+Diagnostics+Deployment+Guide

This wiki guides you through the process of successfully deploying and maintaining ITCAM for Application Diagnostics; it includes links to the Application Diagnostics InfoCenter, and to some product technotes.

If you have time please go through this video for a deep-dive  into ITCAM for WAS

Happy monitoring & debuggging !



December 2006   September 2008   January 2009   February 2009   March 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   October 2010   January 2011   February 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   September 2012   October 2012   November 2012   January 2013   May 2013   June 2013   July 2013   September 2013   October 2013   June 2014   August 2014  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]