WebSphere Application Server JVM Resiliency

When hung thread messages do appear in the log  like so
[3/8/12 11:39:07:481 MST] 00000016 ThreadMonitor W   WSVR0605W: Thread "SipContainer : 13" (0000002f) has been active for 678,382 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung. You can configure the JVM to run a script that will kill the JVM


-Xtrace:trigger=method{com/ibm/ws/runtime/component/ThreadMonitorImpl.threadIsHung,javadump,abort,2,1}

This will generate a javacore after two hung threads and terminate the JVM on the second occurrence of the hung thread.

• Once the JVM aborts, the nodeagent if configured for the Automatic Restart monitoring policy will spawn a new server JVM.Here's the setting for the NA to restart a dead process: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=matt&product=was-nd-mp&topic=urun_rmonitorpol Automatic Restart = true, which is the default for distributed
• Alternately you can also write a watchdog os script to start the server again see http://www.ibm.com/developerworks/websphere/techjournal/1001_webcon/1001_webcon.html

You can write a range of filters that capture JVM events and run a script
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/com.ibm.java.doc.diagnostics.60/diag/tools/dumpagents_options.html
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/com.ibm.java.doc.diagnostics.60/diag/tools/dump_agents.html
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/com.ibm.java.doc.diagnostics.60/diag/tools/dumpagents_tool.html
http://wasdynacache.blogspot.com/2011/12/jvm-xdump-options-for-scenario-based.html
http://www.ibm.com/developerworks/java/library/j-memoryanalyzer/

Thanks to Kevin Grigorenko for contributing. check his blog out at
https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/