Generating system dumps and thread dumps automatically

Generating a system dump instead of a heap dump automatically on an OOM
To generate a system dump as default on an OOM error or when the JVM receives the SIGQUIT.
Please set the following JVM generic custom property
-Xdump:java+system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,priority=999,request=exclusive+compact+prepwalk
Please also ensure that there is adequate space for the system dump by following this technote .

Generating a thread dump automatically on a thread hang
To configure a hang detection policy that dumps a javacore when a thread is hung by setting the following property

1. From the administrative console, click Servers > Application Servers > server_name OR  System Administration > Deployment Manager  or  System Administration > NodeAgent > node_agent_name
2. Under Server Infrastructure, click Administration > Custom Properties
3. Click New.
4. Add the following property:

Name com.ibm.websphere.threadmonitor.dump.java
Value true

This causes  a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.

 Table of where to set a JVM Custom property

Application server	Servers > Server Types > WebSphere application servers > server_name > Java and Process Management > Process Definition > Java Virtual Machine > Custom Properties
Deployment manager	System Administration > Deployment Manager > Java and Process Management > Process definition > > Java Virtual Machine > Custom Properties
Node agent	System Administration >Node Agent > nodeagent > Java and Process Management > Process definition > Java Virtual Machine > Custom Properties

References:
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Ftools%2Fdiag_collector.html

http:http//publib.boulder.ibm.com/infocenter/javasdk v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Ftools%2Fdiag_collector.html