All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Monday, October 18, 2010

WebSphere OutOfMemory Errors

If your WAS deployment is experiencing OOM issues you can setup the JVM to produce a system dump when the OOM event occurs. This system dump can later be jextracted and sucked into Eclipse Memory Analyzer to do offline analysis.
There are a variety of ways to obtain a dump from the IBM software development kit and various formats for the dump produced. It important to understand the content of each of the dumps so that the correct one can be selected for analysis of a given problem. In essence there are three forms of dump:

A system dump which is a complete dump of all the information in a process. The Dump Analyzer requires this form of dump to perform analysis.
A heap dump which gives a more terse view of the objects in the heap but does not include thread and monitor information. This form of dump is of use to other tools for heap analysis.
A summary dump sometimes called a javacore file. This is a human readable dump designed to summarise the state of the process in high level terms; the threads, monitors etc.

Dumps are produced by the JVM either on demand (a signal from the user) or on event (something happening within the VM). When the JVM starts it registers a number of event handlers which cause dumps to be generated for a default set of events. In the case of an Out of Memory event a heap dump is generated, for a user signal a javacore is generated and for a JVM crash a system dump is generated. For detailed information on setting the JVM options that control dump production.

To generate a system dump automatically on an OOM you will need to set the following JVM generic argument:-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=nodumps+exclusive+prepwalk

In order to generate java core dump, system core dump, heap dump and a snap dump at user signal, the dump agents must be configured through JVM options as follows.-Xdump:java+heap+system+snap:events=user

You can have multiple -Xdump options on the command line.

-Xdump agents are always merged internally by the JVM, as long as none of the agent settings conflict with each other.

-Xdump:java+heap+system+snap:events=user -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=nodumps+exclusive+prepwalk

-X options are specified as generic JVM arguments on WebSphere Application Server 6.1 as follows:
In the Administration Console select Servers

Select Application Servers

Click on the name of your server

In the Server Infrastructure section, expand Java and Process Management and select Process Definition > Java Virtual Machine

Scroll down and locate the textbox for Generic JVM arguments.

Please also see Crash on AIX produces no core or a truncated core to prevent a truncated core from being generated on AIX.

Please note these events ONLY work for IBM Java5 and Java6 JVMs.

Happy debugging :-)

posted by Rohit Kelapure # 6:23 PM 0 Comments

Sunday, October 17, 2010

WebSphere Memory leak Detection and Protection

Tomcat provides excellent memory leak protection and detection features. Starting with tomcat 6.0.25, the manager webapp has a new "Find Leaks" button. When triggered, it displays a list of webapps (their context path) that have been stopped (this includes undeployed and redeployed ones) but whose classloader failed to be GCed.There are two parts to the new memory leak features*-

The first part handles memory leak prevention and is implemented as a life-cycle listener, the JreMemoryLeakPreventionListener. This listener prevents memory leaks by calling various parts of the Java API that are known to retain a reference to the current context class loader. If a web application is the first code to call one of these standard Java APIs, the web application class loader will be pinned in memory which will cause a memory leak. By making sure that Tomcat code makes the calls first, the memory leaks are prevented. If you want to know the gritty details of what this listener actually does, the src is pretty well commented.
The second part of the new features handles detection. This code executes when a web application is stopped, un-deployed or reloaded. It checks, via a combination of standard API calls and some reflection tricks, for known causes of memory leaks and - where it can - fixes them. These checks are implemented in the WebappClassLoader. As always with Tomcat, if you want the details you can look at the WebappClassLoader.java. Start with the clearReferences() method.

All JEE application servers will benefit greatly from these features. If your JEE application and application server suffers from these memory leaks consider implementing this leak prevention and detection function following the steps below -

For leak prevention modify the JreMemoryLeakPreventionListener to implement javax.servlet.ServletContextListener and leverage the servlet spec. mandated ServletContainer lifecycle events instead of the proprietary org.apache.catalina.LifecycleEvent events passed to the LifecycleListener interface.
Leak detection is more complicated. You can take two approaches to skinning this cat.

You will need first encapsulate and later port the code in org.apache.catalina.loader.WebappClassLoader.clearReferences() to a simple delegation classloader that plugs in after the webappclassloader i.e. all classes will first be loaded from the custom classloader that will delegate up to the webappclassloader. This way you do bookkeeping on the classes loaded in the custom classloader. Later when the application stops call the clearReferences method that operates on the loaded classes.
Alternately you will need to figure out a way to access/list ALL the classes (Class objects) loaded by your web application's classloader. With this list in hand you will call clearReferences* methods on those classes once the application stops. //org.apache.catalina.loader.WebappClassLoader.clearReferences( protected void clearReferences() { clearReferencesJdbc(); clearReferencesThreads(); clearReferencesThreadLocals(); clearReferencesRmiTargets(); if(ENABLE_CLEAR_REFERENCES) clearReferencesStaticFinal(); IntrospectionUtils.clear(); if(clearReferencesLogFactoryRelease) LogFactory.release(this); clearReferencesResourceBundles(); Introspector.flushCaches(); } //Custom Leak Detection Classloader to extend the abstract Classloader //Classloader API public abstract class ClassLoader { public Class loadClass(String name); protected Class defineClass(byte[] b); public URL getResource(String name); public Enumeration getResources(String name); public ClassLoader getParent() }

Please let me know if these approaches worked for you.

* From http://www.tomcatexpert.com/blog/2010/04/06/tomcats-new-memory-leak-prevention-and-detection

posted by Rohit Kelapure # 10:12 AM 0 Comments

All Things WebSphere

Monday, October 18, 2010

WebSphere OutOfMemory Errors

Sunday, October 17, 2010

WebSphere Memory leak Detection and Protection

Archives