All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Tuesday, July 24, 2012

RichFaces 4 and WebSphere Critical bugs for scalability

If you are deploying a RichFaces 4.2.x application in production on WebSphere Application Server 8.x under heavy load you will need fixes for [1] and [2]. Please also enable annotation scanning. see [3]

posted by Rohit Kelapure # 11:49 AM 0 Comments

Monday, July 23, 2012

Serviceability Gem: WAS Heap Dump Analyzer Tool for Classloader memory leaks

The Eclipse Memory Analyzer tool downloaded from the IBM support assistant has WebSphere specific extensions to detect and analyze Classloader leaks.

To run this WAS Heap Dump Analyzer tool standalone to detect and provide root cause for classloader leaks run the following command

java -cp C:\development\tools\isa142\Administrator\applications\eclipse\plugins\com.ibm.mat64.win_1.1.0.64-1\eclipse\plugins\com.ibm.java.diagnostics.memory.analyzer.was_1.1.0.201108240003.jar com.ibm.ws.heapdump.Main

washda: expected filename
WASHDA v0.9.40
Usage: java -Xmx[MAXMEM] -jar washda.jar [OPTION]... FILE

WASHDA v0.9.40

-allparents Prints all parent references during leak analysis

-analyzealarms Performs alarm manager analysis

-analyzecls Performs class loader analysis

-analyzeclleaks Performs class loader leak analysis (default)

-analyzeejbc Performs EJB container analysis

-analyzeejbclocks Performs EJB container Entity Option A lock analysis

-analyzeleaks PATTERN Performs leak analysis on the specified object or on

objects matching the specified pattern

-analyzememleaks Performs memory leak analysis

-analyzeorb Performs ORB analysis

-config FILE Reads configuration from the specified file

-console Starts the interactive console

-disablehashcodes Disables object hashcodes to reduce memory use

-disablesizes Disables object size data to reduce memory use

-gui Open the GUI window

-help Prints this help text and exit

-lowmemory Reduces memory consumption (increases processing time)

See also -disablehashcodes and -disablesizes

-maxdepth NUM Stops leak analysis if a parent reference chain

exceeds the specified number during leak analysis

-maxobjects NUM Stops leak analysis if the number of traversed objects

exceeds the specified number during leak analysis

-showconfig Prints the configuration file and exits

Sample run of the command

Opening c:\temp\730409\heapdump.20120312.090607.22606.0002.phd with WASHDA v0.9.40

9068853 objects, 26956 classes, 20646253 references

Class loader summary (762 loaders, 25714 classes)

--------------------

457 sun/reflect/DelegatingClassLoader

213 com/ibm/ws/classloader/CompoundClassLoader

.....

This tool is provided as-is.
Happy Debugging!

posted by Rohit Kelapure # 7:22 AM 0 Comments

Thursday, July 19, 2012

Abstract Syntax Tree manipulation with WebSphere Application Server

Recently I had the chance to work with Abstract Syntax Trees to improve the structure of the servlet code generated by the Jasper compiler. All the eclipse libraries needed for AST manipulation are available in WAS_HOME/plugins directory. Here are some of the articles that I found helpful when writing the servlet code AST manipulator
http://www.eclipse.org/jdt/ui/astview/index.php
http://www.eclipse.org/articles/article.php?file=Article-JavaCodeManipulation_AST/index.html
http://www.ibm.com/developerworks/opensource/library/os-ast/
http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FAST.html
http://publib.boulder.ibm.com/infocenter/iadthelp/v6r0/index.jsp?topic=/org.eclipse.jdt.doc.isv/reference/api/org/eclipse/jdt/core/dom/AST.html
http://stackoverflow.com/questions/9105019/how-to-find-an-element-given-its-source-position-in-a-java-ast
http://www.eclipse.org/forums/index.php/t/229105/
http://www.eclipse.org/articles/article.php?file=Article-JavaCodeManipulation_AST/index.html
http://www.eclipse.org/forums/index.php/t/83151/

posted by Rohit Kelapure # 10:25 AM 0 Comments

How WebSphere and WebLogic deal with stuck and hung threads

To understand how WebLogic deals with stuck threads read http://adfhowto.blogspot.com/2011/11/dealing-with-weblogic-stuck-threads.html

WebSphere Application Server has similar capabilities to detect hung threads; however; WebLogic goes one step further for remediation . It restarts the server when all threads are stuck or a certain # of threads beyond a threshold are stuck.

Starting WebSphere Application Server 8.5, the new Intelligent health management feature supports restarting a server that has a certain configurable threshold of stuck threads. That is, you can configure a health policy which says "when the number of hung threads exceeds some threshold, then perform some actions", and one of those actions is to restart the server. We are better than WebLogic because WAS restarts the server in a smart way in order to avoid an outage; that is, we know how many cluster members are running.

posted by Rohit Kelapure # 5:42 AM 0 Comments

Wednesday, July 18, 2012

ClassLoader memory leak detection in WebSphere Application Server 8.5

WebSphere Application Server provides support for classloader memory leak detection, protection and fixing in its latest release 8.5. This feature was inspired from Tomcat and will help application developers and ISVs debug and auto-fix memory leaks, a pervasive issue on applications server JVMs.

Read all about Memory leaks in Java Platform, Enterprise Edition applications
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.nd.doc/ae/ctrb_memleakdetection.html

How to configure the classloader memory leak policy here
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=/com.ibm.websphere.nd.doc/ae/ttrb_configmemleak.html

Classloader leak detection in websphere application server from kelapure

posted by Rohit Kelapure # 8:45 AM 0 Comments

Serviceability gem: NullPointerExceptions in WAS Web Container (com.ibm.ws.webcontainer.srt.SRTServletRequest)

In my ongoing quest to bring you the serviceability gems in WebSphere Application Server, ladies and gentlemen, I present to you WebContainer custom property

Name: com.ibm.ws.webcontainer.checkrequestobjectuse
Value: true

This custom property helps determine if WAS internal components or the application is using the wrong request object on the thread which can lead to NullPointerException in WebContainer code.
Set this property along with the trace string com.ibm.ws.webcontainer*=all

If there is a problem with request object reuse or mismatch on the same thread then the following error message is expected to be seen in the log. You can grep for these strings
ERROR: Wrong request object in use on Thread. Object Expected: X, Found: Y
Wrong request object in use on Thread. IllegalStateException

To specify Web container custom properties:

In the administrative console click Servers > Application Servers > server_name > Web Container settings > Web Container
Under Additional Properties select Custom Properties.
On the Custom Properties page, click New.
On the settings page, enter the name of the custom property that you want to configure in the Name field and the value that you want to set it to in the Value field.
Click Apply or OK.
Click Save on the console task bar to save your configuration changes.
Restart the server.

WebContainer mustgather doc when there are problems in the webcontainer

http://www-01.ibm.com/support/docview.wss?uid=swg21384592

posted by Rohit Kelapure # 8:19 AM 0 Comments

WAS Performance Redbooks and other useful resources for tuning websphere

The WebSphere Application Server roadmaps on developerWorks at http://www.ibm.com/developerworks/websphere/zones/was/roadmaps/ list Redbooks, information center topics, articles, books, etc. that customers might find helpful. At this time, there are roadmaps for WAS v6.1 - v8.0. We are working on the WAS v8.5 roadmap and intend to publish it soon.

Foe example the WAS v8.0 roadmap has the following info under "Tuning" at http://www.ibm.com/developerworks/websphere/zones/was/roadmaps/was8_roadmap.html#TUNIN546

Understand and analyze performance data, troubleshoot performance problems, develop applications that perform better, and tune many aspects of the application serving environment.

Performance This IBM Education Assistant module provides an overview of performance monitoring and tuning.

WebSphere Application Server Performance information This WebSphere Application Server Performance website provides a centralized access to helpful performance reports, tools, and downloads.

Designing and Coding Applications for Performance and Scalability in WebSphere Application Server This Redbooks publication discusses performance and scalability considerations for developing and coding WebSphere Application Server applications.

Performance Monitoring and Best Practices for WebSphere on z/OS This Redbooks publication discusses how to set up an environment that is tuned to meet best performance and at the same can be monitored to catch eventual performance bottlenecks.

WebSphere Application Server, Network Deployment - Tuning performance This information center section discusses how to improve performance, including tools and tips for tuning servers and various application types.

Garbage collection in WebSphere Application Server V8, Part 1: Generational as the new default policy In IBM WebSphere Application Server Version 8, the default garbage collection policy uses both generational and concurrent collection strategies. This developerWorks article describes the garbage collection policies available in the IBM Java virtual machine (JVM) and provides a guide for configuring the new default policy.

Garbage collection in WebSphere Application Server V8, Part 2: Balanced garbage collection as a new option IBM WebSphere Application Server Version 8 introduces the new "balanced" garbage collection policy. This technology is optimized for large heaps and aims to even out pause times that are associated with garbage collection. The information in this developerWorks helps you determine whether the balanced collector is a good fit for your applications, and explains how to tune it for maximum performance.

Case study: Tuning WebSphere Application Server V7 and V8 for Performance IBM WebSphere Application Server supports an ever-growing range of applications, each with their own unique set of features, requirements, and services. Just as no two applications use an application server in exactly the same way, no single set of tuning parameters likely provides the best performance for any two different applications. Most applications generally realize some performance improvement from tuning in three core areas: the JVM, thread pools, and connection pools. This developerWorks article uses the Apache DayTrader Peformance Benchmark Sample application to demonstrate what you can tune and how to tune it, depending on the major server components and resources that your application uses.

posted by Rohit Kelapure # 6:47 AM 0 Comments

Excellent resources on JSF 2 Tuning and other miscellaneous tips

Trace strings to enable RichFaces logging in WAS org.richfaces.log.Resource=all and similarly for other parts of RichFaces

RESOURCE("Resource"),

RENDERKIT("Renderkit"),

CONFIG("Config"),

CONNECTION("Connection"),

APPLICATION("Application"),

CACHE("Cache"),

CONTEXT("Context"),

COMPONENTS("Components"),

WEBAPP("Webapp"),

UTIL("Util"),

MODEL("Model");

RichFaces 4 annotations not being recognized in WebSphere Application Server
see https://community.jboss.org/message/747005#747005

WebSphere application Server keeps a file called WAS_HOME/properties/amm.filter.properties that excludes the scanning of certain packages for annotations for performance reasons. see http://pic.dhe.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.nd.doc/info/ae/ae/trun_app_reduce_annot.html If you want RichFaces 4 and Richfaces-bootstrap to work correctly with WAS please remove org.richfaces from the list of Ignore-Scanning-Packages

Excellent resources for JSF tuning
http://blog.eisele.net/2009/10/jsf-facelets-myfaces-and-richfaces.html
http://javaspecialist.wordpress.com/2010/05/30/performance-tuning-of-seam-jsf-richfaces-for-webapps/
http://wiki.apache.org/myfaces/Performance
http://planet.jboss.org/post/performance_tuning_of_seam_jsf_richfaces_for_webapps
These contain all the web context parameters and more to improve the performance of your JSF web application.

posted by Rohit Kelapure # 6:21 AM 0 Comments

Monday, July 16, 2012

Low cost pre-emptive testing of session persistence with WebSphere Application Server

Testing session persistence is often painful because it requires the setup of a cluster. Using the element in the web.xml you can mark your web module as distributable like so

An easy way to test your application for persistence is to set the distributable element in the web.xml and enable com.ibm.ws.session*=all trace. If objects aren't serializable, you will see error messages in the SystemOut.log like so

IllegalArgumentException("Attribute with name " + pName + " is not java.io.Serializable. All attributes stored in session must be Serializable when web module is marked as distributable");

You proceed to test session persistence with a single server using persistence to database or a JDBCStore(in tomcat).

Errors and NPEs in HTTPSession persistence can be flushed out by making sure that that Session Manager is configured correctly. http://www.redbooks.ibm.com/redpapers/pdfs/redp4580.pdf is an excellent resource for Session Management.

Please refer to http://wiki.metawerx.net/wiki/Web.xml.Distributable for some basic guidelines when coding with HTTPSessions
1. All session attributes must be serializable (must implement java.io.Serializable). Most native java objects, including data types such as Strings and Integers implement this functionality.
2. After changing any objects which are stored in the session, HttpSession.setAttribute() must be called to inform the session replicator that the session has changed.
3. Ideally, sessions should be kept relatively small, to ensure less network traffic between the each clustered VM.

posted by Rohit Kelapure # 3:16 PM 0 Comments

Wednesday, July 11, 2012

How to monitor the contents of your JDBC connections from the connection pool to the back end database

Diagnostic trace which watches for any connections held in-use for more than 10 seconds:
ConnLeakLogic=all
The diagnostic trace WAS.j2c implicitly includes the above trace string.
Search trace for "Connection Leak Logic Information"

Using wsadmin: showPoolContents
Only available when WAS.j2c trace is enabled. Apply PK83902 (http://www-01.ibm.com/support/docview.wss?uid=swg1PK83902) to make showPoolContents available with just ConnLeakLogic=all.

AdminControl.invoke(AdminControl.queryNames("*:name=$DATASOURCE_DISPLAY_NAME,process=$SERVER_NAME,*"), "showPoolContents")
Note: use the display name of the data source, not the JNDI name.

Standalone Java program calling the JMX bean: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.nd.doc/info/ae/ae/rdat_adexjava.html

posted by Rohit Kelapure # 1:45 PM 0 Comments

Customizing WebSphere application configuration for production and test environments

There are often situations where an application on WAS needs to be configured differently in production and test environments. There are multiple ways to configure an application without modifying the packaged war/ear file depending on the environment.

A common customer question on configuration of the application goes like this ...
We want to load our web application twice in the same WAS server instance ... once for production and once for test. The web app has a configuration file that is different for prod vs test. Our current "technique" is we open the jar in each deployment to update a property file with the fully qualified path/name of the config file. Ick. There must be a better way. I see "custom properties" on a per-application basis. I could add the property there, then add code to read the property in our code. I see all kinds of examples of doing this for JVM properties, but no examples for web-app configuration So, looking for advice, sample code, ...

4 ways of achieving this...

Using the Contexts and Dependency Injection support for alternatives and/or producers or with third party extensions like CODI in WAS 8.
Using a resource environment as explained here http://www.ibm.com/developerworks/websphere/library/techarticles/0611_totapally/0611_totapally.html
Using WebSphere Environment Variables is also another way to configure your application in an environment agnostic fashion. http://pic.dhe.ibm.com/infocenter/wasinfo/fep/topic/com.ibm.websphere.soafep.multiplatform.doc/info/ae/ae/tcws_variable.html
A property file can be made accessible to Java EE application components in a standards-compliant way by configuring it as a URL resource in the application server and making a reference to it available for lookup in the JNDI namespace. http://www.ibm.com/developerworks/websphere/library/techarticles/0502_botzum/0502_botzum.html

This topic is also covered in "IBM WebSphere Deployment and Advanced Configuration" by Barcia, et a,..

posted by Rohit Kelapure # 1:33 PM 0 Comments