All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Wednesday, December 28, 2011

IBM JVM and JDK -Xdump and -Xtrace options for scenario based problem determination

There are multiple ways of configuring the IBM JDK shipped by WebSphere Application Server to generate WebSphere and JDK trace in situations where you have no control of the application or the server runtime.

Scenario 1:
In a long running server when an exception occurs say a javax.transaction.RollbackException, we need to enable a particular WebSphere runtime trace for a period of 1 hour to collect pertinent logs. This allows for collection for logs ONLY and IF a critical incident occurs in long run and production deployment scenarios.

Using the JVM dump engine ability to run command lines based on events, you could trigger a command line on, say, the uncaught javax.transaction.RollbackException:
-Xdump:tool:events=uncaught.filter="javax/transaction/RollbackException",exec="command line to run"
Where we could then use wsadmin to change the trace settings:
exec="sh /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -lang jython -f setWASTrace.py"

For (setWASTrace.py) scripting with WebSphere Application Server see this blog post.

Scenario 2:
Print a thread stack trace whenever the method com.ibm.ws.util.ThreadPool$Worker.setStartTime method is called
-Xtrace:print=mt,methods={com/ibm/ws/util/ThreadPool$Worker.setStar tTime},trigger=method{com/ibm/ws/util/ThreadPool$Worker.setStartTime,jstack trace}

Scenario 3:
Print JVM entry and exit trace in clear text whenever any code is executed in the sun.net.www.protocol.http and sun.net.www.http packages
-Xtrace:methods={sun/net/www/protocol/http*,sun/net/www/http*},print=mt

Scenario 4
Take Javacores when CPU Starvation is detected. This will write to a file named cpu_starvation%pid%.txt in the current working directory of the JVM (i.e. the same directory as the javacores) when CPUStarvationDetectedException is thrown

-Xdump:tool:events=throw,range=1..0,filter=com/ibm/loglistener/CPUStarvationDetectedException,exec="cmd /c c:\cpu_starvation.bat end %pid"

Scenario 5
Take java stack traces when multiple methods are fired. This is useful to determine the root cause of memory leaks if one method allocates an object (i.e. puts in a map) and other method releases the resource (removes the object from the map)

-Xtrace:trigger=method{com/ibm/ws/webbeans/services/JCDIComponentImpl.storeCreationalContext,jstacktrace} -Xtrace:trigger=method{com/ibm/ws/webbeans/services/JCDIComponentImpl.releaseCreationalContext,jstacktrace}

Labels: debugging

posted by Rohit Kelapure # 4:01 PM 0 Comments

Tuesday, December 20, 2011

Perl script for finding invalidations in Dynacache Trace

Dynatrace.pl - Tool for finding invalidations in Dynacache Trace

This tool searches Dynacache Trace for invalidations and generates two CSV files.

Pre-req

This tool requires Perl. ActivePerl can be installed on Windows. The DateTime package is required.

The log to be parsed needs to contain Dynacache Tracing.

Usage

C:\>dynatrace.pl

usage: C:\dynatrace.pl [OPTIONS] FILE

Trace should contain this trace:
com.ibm.ws.cache.*=all:com.ibm.ws.drs.*=all

Tool looks for this line:
Cache.remove() cacheName=baseCache id=Customers cause=1 source=2

Filtering Options:

-i | --instance CACHE_INSTANCE
Tool uses the contains reg ex

-c | --cause CAUSE
Use either number of string
   0    UNSET
   1    DIRECT
   2    LRU
   3    TIMEOUT
   6    INACTIVITY
   7    DISK_GARBAGE_COLLECTOR
   8    DISK_OVERFLOW

-s | --source SOURCE
   1    MEMORY
   2    REMOTE
   3    DISK
   4    NOOP
   5    LOCAL

Output:
- FILE.inv.csv
- FILE.inv.nonum.csv

FILE.inv.nonum.csv can be used for grouping

Example Usage

C:\>dynatrace.pl --source REMOTE trace.log
-----> Grepping file: trace.log
-----> Filtering by source=REMOTE
-----> Processing: trace.log
-----> Writing trace.inv.csv
-----> Writing trace.inv.nonum.csv
-----> Log file format: Default - short
-----> Process trace.log file. Found 3443 invalidations
-----> Done

Download

Download the tool from here ---------> Dynatrace.pl

Labels: debugging

posted by Rohit Kelapure # 6:40 PM 0 Comments

Oracle’s Newly Unveiled WebLogic 12c Server

Oracle WebLogic 12 c (c stands for cloud) was announced on December 1st 2011. I would like to use this blog post to differentiate between WAS and the new launched WebLogic 12c. Please take an analytic look at the Oracle WebLogic Server 12c announcement here. This article puts to rest any confusion regarding differences between WAS and WebLogic. For a comparison of which is faster WebSphere or WebLogic see.

Labels: competitive

posted by Rohit Kelapure # 11:03 AM 0 Comments

Defensive queuing for website surge protection

Today's blog post explains a deployment and tuning methodology to protect your application server from a traffic surge.

In the figure below decreasing values are assigned to the tuning parameters for each of the displayed Web site components. This deliberate reduction in the maximum number of available resources assigned to each successive layer is called the Funnel model methodology. The benefit of the funnel tuning model is that we
want as many clients to connect to our system as possible, but without overwhelming the resources in each of the layers downstream (for example, database connections). The funnel helps us place these requests into various queues at each layer, where they will wait until the next layer has the capacity to process them. In summary, the funnel model helps us handle bursts of client

Queue tuning points in a Java EE Web site: the Funnel model

For some high level guidance and theory see. The proportion and ratios of the various queues to various another is very important in determining the queuing that occurs at load. If every incoming request requires a connection to a back-end database then the datasource connection pool should be at least equal in size to the servlet thread pool.

What is the size of the WebContainer threadpool in relation to the connection pool ?
What should be the IHS thread/process configuration in relation to the WebContainer threadpool ?

Answer from SWAT guru Kevin Grigorenko -->

The ideal proportion is 1:1 with all wait time on the network, but that assumes all layers perform and scale the same, which is rarely (if ever) the case, so only the general recommendation of 1:<=1 is made. Also, requests are not all the same, so one request may use more memory on a thread than others.
In general, I recommend customers "play it safe" -- for example, start at the max database concurrency, N, then set WAS WebContainer to 1.5N (could go more since a lot of times requests are just for static content) and then set IHS to 2N.
But a lot of stress testing is required at the boundary of the incoming queues -- the biggest problem is that customers configure the front queue (e.g. IHS) at N but then don't even create a test that sends N concurrent requests in. When that happens (spike in load, DOS attack, etc.), their JVMs or database crash/OOM.

Labels: defensive queuing

posted by Rohit Kelapure # 10:29 AM 0 Comments

Using Java Server Faces 2 with WebSphere Application Server 7

If you want to use newer versions of Java Server Faces i.e. JSF 2.x with WebSphere Application Server 7 or WebSphere Application Server 6.1 take a look at this blog post MyFaces 2.0 and WebSphere Application Server v7

Labels: JSF

posted by Rohit Kelapure # 10:03 AM 0 Comments