All Things WebSphere

Labels: application performance

Labels: migration

Ever wondered what your application server was up to ? If you want to know what all the threads of the application server are doing or what an individual thread is upto take a look at the two JSPs provided below which simulate a windows task manager for all the threads of the Application Server

These JSPs leverge the ThreadMXBean API in the JDK to display thread information. Caution: This may not work on zOS. These JSPs allow a user to see all threads  in a table view, and then click on one of the threads to get a detailed view.
https://github.com/kelapure/dynacache/blob/master/scripts/AllThreads.jsp
https://github.com/kelapure/dynacache/blob/master/scripts/ViewThread.jsp

Today's objective is to generate -Xtrace for ALL methods on the WebSphere PMI interfaces that our PMI modules implement. These PMI interfaces are called by the WAS container to compute PMI statistics. To do this we use IBM JVM's method trace facility.

While method trace is powerful, it also has a cost. Application throughput is affected by method trace. The impact is proportion to the number of methods traced. Additionally, trace output is reasonably large and might require a large amount of drive space. For instance, a full method trace of a "Hello World" application is over 10 MB
You can control method trace by using the command-line option -Xtrace:. This -Xtrace is set as a JVM generic argument in the WebSphere Application Server admin console.

Please see below for the JVM generic argument for capturing all PMI Event points as JVM trace.
-Xtrace:methods={com/ibm/ws/pmi/server/modules/*.*()},print=mt

This will put the output in native_stderr.log.
You can then write post processors to process the trace or alternately you may be able to suck the output in IBM Health Center to visualize it. I have not tried this yet.

More details on -Xtrace here
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/topic/com.ibm.java.doc.diagnostics.50/diag/tools/trace_options_method.html
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.50%2Fdiag%2Ftools%2Ftrace_options_output.html

Unfortunately this will generate the JVM trace for ALL methods of the PMI modules not just the ones on the PMI interface. This limitation we will need to live with for now.Thanks to WAS SWAT Team guru Kevin Grigorenko for sounding me out as we discussed options.


Happy Debugging!

Labels: debugging

If your WebSphere Application Server deployment is experiencing an imbalance in HTTP traffic routing from the WebSphere Plugin please take the following configuration and recommendations into consideration

1. RANDOM load balancing is better than ROUND_ROBIN for load balancing. With RANDOM for all non-affinity requests, the server is “picked” randomly.The affinity requests are sent to the matching clone id. LoadBalance “weight” is not used at all in the Random case.
2.  An affinity request is one where session affinity is established by the first request. All subsequent requests with the same sesssion ID are termed as "affinity requests". Plugin ALWAYS obeys session affinity when routing requests and routes to the appserver targeted on the first request that established affnity.
3. In Round Robin load balancing, the first request is routed randomly. After that, the plug-in will use the next server in the list and so on… (member1, member2, member3, member4). Each time a server is “picked”, the load balance weight is decremented by one. Any server with a weight below 0 will NO longer be used for new requests. But, affinity requests will still go through.When all servers have a weight of less than 0, the plug-in will “reset” the weights to the starting values. By default affinity requests do NOT cause the plugin to decrement the weight of the server. This can lead to an uneven distribution when there are many affinity requests.
• One options you have is to set the property IgnoreAffinityRequests="false". If IgnoreAffinityRequests="false" then the weight IS lowered by each affinity request, leading to a more balanced routing in a Round Robin environment. This can have some adverse consequences if you get a burst of non-affinity requests, they could all be targeted to the servers with the lowest weight and can cause issues with that server. To protect against this, you can defined the maxconnection setting in conjunction, which would avoid having too many non-affinity requests sent to a single server at any one time.

If you already have plugin traces or are willing to enable debug to triage this issue further WAS IBM HTTP Server guru and WebSphere plugin expert Eric Covener has the following scripts that will help analyze your plugin trace and provide a summary of your plugin HTTP routing statistics https://github.com/covener/plugin-tools

For recommended values of the plugin configuration please look at  http://www-01.ibm.com/support/docview.wss?uid=swg21318463
To understand HTTP Plugin load balancing  see http://www-01.ibm.com/support/docview.wss?uid=swg21219567
Demystifying the WebSphere Plugin http://www-01.ibm.com/support/docview.wss?uid=swg27014498&aid=1

How do you determine if this is an "affinity" request
 ) Check for SSLJSESSION cookie
2) Check for JSESSIONID cookie
3) Check for URL rewrite (jsessionid) in the URL
4) Check partitionID against partition table if any

How do you know which server was picked by the WebSphere Plugin ?
The trace shows which server was “picked”. Search the trace file (http_plugin.log) for the string “picked, weight” to quickly see which servers are being chosen in what order.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
lockedServerGroupUseServer: Server GAME3Node01_member4 picked, weight 6.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
serverGroupIncrementConnectionCount: Server GAME3Node01_member4 picked,
pendingConnectionCount 1 totalConnectionsCount 1.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - DEBUG: ws_server_group:
serverGroupNextRoundRobinServer: use server GAME3Node01_member4

Labels: WebSphere Plugin

Since WebSphere Application Server version 7, we have exposed the First Failure Data Capture (FFDC) API to customer applications. Unfortunately we have not publicized this much in the Infocenter.

One of the key points for FFDC that differentiates it from log and trace is that it is a snapshot and not  history.  Log and Trace tends to give a narrow view, but a history that can be highly valuable.  FFDC tries to provide a wider view at a point in time. 

See Javadoc 
http://publib.boulder.ibm.com/infocenter/wasinfo/fep/topic/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/ffdc/config/package-summary.html
http://publib.boulder.ibm.com/infocenter/wasinfo/fep/topic/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/ffdc/package-summary.html

What is FFDC and Why do I care ?
FFDC is capturing the state of the application server and/or the application when an unexpected error or exception occurs the first time. All subsequent iterations of the same error/exception are ignored. This lightweight diagnostic collector of information is designed to be ALWAYS on in a production environment. WAS code leverages FFDC extensively to collect information and state for a component when an exception or catastrophic event occurs. You can now do the same for your enterprise application.

What differentiates it from logging ?

1. An exception has already occurred. This means that performance has been impacted and the code is in a failure path
2. FFDC is in the category of dump or snapshot type tools. These tools aim to provide a broad view of the system at a particular point in time. Logging keeps a narrow view, over a period of time
3. Each FFDC statement executes only once (some rare caveats). If the statement is executed again, it will know that it has already been executed, and it will simply update summary information. This means that FFDC processing can focus more on capturing all needed information, and less on performance.
4. The FFDC infrastructure provides many points where developers can plug in code that will be called when appropriate. The class or method experiencing a failure rarely knows all of the needed context to resolve the problem. FFDC provides extensions so that more focused serviceability code can take a more holistic view and gather a much broader context.

How do I use it ?

try { 

// Application code here 


} catch (Exception e) { 

    Ffdc.log(e, myClass, myClassName+myMethodName, “lineNumber”, cde1, cde2, …) ; 

}
// Args:  Exception, reporting class, “sourceId”, “probeId”, context data elements
// where sourceId and probeId are any strings, but this pattern is common 

package howto_ffdc._1_simple;
import static com.ibm.ffdc.Manager.Ffdc;
// import com.ibm.ffdc.Ffdc; // Used if alternate call is done below
public class SimpleTest extends TestCase {
 
   protected void setUp() throws Exception {
     System.setProperty("com.ibm.ffdc.log", “/opt/IBM/WebSphere/logs/ffdc/");
    }
    
    public void testWithoutFormatter(){
  try {
         // ... do work
  } catch (Exception e) {
         Ffdc.log(e, this, getClass().getName(),"24", customer);
         /* alternate if generating the parms for the call is expensive */
         // Ffdc ffdc = Ffdc.getFfdc(e, this, getClass().getName(),"24") ;
         //if (ffdc.isLoggable())
         //   MyData myData = expensiveCallToGetData() ;
         //   ffdc.log(customer, myData) ;
   }
 }
}

An excellent deep dive into FFDC by Michael Casile, WebSphere Serviceability Team Lead shows how you can use FFDC in your Java Enterprise application with WebSphere Application Server. http://slidesha.re/pWODOZ

In summary FFDC is a simple WebSphere application server facility to improve the serviceability of you enterprise software. It is low cost to implement, extensible, can grow with you (Data Collectors, Formatters, Providers, and Incident Forwarders) and the extensions do not impact core code i.e. no change needed to Ffdc.log statements to affect improved information.

Labels: dynacache

Labels: debugging

Labels: application performance