All Things WebSphere

WebSphere Application Server on zOS in v7 introduced dispatch timeout improvements. This is a long way of saying that on zOS when a thread times out waiting on a connection or reading data from a socket, zOS, WAS and the JDK can nudge the request along. Read more here. This is also called ITI (Interruptible Thread Infrastructure). We love acronyms in IBM :-)  This dispatched thread interruption applies to ALL stalled threads, even those consuming excessive CPU.

So how does one set up ITI, ? You get ITI by default IF you do NOT have recovery set to session.

Remove the following JVM custom properties

Add the following properties 
protocol_http_timeout_output_recovery         Servant
protocol_https_timeout_output_recovery        Servant
server_region_stalled_thread_threshold_percent    80

There are also some other cool features like the MODIFY,DISPLAY, THREADS command

MODIFY ,DISPLAY,THREADS,TIMEDOUT,DETAILS
MODIFY ,DISPLAY,THREADS,AGE=30,DETAILS

This command, which offers both a SUMMARY and DETAILS format, displays information about server activity, which you can use for identifying requests that run longer than expected.

Another favorite runtime monitoring and  introspection feature of mine is the Dispatch Progress Monitor. In the flood of trace output from hundreds or thousands of requests, it is difficult to find traces related to the occasional long running request. This is where DPM helps as it can be enabled/disabled at runtime.

To enable the DPM, enter the MODIFY command, as follows:
MODIFY ,DPM, xxxx=
default dump action is the TRACEBACK.

Here is an example with the dump action set to TRACEBACK (and formatted for readability):
BBOJ0118I: ThreadDetails: ASID = 005B, TCB = 0X008CBE88, Request = fffff503,
Is JVM Blocked = false, Tried to interrupt = false, Given up = false,
Internal Work Thread = false, Hung Reason = Not Hung,
SR Dispatch Time = 2008/05/05 12:15:31.371625,
CTL Receive Time = 2008/05/05 12:15:31.366693,
CTL Queued to WLM Time = 2008/05/05 12:15:31.371328,
Details = , ODI Details = .JVM INTERRUPTIBLE THREAD, Monitor ACTIVE.
 
BBOJ0117I: JAVA THREAD STACK TRACEBACK FOR THREAD WebSphere:ORB.thread.pool
t=008cbe88: Dispatch Progress Monitor
Traceback for thread WebSphere:ORB.thread.pool t=008cbe88:
com.ibm.ws390.orb.ClientDelegate.invokeRequestCFW(Native Method)
com.ibm.ws390.orb.ClientDelegate.commonInvoke(ClientDelegate.java:998)

Often times developers will put objects in session and modify them without calling javax.servlet.http.HttpSession.setAttribute(String, Object) or javax.servlet.http.HttpSession.removeAttribute(String).

This is BAD because if session persistence is enabled the session manager does not know that the session is touched and fails to replicate to the persistence store. This persistence store can be another server, a database, a data grid or an appliance.

To solve this problem I have written a servlet that forces a sync and writes out all the session attributes to the persistence store at the end of the servlet request.

Take a look and download the src here ..
If you will also need to add the following filter.xml to your web.xml  sorry not everyone is on servlet 3.0 :-)  

The advantage of pointcuts in Spring is that we don’t have to change every individual piece of code we want to be notified about.  We simply write the pointcut with a wildcard match. JEE interceptors seem to be able to be configured using XML instead of annotations

With CDI, Interceptors are defined and configured via annotations (@Interceptor and @InterceptorBinding) and are ONLY enabled via XML. In your case you will ONLY enable the FrameworkUsageInterceptor in the beans.xml file. You will have the following beans,xml in your application
<beans
  xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
  http://java.sun.com/xml/ns/javaee/beans_1_0.xsd">
  <interceptors>
     <class>com.allstate.audit.FrameworkUsageInterceptor</class>
  </interceptors>
</beans>

The nice thing then about AOP is that we can create some kind of cross cut notification of code use.  The advantage of pointcuts in Spring is that we don’t have to change every individual piece of code we want to be notified about.  We simply write the pointcut with a wildcard match.  

This can be done with Java EE CDI by creating a FrameworkLoggingInterceptor that intercepts ALL method invocations and applying the InterceptorBinding (say @FrameworkAudit) dynamically at runtime during application startup to all the application code. http://www.warski.org/blog/2011/03/timing-interceptor-for-cdiweld/  solves a similar problem and explains how to write a timing interceptor where the interceptor applies to any arbitrary code in the app. Code in the application is agnostic of the timing interceptor. You have to follow a very similar approach for your common framework audit interceptor.  CDI extensions will work across application servers since they code to SPIs defined in the Java EE spec.

You will be writing a CDI extension similar to how you will be writing java advice or a pointcut. the beans advised with AOP need to be annotated with @Service or @Component. The CDI extension does this automatically for you i.e. make all the application classes eligible for interception.

Another advantage to Spring AOP is that we can use it outside of a JEE container,

CDI container can also be run standalone i.e. outside the container. Currently WAS does not support this client mode; however the CDI spec and other open source projects are all moving in this direction i.e. to define CDI behavior in a client container. see  http://struberg.wordpress.com/2012/03/17/controlling-cdi-containers-in-se-and-ee/  The reference implementation of CDI called Weld comes with a client container.  Please also look at project Arquillian which makes testing with CDI and Java EE easy.  My advice would be to stick with Java EE and do integration testing with httpunit or httpclient, You don't need to use arquillian or Weld. A standalone CDI container would have been nice for WAS.

WebSphere Application Server in WAS v8 introduced a feature called HPEL.

Why should you chose HPEL over classic logging in WAS
pros:
 - HPEL allows to use one log format on both distributed and zOS WAS
-  HPEL allows you to rotate log files on zOS which could NOT be done with earlier versions of WAS.
-  HPEL improves performance by 33% on distributed
-  HPEL allows enhanced log and trace filtering by time, logger ids and other criteria
-  HPEL has a binary log format which leads to reduced log file sizes
cons:
- HPEL has a binary log format which means you have to use a logViewer post processor tool to read the file in text

Taking all these considerations we can clearly see that HPEL is badass and should be preferred over classic logging in WAS v8 and beyond.

Ensure that your projects are single-rooted. Projects that are single-rooted can be directly consumed by WebSphere Application Server. If the project is not single-rooted, it needs to be copied as part of the publishing operation. If a project is not single-rooted, you see a warning message in the Marker's view.

Minimize the number of Java archives (JARs) in the WEB-INF/lib directories. If the JAR is no longer needed, remove it. If you see that the same jar is repeated in many lib directories, see if it can be moved to a shared directory on the server.

Start the test server in debug mode. This approach, which allows you to change the code by using the "hot method replace" and directly inserting the code into the running server, eliminates the need to perform a publish. There are cases where the Java virtual machine (JVM) cannot replace a method. In those cases, you get a warning message and then you need to publish the application.

Use Remote Method Invocation (RMI) as the connection type, because no polling is involved.

Remove unused applications. Removing unused applications shortens up your server start-up time.

Turn off your server's security, which improves the start-up time for your server as well as improving your publish times.

Periodically clean out the server's wstemp and temp directories.

Publish is an expensive operation because of all of the Rational Application Developer code, WebSphere Application Server code, and application code, for example, servlet destruction code, that must be processed. The best publish is no publish. It is best if you are in control of when publishes occur, so we recommend turning off automatic publishes. see figure

Look at this appendix of  links for faster RAD performance with WebSphere

• Better hardware  
• Shared EARs (binary modules)
• Annotations
• Publishing
• Shorter build time by tuning validation
• Only install what you need
• No circular dependencies
• Using a remote test server
• Tuning your anti-virus program
• Defragmenting disks  

Set LoadBalanceWeight to a value other than default say 20.  If there are 3 clones, you might choose the starting LoadBalanceWeights to be: 20, 20, 19. After normalization the weights will be unchanged. It is recommended that all clones have same weight except one clone off by one (for example: 20, 20, 19)

Set set IgnoreAffinityRequests="false".If using Round Robin for the LoadBalance option, by default the affinity requests do NOT lower the LoadBalanceWeight (IgnoreAffinityRequests="true"). This can cause an uneven distribution across the servers in environments that make use of session affinity. But, If IgnoreAffinityRequests="false" then the weight IS lowered by each affinity request, leading to a more balanced Round Robin environment. When using Random, the affinity requests are still handled correctly (sent to same cloneid as before). But new requests are routed randomly, and the LoadBalanceWeight is not used.

<ServerCluster CloneSeparatorChange="false" GetDWLMTable="true" IgnoreAffinityRequests="false" LoadBalance="Round Robin" Name="wp_cluster" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">