Interruptible Thread Infrastructure - WebSphere Application Server on zOS

WebSphere Application Server on zOS in v7 introduced dispatch timeout improvements. This is a long way of saying that on zOS when a thread times out waiting on a connection or reading data from a socket, zOS, WAS and the JDK can nudge the request along. Read more here. This is also called ITI (Interruptible Thread Infrastructure). We love acronyms in IBM :-)  This dispatched thread interruption applies to ALL stalled threads, even those consuming excessive CPU.

So how does one set up ITI, ? You get ITI by default IF you do NOT have recovery set to session.

Remove the following JVM custom properties

protocol_http_timeout_output_recovery        SESSION
protocol_https_timeout_output_recovery       SESSION

Add the following properties 
protocol_http_timeout_output_recovery         Servant
protocol_https_timeout_output_recovery        Servant
server_region_stalled_thread_threshold_percent    80

There are also some other cool features like the MODIFY,DISPLAY, THREADS command

MODIFY ,DISPLAY,THREADS,TIMEDOUT,DETAILS
MODIFY ,DISPLAY,THREADS,AGE=30,DETAILS

This command, which offers both a SUMMARY and DETAILS format, displays information about server activity, which you can use for identifying requests that run longer than expected.

Another favorite runtime monitoring and  introspection feature of mine is the Dispatch Progress Monitor. In the flood of trace output from hundreds or thousands of requests, it is difficult to find traces related to the occasional long running request. This is where DPM helps as it can be enabled/disabled at runtime.

To enable the DPM, enter the MODIFY command, as follows:
MODIFY ,DPM, xxxx=

MODIFY ,DPM,DUMP_ACTION=JAVACORE

default dump action is the TRACEBACK.

Here is an example with the dump action set to TRACEBACK (and formatted for readability):
BBOJ0118I: ThreadDetails: ASID = 005B, TCB = 0X008CBE88, Request = fffff503,
Is JVM Blocked = false, Tried to interrupt = false, Given up = false,
Internal Work Thread = false, Hung Reason = Not Hung,
SR Dispatch Time = 2008/05/05 12:15:31.371625,
CTL Receive Time = 2008/05/05 12:15:31.366693,
CTL Queued to WLM Time = 2008/05/05 12:15:31.371328,
Details = , ODI Details = .JVM INTERRUPTIBLE THREAD, Monitor ACTIVE.
 
BBOJ0117I: JAVA THREAD STACK TRACEBACK FOR THREAD WebSphere:ORB.thread.pool
t=008cbe88: Dispatch Progress Monitor
Traceback for thread WebSphere:ORB.thread.pool t=008cbe88:
com.ibm.ws390.orb.ClientDelegate.invokeRequestCFW(Native Method)
com.ibm.ws390.orb.ClientDelegate.commonInvoke(ClientDelegate.java:998)