Alert mechanism - to detect WebSphere Application Server threadpool exhaustion

Another serviceability gem ....  I recommend setting JVM custom property com.ibm.ws.runtime.logThreadPoolGrowth on ALL your servers.

PM13147 - Log growth of thread pool beyond maximum size


PROBLEM DESCRIPTION: 
Growth of thread pools defined as  "growable" beyond the configured      
maximum size is not logged.          

RECOMMENDATION:                                          
"Growable" thread pools are configured with a maximum size but
allowed to increase in size beyond that maximum, and such
growth is not captured by any logging.

Problem conclusion
This change introduces a custom property,
"com.ibm.ws.runtime.logThreadPoolGrowth", that, when enabled,
will print a message to the log when a growable thread pool
increases beyond its configured maximum.

The fix for this APAR is currently targeted for inclusion in
fix packss 6.1.0.35 and 7.0.0.15.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

WSVR0630I=WSVR0630I: Growable thread pool \"{0}\" has been expanded beyond its initially-defined maximum capacity.  The pool size is currently \"{1}\".
WSVR0630I.explanation=The specified thread pool has reached its predefined maximum size and will be expanded.
WSVR0630I.useraction=No immediate action is required; however, the pool will continue to grow as long as work requests outpace the current number of available threads.

Once you see this problem you will need to undergo an exercise in threadpool size tuning as described in
http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html

You can use the thread pool counters that the Performance Monitoring Infrastructure (PMI) and Tivoli® Performance Viewer (TPV) collect to monitor your thread pool activity.

Counter definitions

Table 1. Counter definitions. The following table describes thread pool counters.

Name	Key	Description	Granularity	Type	Level	Overhead	ID
CreateCount	threadPoolModule.threadCreates	The total number of threads created	Per thread pool	CountStatistic	All	Low	1
DestroyCount	threadPoolModule.threadDestroys	The total number of threads destroyed	Per thread pool	CountStatistic	All	Low	2
ActiveCount	threadPoolModule.activeThreads	The number of concurrently active threads Note: The ActiveCount value can include a count for a long-running thread that is used for asynchronous I/O. Under these circumstances, it is possible that even when there is no apparent activity on the thread pool, the ActiveCount value will never reach zero	Per thread pool	BoundedRangeStatistic	Extended, prior to Version 7.0.0.19 Basic, starting with Version 7.0.0.19	High	3
PoolSize	threadPoolModule.poolSize	The average number of threads in pool	Per thread pool	BoundedRangeStatistic	Basic	High	4
PercentMaxed	threadPoolModule.percentMaxed	The average percent of the time that all threads are in use	Per thread pool	BoundedRangeStatistic	All	High	5
DeclaredthreadHungCount	threadPoolModule.declaredThreadHung	The number of threads declared hung	Per thread pool	CountStatistic	All	Max	6
ClearedThreadHangCount	threadPoolModule.declaredThreadHangCleared	The number of thread hangs cleared	Per thread pool	CountStatistic	All	Max	7
ConcurrentHungThreadCount	threadPoolModule.concurrentlyHungThreads	The number of concurrently hung threads	Per thread pool	BoundedRangeStatistic	All	Max	8
ActiveTime	threadPoolModule.activeTime	The average time in milliseconds the threads are in active state	Per thread pool	TimeStatistic	All	Max	9