d8888 888 888      88888888888 888      d8b                                 888       888          888       .d8888b.           888                               
      d88888 888 888          888     888      Y8P                                 888   o   888          888      d88P  Y88b          888                               
     d88P888 888 888          888     888                                          888  d8b  888          888      Y88b.               888                               
    d88P 888 888 888          888     88888b.  888 88888b.   .d88b.  .d8888b       888 d888b 888  .d88b.  88888b.   "Y888b.   88888b.  88888b.   .d88b.  888d888 .d88b.  
   d88P  888 888 888          888     888 "88b 888 888 "88b d88P"88b 88K           888d88888b888 d8P  Y8b 888 "88b     "Y88b. 888 "88b 888 "88b d8P  Y8b 888P"  d8P  Y8b 
  d88P   888 888 888          888     888  888 888 888  888 888  888 "Y8888b.      88888P Y88888 88888888 888  888       "888 888  888 888  888 88888888 888    88888888 
 d8888888888 888 888          888     888  888 888 888  888 Y88b 888      X88      8888P   Y8888 Y8b.     888 d88P Y88b  d88P 888 d88P 888  888 Y8b.     888    Y8b.     
d88P     888 888 888          888     888  888 888 888  888  "Y88888  88888P'      888P     Y888  "Y8888  88888P"   "Y8888P"  88888P"  888  888  "Y8888  888     "Y8888  
                                                                 888                                                          888                                        
                                                            Y8b d88P                                                          888                                        
                                                             "Y88P"                                                           888   

All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Tuesday, February 28, 2012

 

Alert mechanism - to detect WebSphere Application Server threadpool exhaustion



Another serviceability gem ....  I recommend setting JVM custom property com.ibm.ws.runtime.logThreadPoolGrowth on ALL your servers.

PM13147 - Log growth of thread pool beyond maximum size


PROBLEM DESCRIPTION: 
Growth of thread pools defined as  "growable" beyond the configured      
maximum size is not logged.          

RECOMMENDATION:                                          
"Growable" thread pools are configured with a maximum size but
allowed to increase in size beyond that maximum, and such
growth is not captured by any logging.

Problem conclusion
This change introduces a custom property,
"com.ibm.ws.runtime.logThreadPoolGrowth", that, when enabled,
will print a message to the log when a growable thread pool
increases beyond its configured maximum.

The fix for this APAR is currently targeted for inclusion in
fix packss 6.1.0.35 and 7.0.0.15.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

WSVR0630I=WSVR0630I: Growable thread pool \"{0}\" has been expanded beyond its initially-defined maximum capacity.  The pool size is currently \"{1}\".
WSVR0630I.explanation=The specified thread pool has reached its predefined maximum size and will be expanded.
WSVR0630I.useraction=No immediate action is required; however, the pool will continue to grow as long as work requests outpace the current number of available threads.

Once you see this problem you will need to undergo an exercise in threadpool size tuning as described in
http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html

You can use the thread pool counters that the Performance Monitoring Infrastructure (PMI) and TivoliĀ® Performance Viewer (TPV) collect to monitor your thread pool activity.

Counter definitions


Table 1. Counter definitions. The following table describes thread pool counters.
Name Key Description Granularity Type Level Overhead ID
CreateCount threadPoolModule.threadCreates The total number of threads created Per thread pool CountStatistic All Low 1
DestroyCount threadPoolModule.threadDestroys The total number of threads destroyed Per thread pool CountStatistic All Low 2
[Updated in October 2011] ActiveCount [Updated in October 2011] [Updated in October 2011] threadPoolModule.activeThreads [Updated in October 2011] [Updated in October 2011] The number of concurrently active threads
Note: The ActiveCount value can include a count for a long-running thread that is used for asynchronous I/O. Under these circumstances, it is possible that even when there is no apparent activity on the thread pool, the ActiveCount value will never reach zero
[Updated in October 2011]
[Updated in October 2011] Per thread pool [Updated in October 2011] [Updated in October 2011] BoundedRangeStatistic [Updated in October 2011] [Updated in October 2011] Extended, prior to Version 7.0.0.19[Fix Pack 19 or later] Basic, starting with Version 7.0.0.19
[Updated in October 2011]
[Updated in October 2011] High [Updated in October 2011] [Updated in October 2011] 3 [Updated in October 2011]
PoolSize threadPoolModule.poolSize The average number of threads in pool Per thread pool BoundedRangeStatistic Basic High 4
PercentMaxed threadPoolModule.percentMaxed The average percent of the time that all threads are in use Per thread pool BoundedRangeStatistic All High 5
DeclaredthreadHungCount threadPoolModule.declaredThreadHung The number of threads declared hung Per thread pool CountStatistic All Max 6
ClearedThreadHangCount threadPoolModule.declaredThreadHangCleared The number of thread hangs cleared Per thread pool CountStatistic All Max 7
ConcurrentHungThreadCount threadPoolModule.concurrentlyHungThreads The number of concurrently hung threads Per thread pool BoundedRangeStatistic All Max 8
ActiveTime threadPoolModule.activeTime The average time in milliseconds the threads are in active state Per thread pool TimeStatistic All Max 9

Labels:


Comments:

Post a Comment

Subscribe to Post Comments [Atom]



Links to this post:

Create a Link



<< Home

Archives

December 2006   September 2008   January 2009   February 2009   March 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   October 2010   January 2011   February 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   September 2012   October 2012   November 2012   January 2013   May 2013   June 2013   July 2013   September 2013   October 2013   June 2014   August 2014  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]