d8888 888 888      88888888888 888      d8b                                 888       888          888       .d8888b.           888                               
      d88888 888 888          888     888      Y8P                                 888   o   888          888      d88P  Y88b          888                               
     d88P888 888 888          888     888                                          888  d8b  888          888      Y88b.               888                               
    d88P 888 888 888          888     88888b.  888 88888b.   .d88b.  .d8888b       888 d888b 888  .d88b.  88888b.   "Y888b.   88888b.  88888b.   .d88b.  888d888 .d88b.  
   d88P  888 888 888          888     888 "88b 888 888 "88b d88P"88b 88K           888d88888b888 d8P  Y8b 888 "88b     "Y88b. 888 "88b 888 "88b d8P  Y8b 888P"  d8P  Y8b 
  d88P   888 888 888          888     888  888 888 888  888 888  888 "Y8888b.      88888P Y88888 88888888 888  888       "888 888  888 888  888 88888888 888    88888888 
 d8888888888 888 888          888     888  888 888 888  888 Y88b 888      X88      8888P   Y8888 Y8b.     888 d88P Y88b  d88P 888 d88P 888  888 Y8b.     888    Y8b.     
d88P     888 888 888          888     888  888 888 888  888  "Y88888  88888P'      888P     Y888  "Y8888  88888P"   "Y8888P"  88888P"  888  888  "Y8888  888     "Y8888  
                                                                 888                                                          888                                        
                                                            Y8b d88P                                                          888                                        
                                                             "Y88P"                                                           888   

All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Wednesday, February 29, 2012


Tuning Session Initiation Protocol Container on WebSphere Application Server

If deploying SIP applications on WebSphere Application Server please do yourself a favor and read the following white papers on developerworks


It covers the following SIP Configurations:
  • Stand-alone SIP Container Environment
  • Single Proxy High Availability (HA) Environment
  • Dual Proxy High Availability (HA) Environment with IBM WebSphere Application Server Edge Components Load Balancer

It also includes the following appendices:
  • WebSphere SIP Application Server Tuning
  • WebSphere SIP Proxy Server Tuning
  • Calculating SIP Container Overload Values
  • Example IBM WebSphere Application Server Edge Components Load Balancer Aliasing Script
  • Example IBM WebSphere Application Server Edge Components Load Balancer Configuration File
  • Linux Network and Kernel Tuning Parameters


Tuesday, February 28, 2012


Alert mechanism - to detect WebSphere Application Server threadpool exhaustion

Another serviceability gem ....  I recommend setting JVM custom property com.ibm.ws.runtime.logThreadPoolGrowth on ALL your servers.

PM13147 - Log growth of thread pool beyond maximum size

Growth of thread pools defined as  "growable" beyond the configured      
maximum size is not logged.          

"Growable" thread pools are configured with a maximum size but
allowed to increase in size beyond that maximum, and such
growth is not captured by any logging.

Problem conclusion
This change introduces a custom property,
"com.ibm.ws.runtime.logThreadPoolGrowth", that, when enabled,
will print a message to the log when a growable thread pool
increases beyond its configured maximum.

The fix for this APAR is currently targeted for inclusion in
fix packss and  Please refer to the
Recommended Updates page for delivery information:

WSVR0630I=WSVR0630I: Growable thread pool \"{0}\" has been expanded beyond its initially-defined maximum capacity.  The pool size is currently \"{1}\".
WSVR0630I.explanation=The specified thread pool has reached its predefined maximum size and will be expanded.
WSVR0630I.useraction=No immediate action is required; however, the pool will continue to grow as long as work requests outpace the current number of available threads.

Once you see this problem you will need to undergo an exercise in threadpool size tuning as described in

You can use the thread pool counters that the Performance Monitoring Infrastructure (PMI) and Tivoli® Performance Viewer (TPV) collect to monitor your thread pool activity.

Counter definitions

Table 1. Counter definitions. The following table describes thread pool counters.
Name Key Description Granularity Type Level Overhead ID
CreateCount threadPoolModule.threadCreates The total number of threads created Per thread pool CountStatistic All Low 1
DestroyCount threadPoolModule.threadDestroys The total number of threads destroyed Per thread pool CountStatistic All Low 2
[Updated in October 2011] ActiveCount [Updated in October 2011] [Updated in October 2011] threadPoolModule.activeThreads [Updated in October 2011] [Updated in October 2011] The number of concurrently active threads
Note: The ActiveCount value can include a count for a long-running thread that is used for asynchronous I/O. Under these circumstances, it is possible that even when there is no apparent activity on the thread pool, the ActiveCount value will never reach zero
[Updated in October 2011]
[Updated in October 2011] Per thread pool [Updated in October 2011] [Updated in October 2011] BoundedRangeStatistic [Updated in October 2011] [Updated in October 2011] Extended, prior to Version[Fix Pack 19 or later] Basic, starting with Version
[Updated in October 2011]
[Updated in October 2011] High [Updated in October 2011] [Updated in October 2011] 3 [Updated in October 2011]
PoolSize threadPoolModule.poolSize The average number of threads in pool Per thread pool BoundedRangeStatistic Basic High 4
PercentMaxed threadPoolModule.percentMaxed The average percent of the time that all threads are in use Per thread pool BoundedRangeStatistic All High 5
DeclaredthreadHungCount threadPoolModule.declaredThreadHung The number of threads declared hung Per thread pool CountStatistic All Max 6
ClearedThreadHangCount threadPoolModule.declaredThreadHangCleared The number of thread hangs cleared Per thread pool CountStatistic All Max 7
ConcurrentHungThreadCount threadPoolModule.concurrentlyHungThreads The number of concurrently hung threads Per thread pool BoundedRangeStatistic All Max 8
ActiveTime threadPoolModule.activeTime The average time in milliseconds the threads are in active state Per thread pool TimeStatistic All Max 9


Tuesday, February 21, 2012


How to debug File I/O issues with WAS

Today blog post is a response from JVM serviceability expert Chris Bailey

Question ?
How does one determine what code in WAS is creating files and filling up the temp directory.
*war*.tmp files are created in linux /tmp directory. Who is creating them. These files can cause issues if they aren't cleaned up. Ultimately the /tmp filesystem gets full. The files look like :

The best approach is to hack and patch the JVM java.io.File class and dump the thread stack in the File.delete method if the path equals the filename / dir name you are interested in. This works best if you have the environment under your control.

There are three possible approaches to this. Its probably worth taking a combination of approaches:

1. Generating a system dump and using Memory Analyzer
This will let you find the File objects that own the .tmp files, and the objects that are keeping the File objects alive. This should help find out who owns it, although if the code that creates the Files is relying on deleteOnExit() to remove the files then it may no-longer have a reference to the Files itself - although most likely there will be one of more "in-flight" File objects which is still being used by the allocating code.
2. Using HealthCenter
The method profiling view will give you the method calls that call the File constructor, so assuming we have samples that have hit the call to create the File objects we should be able to see which candidates are responsible. Note that we'll only know what Files are being created by the code, not what the file is for (ie, if its a .tmp file or not).
3. Using method trace
We could trace every call to the File constructor and generate a stack trace (I think thats the "jstack" option"). This has the highest performance impact, and like Health Center doesn't know if the allocation is for a File with a .tmp suffix or not.

The constructor methods are the initializers, eg. File.



WebSphere Application Server Performance Tuning Toolkit

IBM has released the WebSphere Application Server Performance Tuning Toolkit that gives the ability to monitor multiple WebSphere Application Servers from an eclipse workspace*. This tools uses WAS Performance Monitoring statistics to get and chart statistics that indicate the health and well being of your server. * Please note that the tool has been released on an as-is basis with no support

Now available in IBM Support Assistant


Why this tool instead of the other monitoring solutions
1. Its free :-)
2. No agents needed on the server
2.  Monitor multiple servers

3. This tool  provides a standalone viewer for PMI and mbean statistics. It has a RCP GUI which uses the  WebSphere application server admin thin client  to call a JMX proxy, the JMX proxy will communicate with the WAS via the SOAP/RMI.

4, Predefined alerts and actions.
· If the heapsize reaches the maximum heap size, it will generate a runtime alert.
· If the thread pool reaches the threshold, it will generate a threadpool alert.
· If the CPU usage reaches 90%, it will generate a runtime alert.
· If some servlet error occur, it will generate a servlet alert.
· If some connection timeout occurs, it will generate a connection alert
· If there are some thread waiting for connection, it will generate a connection alert
· If there are more then 1000 prepared statement discarded, it will generate a connection alert
· If application try to use some invalidated session, it will generate a session alert
· If there is no room for new session, it will generate a session alert
· If some hung thread decleared, it will generate a threadpool alert
· If some transactions rolled back, it will generate a transaction alert
· If some transactions timeout, it will generate a transaction alert
· You can define your own rules.

5. Monitoring

The monitor panel presents performance data (Servlet Response Time, Transaction Response Time, Jdbc Response Time, Servlet Throughput, Transaction Throughput, Jdbc Throughput, Concurrency) in a classified views to help you with correlation and provide a comprehensive view of the state of the server. 
you analyze the performance problems in your environment.

Response Time page
The response time page presents the response time of a server. It collects data from four components i.e. servlet, transaction, EjbMethod and JDBC.

Throughoutput page
The throughoutput page presents the throughoutpage of a server. 
It displays data from four components. i.e. servlet, transaction, EjbMethod and JDBC.

Concurrency page
The concurrency page presents the concurrency of a server collecting data rom four components transaction, EjbMethod and JDBC.

Alert page
The alert page presents the alert evens of a server. 
You can find the alert detail from the alert type and alert description columns.

Servlet page
The servlet page presents the servlet detail informations of a server. 
You can find the response time, throughoutput, concurrency and error count in the columns. 
When you expand the row , you will get the servlet page detail of you application.

Transaction page
The transaction page presents the transaction information of a server.
 It displays  response time, through output, roll back count, global timeout count, local timout count, global response time, local response time.

EJB page
The EJB page presents the EJB information of a server. It collects and displays response time, throughput and concurrent invocation data. 
When you expand the row, you can find the detail contribution of your application. The different colors mean the degree of the contributions.

Connection Pool page
The connection pool page presents the connection pool information of a server. 
It collects data from connection use time, connection through output, error count, wait time, JDBC through output, JDBC response time, JDBC concurrency, statement discard.
When you expand the row, you can find the detail information of different JDBC connections. 

Runtime page
The runtime page presents the runtime information of a server. 
It collects data from CPU usage, Max heap size, heap size, used heap size.

Session page
The session page presents the session information of a server.
It collects data from live count, life time, affinity break, time out count, active non-exist count, no room count, cache discard count. 
When you expand the row, you can find the detail information of your different applications. The different colors mean the different contribution to the whole time.

ThreadPool page
The Thread pool page presents the thread pool information of a server.
It collects data from max size, pool size, use pool, min size, declared thread hung count, concurrent hung thread count, cleared thread hung count. 
When you expand the row, you can find the detail information of different thread pools. The colors mean the different contributions.

For more see http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html


Has this tool been used on most distributed platforms (windows, linux, AIX, solairs)?   z/OS supported?   I saw the UI option for 'tail systemout.log', will this work on Windows?  or the tool can do some converting to allow 'tail' to work?
The tool can monitor all the platform include z/OS. but some configuration functions do not work zWAS because of the difference between ZOS and distribute platform. The tail function works on window. Infact, the log was collected via JMX. It does not use the "tail -f" command, and you can take it as a improved log viewer in admin console. The toolkit is based on the PMI and JMX, but it does not use any functions of TPV.

 The tool seems to be able to grab the server information including proxy.  What if we have IHS web server configured, will this web server be monitored?   If messaging SIBus is configured, I guess this will not be monitored?  If we have application client (for java client testing) installed, which is not viewable from the dmgr cell, assuming it will not be monitored either?
The toolkit does not support IHS  by now.  It does NOT monitor SIBus as well,  but we may be able to add this function in future if needed. The app client is also not monitored.


Monday, February 13, 2012


WebSphere Application Server version 8 zOS startup performance

Two APARs all WebSphere Application Server zOS 8002 customers should consider applying -


* USERS AFFECTED:  All IBM WebSphere Application Server V8.0  users of Base and Network Deployment  editions.                              
* PROBLEM DESCRIPTION: Excessive JIT compilation increases  the start time of servers and clusters             
* RECOMMENDATION:                                            
The Java JIT compiler provides a performance boost by compiling frequently used classes in such a way that future invocations of those classes perform better.  However, the initial compilation of these classes takes additional time. For classes that are used often throughout the life of the server, it is worth taking the initial time to compile them in order to obtain future performance benefits.During server startup, numerous classes are loaded that will rarely be invoked after startup is complete.  Compiling all of these classes is often unnecessary and can increase server start time.  The JIT compiler should be optimzied to avoid unnecessary compilations during server startup.
Problem conclusion
WebSphere Application Server was modified to inform the JIT compiler of the beginning and end of server  startup.  The JIT compiler can use this information to make better decisions about which classes to compile, thereby improving the start time of servers and clusters.

Fix on 8002: http://www.ibm.com/support/fixcentral/quickorder?product=ibm/WebSphere/WebSphere+Application+Server&fixids=

The fix for this APAR is currently targeted for inclusion in fix pack  Please refer to the Recommended Updates page for delivery information:

Read more »


Saturday, February 4, 2012


Websphere logging quick tip

You can make the WebSphere for z/OS logs look the same as Distributed by setting the following nifty Java system property: com.ibm.ws.dist.trace.format.on.zos=true

* To use this property in V7.0 you must be at or above service level Using this property at an older service level will cause a NullPointerException.
* This property only affects the SYSPRINT DD of WebSphere for z/OS output, the remaining DDs are unaffecte



Wednesday, February 1, 2012


WebSphere Proxy Server Best Practice

When configuring the WebSphere Proxy Server for caching  it is a best practice to ENABLE disk offload & set the location to a dedicated disk partition. Even if you do not have a disk partition consider enabling disk offload for your static and dynamic cache. 



December 2006   September 2008   January 2009   February 2009   March 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   October 2010   January 2011   February 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   September 2012   October 2012   November 2012   January 2013   May 2013   June 2013   July 2013   September 2013   October 2013   June 2014   August 2014  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]