All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Wednesday, February 29, 2012

Tuning Session Initiation Protocol Container on WebSphere Application Server

If deploying SIP applications on WebSphere Application Server please do yourself a favor and read the following white papers on developerworks

https://www.ibm.com/developerworks/mydeveloperworks/wikis/form/anonymous/api/library/b64e8a50-2d2c-40dc-9433-c5c83665426e/document/4dd9541d-1c0f-41bd-ae1f-ae2ad053d926/attachment/01b07b5e-f011-476c-9aed-7652574b666d/media/ConfiguringandDeployingWebSphereSIPEnvironments_feb282012.pdf

It covers the following SIP Configurations:

Stand-alone SIP Container Environment
Single Proxy High Availability (HA) Environment
Dual Proxy High Availability (HA) Environment with IBM WebSphere Application Server Edge Components Load Balancer

It also includes the following appendices:

WebSphere SIP Application Server Tuning
WebSphere SIP Proxy Server Tuning
Calculating SIP Container Overload Values
Example IBM WebSphere Application Server Edge Components Load Balancer Aliasing Script
Example IBM WebSphere Application Server Edge Components Load Balancer Configuration File
Linux Network and Kernel Tuning Parameters

Labels: SIP

posted by Rohit Kelapure # 11:54 AM 0 Comments

Tuesday, February 28, 2012

Alert mechanism - to detect WebSphere Application Server threadpool exhaustion

Another serviceability gem .... I recommend setting JVM custom property com.ibm.ws.runtime.logThreadPoolGrowth on ALL your servers.

PM13147 - Log growth of thread pool beyond maximum size

PROBLEM DESCRIPTION:
Growth of thread pools defined as "growable" beyond the configured
maximum size is not logged.

RECOMMENDATION:
"Growable" thread pools are configured with a maximum size but
allowed to increase in size beyond that maximum, and such
growth is not captured by any logging.

Problem conclusion
This change introduces a custom property,
"com.ibm.ws.runtime.logThreadPoolGrowth", that, when enabled,
will print a message to the log when a growable thread pool
increases beyond its configured maximum.

The fix for this APAR is currently targeted for inclusion in
fix packss 6.1.0.35 and 7.0.0.15. Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

WSVR0630I=WSVR0630I: Growable thread pool \"{0}\" has been expanded beyond its initially-defined maximum capacity. The pool size is currently \"{1}\".
WSVR0630I.explanation=The specified thread pool has reached its predefined maximum size and will be expanded.
WSVR0630I.useraction=No immediate action is required; however, the pool will continue to grow as long as work requests outpace the current number of available threads.

Once you see this problem you will need to undergo an exercise in threadpool size tuning as described in
http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html

You can use the thread pool counters that the Performance Monitoring Infrastructure (PMI) and Tivoli® Performance Viewer (TPV) collect to monitor your thread pool activity.

Counter definitions

Table 1. Counter definitions. The following table describes thread pool counters.
Name	Key	Description	Granularity	Type	Level	Overhead	ID
CreateCount	threadPoolModule.threadCreates	The total number of threads created	Per thread pool	CountStatistic	All	Low	1
DestroyCount	threadPoolModule.threadDestroys	The total number of threads destroyed	Per thread pool	CountStatistic	All	Low	2
ActiveCount	threadPoolModule.activeThreads	The number of concurrently active threads Note: The ActiveCount value can include a count for a long-running thread that is used for asynchronous I/O. Under these circumstances, it is possible that even when there is no apparent activity on the thread pool, the ActiveCount value will never reach zero	Per thread pool	BoundedRangeStatistic	Extended, prior to Version 7.0.0.19 Basic, starting with Version 7.0.0.19	High	3
PoolSize	threadPoolModule.poolSize	The average number of threads in pool	Per thread pool	BoundedRangeStatistic	Basic	High	4
PercentMaxed	threadPoolModule.percentMaxed	The average percent of the time that all threads are in use	Per thread pool	BoundedRangeStatistic	All	High	5
DeclaredthreadHungCount	threadPoolModule.declaredThreadHung	The number of threads declared hung	Per thread pool	CountStatistic	All	Max	6
ClearedThreadHangCount	threadPoolModule.declaredThreadHangCleared	The number of thread hangs cleared	Per thread pool	CountStatistic	All	Max	7
ConcurrentHungThreadCount	threadPoolModule.concurrentlyHungThreads	The number of concurrently hung threads	Per thread pool	BoundedRangeStatistic	All	Max	8
ActiveTime	threadPoolModule.activeTime	The average time in milliseconds the threads are in active state	Per thread pool	TimeStatistic	All	Max	9

Labels: serviceability

posted by Rohit Kelapure # 11:30 AM 0 Comments

Tuesday, February 21, 2012

How to debug File I/O issues with WAS

Today blog post is a response from JVM serviceability expert Chris Bailey

Question ?
How does one determine what code in WAS is creating files and filling up the temp directory.
*war*.tmp files are created in linux /tmp directory. Who is creating them. These files can cause issues if they aren't cleaned up. Ultimately the /tmp filesystem gets full. The files look like :
abc.war26718.tmp
xyzwar26667.tmp

The best approach is to hack and patch the JVM java.io.File class and dump the thread stack in the File.delete method if the path equals the filename / dir name you are interested in. This works best if you have the environment under your control.

Response

There are three possible approaches to this. Its probably worth taking a combination of approaches:

1. Generating a system dump and using Memory Analyzer
This will let you find the File objects that own the .tmp files, and the objects that are keeping the File objects alive. This should help find out who owns it, although if the code that creates the Files is relying on deleteOnExit() to remove the files then it may no-longer have a reference to the Files itself - although most likely there will be one of more "in-flight" File objects which is still being used by the allocating code.
2. Using HealthCenter
The method profiling view will give you the method calls that call the File constructor, so assuming we have samples that have hit the call to create the File objects we should be able to see which candidates are responsible. Note that we'll only know what Files are being created by the code, not what the file is for (ie, if its a .tmp file or not).
3. Using method trace
We could trace every call to the File constructor and generate a stack trace (I think thats the "jstack" option"). This has the highest performance impact, and like Health Center doesn't know if the allocation is for a File with a .tmp suffix or not.

The constructor methods are the initializers, eg. File.

Labels: debugging

posted by Rohit Kelapure # 3:34 PM 0 Comments

WebSphere Application Server Performance Tuning Toolkit

IBM has released the WebSphere Application Server Performance Tuning Toolkit that gives the ability to monitor multiple WebSphere Application Servers from an eclipse workspace*. This tools uses WAS Performance Monitoring statistics to get and chart statistics that indicate the health and well being of your server. * Please note that the tool has been released on an as-is basis with no support

Now available in IBM Support Assistant
http://www-304.ibm.com/support/docview.wss?uid=swg21577660.

http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html

Performance tuningtoolkitintroduction

View more presentations from kelapure.

Web sphere application server performance tuning workshop

View more documents from kelapure.

Why this tool instead of the other monitoring solutions
1. Its free :-)
2. No agents needed on the server
2. Monitor multiple servers

3. This tool provides a standalone viewer for PMI and mbean statistics. It has a RCP GUI which uses the WebSphere application server admin thin client to call a JMX proxy, the JMX proxy will communicate with the WAS via the SOAP/RMI.

4, Predefined alerts and actions.

· If the heapsize reaches the maximum heap size, it will generate a runtime alert.

· If the thread pool reaches the threshold, it will generate a threadpool alert.

· If the CPU usage reaches 90%, it will generate a runtime alert.

· If some servlet error occur, it will generate a servlet alert.

· If some connection timeout occurs, it will generate a connection alert

· If there are some thread waiting for connection, it will generate a connection alert

· If there are more then 1000 prepared statement discarded, it will generate a connection alert

· If application try to use some invalidated session, it will generate a session alert

· If there is no room for new session, it will generate a session alert

· If some hung thread decleared, it will generate a threadpool alert

· If some transactions rolled back, it will generate a transaction alert

· If some transactions timeout, it will generate a transaction alert

· You can define your own rules.

5. Monitoring

The monitor panel presents performance data (Servlet Response Time, Transaction Response Time, Jdbc Response Time, Servlet Throughput, Transaction Throughput, Jdbc Throughput, Concurrency) in a classified views to help you with correlation and provide a comprehensive view of the state of the server.

you analyze the performance problems in your environment.

Response Time page

The response time page presents the response time of a server. It collects data from four components i.e. servlet, transaction, EjbMethod and JDBC.

Throughoutput page

The throughoutput page presents the throughoutpage of a server.

It displays data from four components. i.e. servlet, transaction, EjbMethod and JDBC.

Concurrency page

The concurrency page presents the concurrency of a server collecting data rom four components transaction, EjbMethod and JDBC.

Alert page

The alert page presents the alert evens of a server.
You can find the alert detail from the alert type and alert description columns.

Servlet page

The servlet page presents the servlet detail informations of a server.
You can find the response time, throughoutput, concurrency and error count in the columns.
When you expand the row , you will get the servlet page detail of you application.

Transaction page

The transaction page presents the transaction information of a server.
It displays response time, through output, roll back count, global timeout count, local timout count, global response time, local response time.

EJB page

The EJB page presents the EJB information of a server. It collects and displays response time, throughput and concurrent invocation data.
When you expand the row, you can find the detail contribution of your application. The different colors mean the degree of the contributions.

Connection Pool page

The connection pool page presents the connection pool information of a server.
It collects data from connection use time, connection through output, error count, wait time, JDBC through output, JDBC response time, JDBC concurrency, statement discard.
When you expand the row, you can find the detail information of different JDBC connections.

Runtime page

The runtime page presents the runtime information of a server.
It collects data from CPU usage, Max heap size, heap size, used heap size.

Session page

The session page presents the session information of a server.
It collects data from live count, life time, affinity break, time out count, active non-exist count, no room count, cache discard count.
When you expand the row, you can find the detail information of your different applications. The different colors mean the different contribution to the whole time.

ThreadPool page

The Thread pool page presents the thread pool information of a server.
It collects data from max size, pool size, use pool, min size, declared thread hung count, concurrent hung thread count, cleared thread hung count.
When you expand the row, you can find the detail information of different thread pools. The colors mean the different contributions.

For more see http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html

FAQ:

Has this tool been used on most distributed platforms (windows, linux, AIX, solairs)? z/OS supported? I saw the UI option for 'tail systemout.log', will this work on Windows? or the tool can do some converting to allow 'tail' to work?
The tool can monitor all the platform include z/OS. but some configuration functions do not work zWAS because of the difference between ZOS and distribute platform. The tail function works on window. Infact, the log was collected via JMX. It does not use the "tail -f" command, and you can take it as a improved log viewer in admin console. The toolkit is based on the PMI and JMX, but it does not use any functions of TPV.

The tool seems to be able to grab the server information including proxy. What if we have IHS web server configured, will this web server be monitored? If messaging SIBus is configured, I guess this will not be monitored? If we have application client (for java client testing) installed, which is not viewable from the dmgr cell, assuming it will not be monitored either?
The toolkit does not support IHS by now. It does NOT monitor SIBus as well, but we may be able to add this function in future if needed. The app client is also not monitored.

Labels: monitoring

posted by Rohit Kelapure # 2:01 PM 0 Comments

Monday, February 13, 2012

WebSphere Application Server version 8 zOS startup performance

Two APARs all WebSphere Application Server zOS 8002 customers should consider applying -

PM51889: DECREASE CLUSTER STARTUP TIME.

****************************************************************
* USERS AFFECTED: All IBM WebSphere Application Server V8.0 users of Base and Network Deployment editions.
****************************************************************
* PROBLEM DESCRIPTION: Excessive JIT compilation increases the start time of servers and clusters
****************************************************************
* RECOMMENDATION:
****************************************************************
The Java JIT compiler provides a performance boost by compiling frequently used classes in such a way that future invocations of those classes perform better. However, the initial compilation of these classes takes additional time. For classes that are used often throughout the life of the server, it is worth taking the initial time to compile them in order to obtain future performance benefits.During server startup, numerous classes are loaded that will rarely be invoked after startup is complete. Compiling all of these classes is often unnecessary and can increase server start time. The JIT compiler should be optimzied to avoid unnecessary compilations during server startup.
Problem conclusion
WebSphere Application Server was modified to inform the JIT compiler of the beginning and end of server startup. The JIT compiler can use this information to make better decisions about which classes to compile, thereby improving the start time of servers and clusters.

Fix on 8002: http://www.ibm.com/support/fixcentral/quickorder?product=ibm/WebSphere/WebSphere+Application+Server&fixids=8.0.0.2-WS-WAS-IFPM51889&source=myna&myns=swgws&mynp=OCSSEQTP&mync=E

The fix for this APAR is currently targeted for inclusion in fix pack 8.0.0.3. Please refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Labels: zos

posted by Rohit Kelapure # 8:02 AM 0 Comments

Saturday, February 4, 2012

Websphere logging quick tip

You can make the WebSphere for z/OS logs look the same as Distributed by setting the following nifty Java system property: com.ibm.ws.dist.trace.format.on.zos=true

Notes:
* To use this property in V7.0 you must be at or above service level 7.0.0.11. Using this property at an older service level will cause a NullPointerException.
* This property only affects the SYSPRINT DD of WebSphere for z/OS output, the remaining DDs are unaffecte

Source: