Concerns and issues relating to all versions of WebSphere Application Server

Saturday, March 3, 2012

WebSphere Application Server large virtual memory process size on Red Hat Enterprise Linux 6 vs RHEL 5

If after migrating websphere application server from RHEL 5 to RHEL you see a increase in the virtual memory size of your JVM checkout

http://www.quora.com/Why-do-some-applications-use-significantly-more-virtual-memory-on-RHEL-6-compared-to-RHEL-5
https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en

This manifests itself as a doubling of the resident size of the JVM as compared to its virtual memory size.

This can be fixed by setting environment variable (https://www-304.ibm.com/support/docview.wss?uid=swg21254153),
Name=MALLOC_ARENA_MAX and Value=1

Labels: VIRTUAL MEMORY

posted by Rohit Kelapure # 6:18 PM 0 Comments Links to this post

Friday, March 2, 2012

IBM Java Health Center

Please find below a detailed yet most concise information on the IBM Java Health Center

The IBM Java Health Center is a very low overhead tool that runs in the IBM JVM and provides information on method profiling, garbage collection, I/O, lock analysis,threads, native memory, and more.
It is fully supported by the IBM Java Tools team through PMRs.– Documentation
It is similar to HotSpot/Oracle's VisualVM and JRockit Mission Control
It runs with the IBM JVM (32- or 64-bit) on: AIX, Linux, Windows, and z/OS.

IBM Health Center Details

View more presentations from kelapure

Labels: health center

posted by Rohit Kelapure # 7:16 AM 0 Comments Links to this post

Wednesday, February 29, 2012

Tuning Session Initiation Protocol Container on WebSphere Application Server

If deploying SIP applications on WebSphere Application Server please do yourself a favor and read the following white papers on developerworks

https://www.ibm.com/developerworks/mydeveloperworks/wikis/form/anonymous/api/library/b64e8a50-2d2c-40dc-9433-c5c83665426e/document/4dd9541d-1c0f-41bd-ae1f-ae2ad053d926/attachment/01b07b5e-f011-476c-9aed-7652574b666d/media/ConfiguringandDeployingWebSphereSIPEnvironments_feb282012.pdf

It covers the following SIP Configurations:

Stand-alone SIP Container Environment
Single Proxy High Availability (HA) Environment
Dual Proxy High Availability (HA) Environment with IBM WebSphere Application Server Edge Components Load Balancer

It also includes the following appendices:

WebSphere SIP Application Server Tuning
WebSphere SIP Proxy Server Tuning
Calculating SIP Container Overload Values
Example IBM WebSphere Application Server Edge Components Load Balancer Aliasing Script
Example IBM WebSphere Application Server Edge Components Load Balancer Configuration File
Linux Network and Kernel Tuning Parameters

Labels: SIP

posted by Rohit Kelapure # 11:54 AM 0 Comments Links to this post

Tuesday, February 28, 2012

Alert mechanism - to detect WebSphere Application Server threadpool exhaustion

Another serviceability gem .... I recommend setting JVM custom property com.ibm.ws.runtime.logThreadPoolGrowth on ALL your servers.

PM13147 - Log growth of thread pool beyond maximum size

PROBLEM DESCRIPTION:
Growth of thread pools defined as "growable" beyond the configured
maximum size is not logged.

RECOMMENDATION:
"Growable" thread pools are configured with a maximum size but
allowed to increase in size beyond that maximum, and such
growth is not captured by any logging.

Problem conclusion
This change introduces a custom property,
"com.ibm.ws.runtime.logThreadPoolGrowth", that, when enabled,
will print a message to the log when a growable thread pool
increases beyond its configured maximum.

The fix for this APAR is currently targeted for inclusion in
fix packss 6.1.0.35 and 7.0.0.15. Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

WSVR0630I=WSVR0630I: Growable thread pool \"{0}\" has been expanded beyond its initially-defined maximum capacity. The pool size is currently \"{1}\".
WSVR0630I.explanation=The specified thread pool has reached its predefined maximum size and will be expanded.
WSVR0630I.useraction=No immediate action is required; however, the pool will continue to grow as long as work requests outpace the current number of available threads.

Labels: serviceability

posted by Rohit Kelapure # 11:30 AM 0 Comments Links to this post

Tuesday, February 21, 2012

How to debug File I/O issues with WAS

Today blog post is a response from JVM serviceability expert Chris Bailey

Question ?
How does one determine what code in WAS is creating files and filling up the temp directory.
*war*.tmp files are created in linux /tmp directory. Who is creating them. These files can cause issues if they aren't cleaned up. Ultimately the /tmp filesystem gets full. The files look like :
abc.war26718.tmp
xyzwar26667.tmp

Response

There are three possible approaches to this. Its probably worth taking a combination of approaches:

1. Generating a system dump and using Memory Analyzer
This will let you find the File objects that own the .tmp files, and the objects that are keeping the File objects alive. This should help find out who owns it, although if the code that creates the Files is relying on deleteOnExit() to remove the files then it may no-longer have a reference to the Files itself - although most likely there will be one of more "in-flight" File objects which is still being used by the allocating code.
2. Using HealthCenter
The method profiling view will give you the method calls that call the File constructor, so assuming we have samples that have hit the call to create the File objects we should be able to see which candidates are responsible. Note that we'll only know what Files are being created by the code, not what the file is for (ie, if its a .tmp file or not).
3. Using method trace
We could trace every call to the File constructor and generate a stack trace (I think thats the "jstack" option"). This has the highest performance impact, and like Health Center doesn't know if the allocation is for a File with a .tmp suffix or not.

The constructor methods are the initializers, eg. File.

Labels: debugging

posted by Rohit Kelapure # 3:34 PM 0 Comments Links to this post

WebSphere Application Server Performance Tuning Toolkit

IBM has released the WebSphere Application Server Performance Tuning Toolkit that gives the ability to monitor multiple WebSphere Application Servers from an eclipse workspace*. This tools uses WAS Performance Monitoring statistics to get and chart statistics that indicate the health and well being of your server. * Please note that the tool has been released on an as-is basis with no support.

http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html

Why this tool instead of the other monitoring solutions
1. Its free :-)
2. No agents needed on the server
2. Monitor multiple servers

3. This tool provides a standalone viewer for PMI and mbean statistics. It has a RCP GUI which uses the WebSphere application server admin thin client to call a JMX proxy, the JMX proxy will communicate with the WAS via the SOAP/RMI.

4, Predefined alerts and actions.

· If the heapsize reaches the maximum heap size, it will generate a runtime alert.

· If the thread pool reaches the threshold, it will generate a threadpool alert.

· If the CPU usage reaches 90%, it will generate a runtime alert.

· If some servlet error occur, it will generate a servlet alert.

· If some connection timeout occurs, it will generate a connection alert

· If there are some thread waiting for connection, it will generate a connection alert

· If there are more then 1000 prepared statement discarded, it will generate a connection alert

· If application try to use some invalidated session, it will generate a session alert

· If there is no room for new session, it will generate a session alert

· If some hung thread decleared, it will generate a threadpool alert

· If some transactions rolled back, it will generate a transaction alert

· If some transactions timeout, it will generate a transaction alert

· You can define your own rules.

5. Monitoring

The monitor panel presents performance data (Servlet Response Time, Transaction Response Time, Jdbc Response Time, Servlet Throughput, Transaction Throughput, Jdbc Throughput, Concurrency) in a classified views to help you with correlation and provide a comprehensive view of the state of the server.

you analyze the performance problems in your environment.

Response Time page

The response time page presents the response time of a server. It collects data from four components i.e. servlet, transaction, EjbMethod and JDBC.

Throughoutput page

The throughoutput page presents the throughoutpage of a server.

It displays data from four components. i.e. servlet, transaction, EjbMethod and JDBC.

Concurrency page

The concurrency page presents the concurrency of a server collecting data rom four components transaction, EjbMethod and JDBC.

Alert page

The alert page presents the alert evens of a server.
You can find the alert detail from the alert type and alert description columns.

Servlet page

The servlet page presents the servlet detail informations of a server.
You can find the response time, throughoutput, concurrency and error count in the columns.
When you expand the row , you will get the servlet page detail of you application.

Transaction page

The transaction page presents the transaction information of a server.
It displays response time, through output, roll back count, global timeout count, local timout count, global response time, local response time.

EJB page

The EJB page presents the EJB information of a server. It collects and displays response time, throughput and concurrent invocation data.
When you expand the row, you can find the detail contribution of your application. The different colors mean the degree of the contributions.

Connection Pool page

The connection pool page presents the connection pool information of a server.
It collects data from connection use time, connection through output, error count, wait time, JDBC through output, JDBC response time, JDBC concurrency, statement discard.
When you expand the row, you can find the detail information of different JDBC connections.

Runtime page

The runtime page presents the runtime information of a server.
It collects data from CPU usage, Max heap size, heap size, used heap size.

Session page

The session page presents the session information of a server.
It collects data from live count, life time, affinity break, time out count, active non-exist count, no room count, cache discard count.
When you expand the row, you can find the detail information of your different applications. The different colors mean the different contribution to the whole time.

ThreadPool page

The Thread pool page presents the thread pool information of a server.
It collects data from max size, pool size, use pool, min size, declared thread hung count, concurrent hung thread count, cleared thread hung count.
When you expand the row, you can find the detail information of different thread pools. The colors mean the different contributions.

For more see http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html

Labels: monitoring

posted by Rohit Kelapure # 2:01 PM 0 Comments Links to this post

Monday, February 13, 2012

WebSphere Application Server version 8 zOS startup performance

Two APARs all WebSphere Application Server zOS 8002 customers should consider applying -

PM51889: DECREASE CLUSTER STARTUP TIME.

****************************************************************
* USERS AFFECTED: All IBM WebSphere Application Server V8.0 users of Base and Network Deployment editions.
****************************************************************
* PROBLEM DESCRIPTION: Excessive JIT compilation increases the start time of servers and clusters
****************************************************************
* RECOMMENDATION:
****************************************************************
The Java JIT compiler provides a performance boost by compiling frequently used classes in such a way that future invocations of those classes perform better. However, the initial compilation of these classes takes additional time. For classes that are used often throughout the life of the server, it is worth taking the initial time to compile them in order to obtain future performance benefits.During server startup, numerous classes are loaded that will rarely be invoked after startup is complete. Compiling all of these classes is often unnecessary and can increase server start time. The JIT compiler should be optimzied to avoid unnecessary compilations during server startup.
Problem conclusion
WebSphere Application Server was modified to inform the JIT compiler of the beginning and end of server startup. The JIT compiler can use this information to make better decisions about which classes to compile, thereby improving the start time of servers and clusters.

Fix on 8002: http://www.ibm.com/support/fixcentral/quickorder?product=ibm/WebSphere/WebSphere+Application+Server&fixids=8.0.0.2-WS-WAS-IFPM51889&source=myna&myns=swgws&mynp=OCSSEQTP&mync=E

The fix for this APAR is currently targeted for inclusion in fix pack 8.0.0.3. Please refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Labels: zos

posted by Rohit Kelapure # 8:02 AM 2 Comments Links to this post