All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Monday, May 30, 2011

Reducing WebSphere Application Server I/O overhead

Best practices for reducing I/O overhead of WebSphere Application Server:

If you applications on WAS are I/O intensive i.e do a lot of logging then following the best practices below will help reduce the stress on the OS I/O sub-system. If you run into issues with incomplete logging or with truncated javacores then reducing the amount of information logged to the system may help reduce the severity of the problem.

1. Disable the WebSphere Application Server service log/activity log
The service log is more commonly known as the activity.log and is found in the /profiles//logs directory. There is only one activity.log for each node. WebSphere Application Server runtime events are logged to the activity.log. It is written in binary format, so it cannot be viewed in a text editor.

The main purpose of the activity.log is that it can be viewed with the Log Analyzer tool, is a graphical user interface that displays the events from the activity.log and uses a symptom database to analyze the events and diagnose problems. This service is not essential to WAS and very few system administrator make use of the activity / service log.

You can configure properties of the activity.log in the administrative console:
1. Select Troubleshooting → Logs and Trace.
2. Select the WebSphere Application Server process.
3. Select IBM® Service Logs. Disable the activity.log.

posted by Rohit Kelapure # 6:19 AM 0 Comments

Tuesday, May 24, 2011

Dynacache vs Memcached

This is a series of posts comparing caching technologies. The first in this series is Dynacache vs Memcached.

Caching technology comparison

View more presentations from kelapure

Labels: dynacache

posted by Rohit Kelapure # 5:02 AM 0 Comments

Thursday, May 19, 2011

WAS GenCon Garbage Collection Policy

In WebSphere Application Server v8 that will be available from June 17 the default garbage collection policy has been changed to gencon.
You may wonder why this policy is now the default ?

■ GenCon is good for transactional workloads: when the transaction is done, most of the objects are thrown away
■ GenCon is good for interactive workloads: short GC pauses mean high responsiveness
■ Many applications fit one of these models
■ Many common Java idioms create short-lived helper objects:
– e.g. StringBuffer / StringWriter,
– e.g. Enumerator / Iterator
■ A smaller collection area means a smaller working set, improving cache utilization
■ The young generation collector compacts as it collects, reducing fragmentation
■ It also tends to relocate objects so that related objects move closer together
– For example: a String and its char[] array, or a HashTable$Entry and its key.
– This can bring significant cache benefits

Things to be aware of when using gencon
■ Tenured objects are rarely collected (by design!) -- this can lead to objects living longer than
expected. A common case of this is class unloading
■ Similar problems exist for finalization and reference objects that survive long enough to be
tenured
■ Newspace overhead: since the young generation is divided into semi-spaces, there is
always a small amount of unusable heap

Please look at this presentation on Generational Garbage Collection: Theory and Best Practices on the UK WebSphere Users Group http://www.websphereusergroup.org.uk/wug/files/presentations/31/Chris_Bailey_-_Generational_GC.pdf

posted by Rohit Kelapure # 7:58 AM 0 Comments

Wednesday, May 18, 2011

WebSphere Application Server Top 10 Tuning Recommendations

This blog post is based on a talk given at IMPACT 2011 by WebSphere Chief Performance Architect Surya Duggirala

#10 - Properly Tune the Operating System

Operating System is consistently overlooked for functional tuning as well as performance tuning.
Understand the hardware infrastructure backing your OS. Processor counts, speed, shared/unshared, etc
ulimit values need to be set correctly. Main player here is the number of open file handles (ulimit –n). Other process size and memory ones may need to be set based on application
Make sure NICs are set to full duplex and correct speeds
Large pages need to be enabled to take advantage of –Xlp JDK parametes
If enabled by default check RAS settings on OS and tune them down
Configure TCP/IP timeouts correctly for your applications needs
Depending on the load being placed on the system look into advanced tuning techniques such as pinning WAS processes via RSET or TASKSET as well as pinning IRQ interrupts

WAS Throughput with processor pinning

#9 – Keep Application Logging to a Minimum

Never should there be information outside of error cases being written to SystemOut.log
If using logging build your log messages only when needed

Good

if(loggingEnabled==true){ errorMsg = “This is a bad error” + “ “ + failingObject.printError();}

Bad

errorMsg = “This is a bad error” + “ “ + failingObject.printError();
If(loggingEnabled==true){ System.out.println(errorMsg); }

Keep error and log messages to the point and easy to debug
If using Apache Commons, Log4J, or other frameworks ensure performance on your system is as expected
Ensure if you must log information for audit purposes or other reasons that you are writing to a fast disk

#8 – Understand and Tune Infrastructure (databases & other interactive server systems)

WebSphere Application Server and the system it runs on is typically only one part of the datacenter infrastructure and it has a good deal of reliance on other areas performing properly.Think of your infrastructure as a plumbing system. Optimal drain performance only occurs when no pipes are clogged.
On the WAS system itself you need to be vary aware of

What other WAS instances (JVMs) are doing and their CPU / IO profiles
How much memory other WAS instance (or other OS’s in a virtualized case) are using
Network utilization of other applications coexisting on the same hardware

In the supporting infrastructure

Varying Network Latency can drastically effect split cell topologies, cross site data replication and database query latency

Ensure network infrastructure is repeatable and robust
Don’t take for granted bandwidth or latency before going into production always test as labs vary

Firewalls can cause issues with data transfer latencies between systems

On the database system

Ensure that proper indexes and tuning is done for the applications request patterns
Ensure that the database supports the number of connected clients your WAS runtime will have
Understand the CPU load and impacts of other applications (batch, OLTP, etc all competing with your applications)

On the database system

Ensure that proper indexes and tuning is done for the applications request patterns
Ensure that the database supports the number of connected clients your WAS runtime will have
Understand the CPU load and impacts of other applications (batch, OLTP, etc all competing with your applications)

On other application server systems or interactive server systems

Ensure performance of connected applications is up for the load being requested of it by the WAS system
Verify that developers have coded specific handling mechanisms for when connected applications go down (You need to avoid storm drain scenarios)

#7 – Minimize HTTP Session Content

High performance data replication for application availability depends on correctly sized session data

Keep it under 1MB in all cases if possible

Only should be storing information critical to that users specific interaction with the server
If composite data is required build it progressively as the interaction occurs

Configure Session Replication in WAS to meet your needs
Use different configuration options (async vs. synch) to give you the availability your application needs without compromising response time.
Select the replication topology that works best for you (DB, M2M, M2M Server)
Keep replication domains small and/or partition where possible

#6 – Correctly Tune Thread Pools

Thread pools and their corresponding threads control all execution on the hardware threads.
Understand which thread pools your application uses and size all of them appropriately based on utilization you see in tuning exercises

Thread dumps, PMI metrics, etc will give you this data
Thread Dump Memory Analyzer and Tivoli Performance viewer (TPV) will help in viewing this data.

Think of the thread pool as a queuing mechanism to throttle how many active requests you will have running at any one time in your application.

Apply the funnel based approach to sizing these pools

Example IHS (1000) -> WAS ( 50) -> WAS DB connection pool (30) ->
Thread numbers above vary based on application characteristics

Since you can throttle active threads you can control concurrency through your codebase

Thread pools needs to be sized with the total number of hardware processor cores in mind

If sharing a hardware system with other WAS instances thread pools have to be tuned with that in mind.
You need to more than likely cut back on the number of threads active in the system to ensure good performance for all applications due to context switching at OS layer for each thread in the system
Sizing or restricting the max number of threads a application can have can sometimes be used to prevent rouge applications for impacting others.

Default sizes for WAS thread pools on v6.1 and above are actually a little to high for best performance

Two to one ratio (threads to cores) typically yields the best performance but this varies drastically between applications and access patterns

TPV & TDMA tool snapshots

#5 –Tune JDBC Data Sources

Correct database connection pool tuning can yield significant gains in performance
This pool is highly contended in heavily multithreaded applications so ensuring significant available connections are in the pool leads to superior performance.
Monitor PMI metrics via TPV or others tools to watch for threads waiting on connections to the database as well as their wait time.

If threads are waiting increase the number of pooled connections in conjunction with your DBA OR decrease the number of active threads in the system
In some cases, a one-to-one mapping between DB connections and threads may be ideal

Frequently database deadlocks or bottlenecks first manifest themselves as a large number of threads from your thread pool waiting for connections
Always use the latest database driver for the database you are running as performance optimization in this space between versions are significant
Tune the Prepared Statement Cache Size for each JDBC data source

Can also be monitored via PMI/TPV to determine ideal value

#4 –Create Cells To Group Like Applications

Create Cells and Clusters of application servers with an express purpose that groups them in some manner
Large Cells (400-500-1000 members) for the most part while supported don’t make sense
Group applications that need to replicate data to each other or talk to each other via RMI, etc and create cells and clusters around those commonalities.
Keeping cell size smaller leads to more efficient resource utilization due to less network traffic for configuration changes, DRS, HAManager, etc.

For example, core groups should be limited to no more than 40 to 50 instances

Smaller cells and logic grouping make migration forward to newer versions of products easier and more compartmentalized.

#3 – Ensure Uniform Configuration Across Like Servers

Uniform configuration of software parameters and even operating systems is a common stumbling block
Most times manifests itself as a single machine or process that is burning more CPU, Memory or garbage collecting more frequently
Easiest way to manage this is to have a “dump configuration” script that runs periodically
Store the scripts results off and after each configuration change or application upgrade track differences
Leverage the Visual Configuration Explorer (VCE) tool available within ISA

Visual Configuration Explorer (VCE)

#2 – Correctly Tune The JVM

Correctly tuning the JVM in most cases will get you nearly 80% of the possible max performance of your application
The big area to focus on for JVM tuning is heap size

Monitor verbose:gc and target GCing no more than once every 10 seconds with a max GC pause of a second or less.
Incremental testing is required to get this area right running with expected customer load on the system
Only after you have the above boundary layers met for GC do you want to start to experiment with differing garbage collection policies

Beyond the Heap Size settings most other parameters are to extract out max possible performance OR ensure that the JVM cooperates nicely on the system it is running on with other JVMs
The Garbage Collector Memory Visualizer is an excellent tool tool for diagnosing GC issues or refining JVM performance tuning.

Provided as a downloadable plug-in within the IBM Support Assistant

Garbage Collection Memory Visualizer (GCMV)

#1 – Perform Proper Load Testing

Properly load testing your application is the most critical thing you can do to ensure a rock solid runtime in production.
Replicating your production environment isn’t always 100% necessary as most times you can get the same bang for your buck with a single representative machine in the environment

Calculate expected load across the cluster and divide down to single machine load
Drive load and perform the usual tuning loop to resolve the parameter set you need to tweak and tune.
Look at load on the database system, network, etc and extrapolate if it will support the full systems load and if not of if there are questions test

Performance testing needs to be representative of patterns that your application will actually be executing
Proper performance testing keeps track of and records key system level metrics as well as throughput metrics for reference later when changes to hardware or application are needed.
Always over stress your system. Push the hardware and software to the max and find the breaking points.
Only once you have done real world performance testing can you accurately size the complete set of hardware required to execute your application to meet your demand.

Additional Links

WebSphere Application Server Performance site
DeveloperWorks Article: Performance Tuning Case Study based on DayTrader - Step-by-step approach to tuning the application server based on a sample application
WebSphere Application Server Sample Performance Tuning Script - Can be used to adjust common tuning parameters based on predefined templates or customized to support additional fine tuning. Now available within v7.0.0

posted by Rohit Kelapure # 5:38 AM 0 Comments

Thursday, May 12, 2011

Dynacache cache entry priority questions

Smaller the size of the cache and more tuned to your working set, the better performance you will get. (Smaller footprint, better concurrency etc ... )

Q Should we still avoid timeouts as our invalidation policy if at all possible?
Absolutely. Explicitly invalidating the cache-entries brings down the cache size immediately and prevents LRU activity.

Q Our application's usage is currently assigning a fixed default "Priority" for each-and-every item placed in the Cache using the DynaCache API (using timeouts as our invalidation policy).
This is fairly normal. Priorities are typically used when there are different classes of cached items some more important than others. Lower priority cache entries are considered first when it comes to making room in the cache.

Q Default priority for each Cache item. Admin versus API... what wins?
Dynacache API priority overrides priority set on the admin console. Priority determines how many cycles in the clock algorithm must pass before an unused entry is chosen as a victim. Each entry's clock starts with the default or value set programatically/cachespec,xml and is decremented each clock cycle. A clock value of <= 0 implies a victim candidate. The default is 1.
I would keep the default priority of entries as 1 and only set higher priority for those entries that are more important than the rest.

posted by Rohit Kelapure # 5:50 AM 0 Comments

Tuesday, May 10, 2011

Dynacache Questions and Answers on Data Replication Service (DRS) replication best practices

Questions on Dynacache best practices vis a vis replication ...

Q We plan to use one or more replication domains that span a large number of servers with a number of object cache instances, and to use sharing policy PUSH.
Unfortunately PUSH does not scale. Go with NOT_SHARED.

Q We have observed that it takes a little more than one second for the replication to begin after a cache is being looked up the first time. We want to lookup all cache instances early at the server start up process, in order to fill the caches with existing data in the cluster, making it available to the application as soon as the data is requested.
In WAS v7 Dynacache has a JVM generic customer property called "com.ibm.ws.cache.CacheConfig.createCacheAtServerStartup" which when set to "true" will create cache instances automatically at server startup instead of the default on-demand behavior. This definitely should be set if deploying to WAS7. If on an earlier version of WAS look at technote http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21313480 .
See http://wasdynacache.blogspot.com/2010/04/dynacache-and-drs-replication-faq.html .

Q We are concerned about the push mechanism and whether the initial push of say 100-150 Mb existing data distributed over perhaps 30-40 cache instances into a new-started JVM could have (temporary) performance implications on one or multiple JVM:s that are up and running.
Your concern is warranted and that is why I said PUSH does not scale. When a new JVM starts up *ALL* the other JVMs send it cached data. So the same cached data is sent by ALL the running JVMs to the new one. If this is a lot of cache data, I have often seen the older JVMs that are pushing the data die due to OOMs. If you so desire you can completely disable bootstrap by calling DistributedMap.setDRSBootstrap(false). This will prevent any JVM from sending data to the upcoming JVM.

Q When data is pushed into a newborn cache, will all cached element come from the same JVM, or could some elements be pushed from JVM 1 and others from JVM 2
Unfortunately there is no workload distribution when sending data by the existing JVMs to the new one. We don't even optimize and have one JVM send the data, instead ALL the JVMs send all the cache data in that object cache instance to the new JVM.
Some customers write cache population scripts to populate the cache for each JVM after it comes up and configure the cache in the NOT_SHARED mode which only replicates invalidations to keep the cache consistent. Other option is to use PUSH with drs bootstrap disabled.

posted by Rohit Kelapure # 2:25 PM 0 Comments

How to detect a WebSphere Java classloader memory leak via the Eclipse Memory Analyzer tool

A presentation on Class Loading and debugging Class Loader memory leaks in WebSphere Application Server by Ian is an excellent treatise on this topic. Short summary follows:
1. Run "classloader explorer" query
2. Sort classloaders by name - identify the WAS CompoundClassLoaders
3. Examine each CompoundClassLoader object - according to WAS, those whose localClassPath field is set to the empty string should be eligible for collection.
4. For each CompoundClassloader whose localClassPath is the empty, string, run "Classloader->Path to GC roots->Exclude all weak/soft/phantom etc. references."
5. Review the reference chains for each classloader and analyse for possible culprits/owners.

Debugging WAS from Dumps: Diagnose more than leaks with Memory Analyzer and its IBM Extensions pleas is another great presentation to understand how to do JVM debugging

For bonus you can also look at On the Move? WebSphere JDK Migrations, Past, Present and Future... and Beyond

These links and presentations are from the WebSphere UK User Group
http://www.websphereusergroup.org.uk/wug/downloads/31/

posted by Rohit Kelapure # 1:40 PM 0 Comments

WebSphere Application Server - Performance considerations for deploying very large Java Enterprise Edition (JEE) applications

Please consider the following when developing and deploying very large applications (> 30,000 classes) bundled in one ear file.

The cost of having a large number of classes depends strongly on details of how the classes are present in the application, and whether Java EE5 features are enabled, and whether scanning is left enabled on all of the classes.

1) If the classes are exposed in a web module archive "WEB-INF/classes" directory, that will cause very large slowdowns. The converse is for the classes to be in a JAR, either in a simple utility JAR directly contained by the application archive (EAR file), or for the classes to be contained within an EJBJar, or for the classes to be contained in a JAR beneath a web module archive "WEB-INF/lib" directory.

2) If javaEE5 processing is enabled for the classes, either because the classes are present in a javaEE5 enabled EJBJar or WAR, that will cause large slowdowns. The converse is for the classes to be present in a location which will not be scanned, for example, if there is a "META-INF/application.xml" for the EAR, and if all of the modules use a pre-javaEE5 version, or if all of the modules are marked as metadata-complete.

3) Even if javaEE5 processing is enabled, if a large subset of the classes do not require annotation scanning, that can be disabled using special properties in the application and module "META-INF/MANIFEST.MF" manifest files.

4) If using WAS version 7, please upgrade to fixpack 7.0.0.13 or later. We put in some optimizations in 7.0.0.13 to make the reading and writing to disk of applications much more efficient. Large apps (like Commerce) that are over 100 MB in size deployed 30-40% faster.

5) Check if the application is EE5 (in the application.xml, look for the version= tag). If so, check if there are any WAR files inside the EAR that contain numerous JARs under WEB-INF/lib. If those are present they can increase deployment time significantly. There is a tuning option available to help, and it's documented here PK87053: JAVA ENTERPRISE EDITION VERSION 5 APPLICATIONS TAKE A LONG TIME TO DEPLOY http://www-01.ibm.com/support/docview.wss?uid=swg1PK87053

When deploying the application, there will be an unavoidable cost of copying the application files during deployment, which will be proportional to the total size of the classes (but not proportional to the number of classes, unless these are exposed in a WEB-INF/classes directory). If the average class size is, say, 3K, and if there are 30K classes, then that will put the application size at at least 90M, which will take time to expand and to copy into the deployment location. However, that cost is only during deployment.

Note of thanks to Thomas Bitonti and David W. Hare for providing this information

posted by Rohit Kelapure # 7:15 AM 0 Comments