All Things WebSphere

If you  deploy WAS on zOS and serve static (images, css files, static HTML) and dynamic content  (servlets, JSPs) from your application please stop and read this blog post.

Starting WAS 7 WebSphere Application Server for z/OS can be configured to use the Fast Response Cache Accelerator facility of the z/OS Communications

Server TCP/IP FRCA has been used for years inside the IBM HTTP Server to cache static contents like pictures or HTML files.

The high speed-cache can now be used to cache static and dynamic contents, such as servlets and JSP files, instead of using the WebSphere Application Server Dynamic Cache.

ROI Benefits of using FRCA to cache static and dynamic content  in a customer retail application to zOS customers can be seen in the two charts below:

Exploitation of FRCA is really as an extension to the existing Dynamic Cache (“DynaCache”) capability of each Application Server.FRCA is defined as an “External Cache Group,”and the “Adapter Bean”is what provides the function to access FRCA. Application servers > [server] > Dynamic cache service > External cache group.

In addition to enabling FRCA you will also need to write a cachespec.xml that tells Dynacache what to cache in the FRCA cache and drop it in the application's WEB-INF directory. A sample of this cachespec.xml  for caching ALL static content can be found here.
Detailed description on HOW to configure FRCA for WAS can be found in the Chapter 7 of WebSphere Application Server V7 Administration and Configuration Guide, SG24-7615.

Today's post is an answer from WebSphere Application Server SWAT Team member Kevin Grigorenko in response to the question  

1. Leverage the  Trace Request Analyzer plugin from ISA with the detection gap set to 1ms. Trace and Request Analyzer for WebSphere Application Server allows you to find delays and possible hangs from WebSphere trace files and HTTP plug-in traces by parsing call trees of methods and traces and calculating delays in each method and trace.

2. Take three or four javacores spread 20 seconds apart to see if threads are hanging in particular java operation or code paths. Javacores are the poor man's profiler to debug performance issues.

3. Use Request Metrics (you can start off at Hops and then, if necessary, Performance_debug). This will print a line to SystemOut.log for each request, information like the URL and user IP, and how long it took (and if you do component level, how long each component took, e.g. a servlet forward, etc. -- I think Hops will show database times since it's a boundary). This functionality is essentially what monitoring tools use (just as an agent instead of SystemOut.log). Often, ARM to SystemOut.log has a huge overhead, so you can use a filter if you have some idea of what might be causing the slow request. http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.nd.doc/info/ae/ae/uprf_rrequestmetrics.html

4. If IHS or an ODR or some other web server fronts WAS, then you can use something like LogFormat %D or %T to print the time each request takes to access.log and then dive into that slow URL from there. You can also use this to then filter ARM in #3. http://publib.boulder.ibm.com/httpserv/manual70//mod/mod_log_config.html

5. Use a custom Request Metrics agent, that can do, for example, a javacore when a request takes over a threshold, like 1 second -- this can essentially be extended to do what a monitoring product does -- i.e. avoiding the overhead of SystemOut.log -- let me know if you need the code.

6. Use a monitoring product like ITCAM or Wily Introscope or HP Open View.

In summary

There are multiple approaches to tackle this problem all of which involve a certain level of monitoring or logging. These approaches are detailed here
http://wasdynacache.blogspot.com/2011/06/todays-post-is-answer-from-kevin.html 
https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/entry/websphere_application_server_http_s_response_times8?lang=en

I suggest starting in the following order to understand why requests may be taking longer
1. Doing a review of existing PMI data
2. Taking  three or four javacores spread 20 seconds apart during the slow request to see if threads are hanging in particular java operation or code paths. Javacores are the poor man's profiler to debug performance issues.
3. Change log format at both the IHS and WAS tier to log response times in order to get response times for individual requests i.e. methods 0 and 4 in Kevin's blog post
4. IBM -Xtrace
5. Request metrics

There are different kinds of HTTP caches like browser caches, gateway caches and proxy caches.All these can be generically be termed as web caches. They are primarily used to reduce latency and network traffic.

There are two models of managing these HTTP caches i.e. Expiration and Validation. In expiration you specify how long a response should be considered “fresh” by including either or both of the Cache-Control: max-age=N or Expires headers. Caches that understand expiration will not make the same request until the cached version reaches its expiration time and becomes “stale”. The Validation model is used for dynamic web apps where changes in resource state occur frequently and unpredictably. Validation is used by servers and caches to communicate when an representation has changed. By using it, caches avoid having to download the entire representation when they already have a copy locally, but they’re not sure if it’s still fresh. This is done through the use of validators. The two most common validators are 1. the time that the document last changed, as communicated in Last-Modified header and 2. ETags which are unique identifiers that are generated by the server and changed every time the representation does. Because the server controls how the ETag is generated, caches can be surer that if the ETag matches when they make a If-None-Match request, the representation really is the same.

Dynacache does NOT provide support for  HTTP Cache-Control headers. This support is provided by the webcontainer on the server side and the WebSphere Proxy Server, ODR and IHS on the edge tier. Dynacache has been build ground up for page fragment caching which does not fit well with the principles of HTTP caching. The Cache-Control headers that control HTTP Caches apply to the entire response and NOT fragments. Dynacache will not look in the cache and automatically set Cache-Control headers, nor do a conditional GET to return a 304.

Static HTTP Cache-Control caching works really well for static resources like static HTML, jpg, gif, etc. These static resources can be cached in DynaCache using the static caching policy class. On WAS7 zOS the FRCA cache is an excellent place to stash your static content.  The real advantage of caching using HTTP Cache-Control headers is that the cache is distributed across tiers i.e. between one or more Web servers; however the disadvantage is that the cache cannot be purged effectively on demand.

WAS 8 Beta has ended and WAS 8.0 is now generally available for production and support.
Thank you for your participation and support.


Today June 17 2011, we eGA three critical products containing key new function, innovation, and performance enhancements. WAS V8, Compute Grid V8, and the Web 2.0 and Mobile Feature Pack 1.1 all deliver key function at a critical time for customers seeking to transform their business models for increasing growth and agility.Please note that since  the IBM® WebSphere® Application Server Version 8.0 has now been officially released. You may obtain a trial version, free of charge, from here. You will also find information on how to purchase this product, as well as free trial support.

WAS V8 is the fastest, highest quality application server that is the easiest to migrate to, install  and service.
See technical announcement on http://webspherecommunity.blogspot.com/
For production and support please see http://www-01.ibm.com/software/webservers/appserv/was/features
For a free developer version please see http://www.ibm.com/developerworks/downloads/ws/wasdevelopers/index.html
For a trial version please see http://www.ibm.com/developerworks/downloads/ws/was/