Concerns and issues relating to all versions of WebSphere Application Server

WebSphere Application Server version 8 zOS startup performance

2012-02-13T08:02:00.000-08:00

Two APARs all WebSphere Application Server zOS 8002 customers should consider applying -

PM51889: DECREASE CLUSTER STARTUP TIME.

****************************************************************
* USERS AFFECTED: All IBM WebSphere Application Server V8.0 users of Base and Network Deployment editions.
****************************************************************
* PROBLEM DESCRIPTION: Excessive JIT compilation increases the start time of servers and clusters
****************************************************************
* RECOMMENDATION:
****************************************************************
The Java JIT compiler provides a performance boost by compiling frequently used classes in such a way that future invocations of those classes perform better. However, the initial compilation of these classes takes additional time. For classes that are used often throughout the life of the server, it is worth taking the initial time to compile them in order to obtain future performance benefits.During server startup, numerous classes are loaded that will rarely be invoked after startup is complete. Compiling all of these classes is often unnecessary and can increase server start time. The JIT compiler should be optimzied to avoid unnecessary compilations during server startup.
Problem conclusion
WebSphere Application Server was modified to inform the JIT compiler of the beginning and end of server startup. The JIT compiler can use this information to make better decisions about which classes to compile, thereby improving the start time of servers and clusters.

Fix on 8002: http://www.ibm.com/support/fixcentral/quickorder?product=ibm/WebSphere/WebSphere+Application+Server&fixids=8.0.0.2-WS-WAS-IFPM51889&source=myna&myns=swgws&mynp=OCSSEQTP&mync=E

The fix for this APAR is currently targeted for inclusion in fix pack 8.0.0.3. Please refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

PM55170: CPU CONSTRAINED DMGR + NODE STARTUP 95% SLOWER

****************************************************************
* USERS AFFECTED: All IBM WebSphere Application Server V8.0
* users of Base and Network Deployment
* editions.
****************************************************************
* PROBLEM DESCRIPTION: Excessive JIT compilation increases
* the time to federate nodes.
****************************************************************
* RECOMMENDATION:
****************************************************************
The Java JIT compiler provides a performance boost by compiling frequently used classes in such a way that future invocations of those classes perform better. However, the initial compilation of these classes takes additional time.For classes that are not used often throughout the life of the addNode command, it is not worth taking the initial time to compile them as the JVM shuts down after the command completes. Compiling all of classes is unnecessary and can increase the time to federate a node. The JIT compiler should be optimzied to avoid unnecessary compilations during node federation.
Problem conclusion
WebSphere Application Server was modified to inform the JIT compiler of the beginning of the addNode command. The JIT compiler can use this information to make better decisions
about which classes to compile, thereby improve the federate a node.

The fix for this APAR is currently targeted for inclusion in fix pack 8.0.0.3.
Fix on 8002: http://www.ibm.com/support/fixcentral/quickorder?product=ibm/WebSphere/WebSphere+Application+Server&fixids=8.0.0.2-WS-WAS-IFPM55170&source=myna&myns=swgws&mynp=OCSSEQTP&mync=E

Websphere logging quick tip

2012-02-04T23:04:00.001-08:00

You can make the WebSphere for z/OS logs look the same as Distributed by setting the following nifty Java system property: com.ibm.ws.dist.trace.format.on.zos=true

Notes:
* To use this property in V7.0 you must be at or above service level 7.0.0.11. Using this property at an older service level will cause a NullPointerException.
* This property only affects the SYSPRINT DD of WebSphere for z/OS output, the remaining DDs are unaffecte

Source:

WebSphere Proxy Server Best Practice

2012-02-01T11:26:00.000-08:00

When configuring the WebSphere Proxy Server for caching it is a best practice to ENABLE disk offload & set the location to a dedicated disk partition. Even if you do not have a disk partition consider enabling disk offload for your static and dynamic cache.

Class loading and debugging class loader memory leaks in WebSphere Application Server

2012-01-31T09:41:00.001-08:00

See this excellent presentation on Debugging Classloader leaks in application servershttp://www.websphereusergroup.org.uk/wug/files/presentations/31/Ian_Partridge_-_WUG_classloader_leaks.pdf

WebSphere Classloader Memory Leak Prevention

2012-01-31T09:30:00.000-08:00

Solving Application ClassLoader Leaks

Applications tend to want to:

Start new threads using Runnable implementations from the application class loader. Even though the JEE programming model does not support this, customers frequently either directly create new threads or indirectly create them by using Timers. Customers must ensure that these threads are stopped when the corresponding application (or WAR module) is stopped:
- javax.servlet.ServletContextListener.contextDestroyed can be used to be notified when a WAR is being stopped in order to clean up. Note that WARs can be stopped independently from an application when class reloading is enabled.
- javax.ejb.Singleton, javax.ejb.Startup, and javax.annotation.PreDestroy annotations can be used to be notified when an EJB module is being stopped in order to clean it up. Note, singleton EJBs are only available in EJB 3.1 (WAS v8).
- Startup beans can be used to be notified when an EJB module is being stopped in order to clean up. All EJB modules are stopped when an entire application is being stopped.
Use ThreadLocal (storing a ThreadLocal in a static). ThreadLocal values are effectively stored as WeakHashMap in each Thread. Since the values typically include application objects, the application object references its Class, which references its ClassLoader, which references the Class contain the ThreadLocal, the weak reference is never broken, and a leak occurs.
Customers are encouraged either to avoid the use of ThreadLocal, to clear references to the ThreadLocal when the module is stopped (see above), or to ensure that remove() is called after every request.
Register JMX MBeans or NotificationListener with the JMX server. Customers must ensure that these are unregistered when the corresponding application (or WAR module) is stopped.

Arbitrary components

This includes JDBC providers, third-party software, and applications themselvestend to want to:

Start new threads, including the "timer threads" created by the java.util.Timer constructor. When a Thread is created, two pieces of information are copied from the primordial thread:
1. The context class loader (getContextClassLoader()). When an application is executing, the containers set the context class loader to the module class loader, so the newly created thread will keep the context class loader alive for the duration of its existence. This can be avoided by calling setContextClassLoader to a non-application class loader prior to starting the timer and then resetting it afterwards.
2. The AccessControlContext of the calling thread (as documented by AccessController). If the thread is being started due to an API call from an application, then the application's ProtectionDomain will be in theAccessControlContext, and the ProtectionDomain of an application class will include a reference to its ClassLoader. This can be avoided by creating the thread using doPrivileged. Note that care must be taken to ensure that the use of doPrivileged does not allow unprivileged applications to create threads.
For example:
```
// doPrivileged fixes the AccessControlContext leak, and it is also required

// for calls to Thread.get/setContextClassLoader.

Timer timer = AccessController.doPrivileged(new PrivilegedAction() {

  public void run() {

    Thread thread = Thread.currentThread();

    ClassLoader savedCL = thread.getContextClassLoader();

    thread.setContextClassLoader(null);

    try {

      // The Timer constructor will create a Thread, which will copy the

      // context class loader from the current thread, which is now null.

      return new Timer(true);

    } finally {

      thread.setContextClassLoader(savedCL);

    }

  }

});
```
Associate data with the current context class loader. This is typically done via a MapValue>. This Map must either:

Have explicit lifecycle APIs. In this case, the lifecycle API must be called. If the code was introduced by a customer, then the customer is responsible for adding a JMX listener. If the code was introduced by a WAS prereq, then the owner must use a WAS application listener API. If the code belongs to the JDK, then the runtime team will accept responsibility for calling it (e.g., ResourceBundle.clearCache andIntrospector.flushCaches).
Be a WeakHashMap to allow the ClassLoader key to be garbage collected. Note that the value must not hold a non-weak reference to classes or objects created from that ClassLoader, or the entry will never be removed. Either WeakHashMap<ClassLoader, WeakReference<Class>> or WeakHashMap<ClassLoader, Tuple>, where Tuple contains WeakReference<Class> and WeakReference<Object> to an object instantiated from the class. In both cases, the Class is held weakly, which will still allow the ClassLoader to be collected. In the latter case, the reference to the instantiated object will be cleared if a GC occurs, but the assumption is made that it can be cheaply reinstantiated.

These tips are courtesy of WAS guru Brett Kail

New Features in WebSphere Application Server version 8.0.0.2

2012-01-23T09:06:00.000-08:00

WAS 8002 was available for general availability on 1/16/2012. The blog post below provides the necessary links for download and an overview and brief detail of the new features in WebSphere Application Server version 8.0.0.2. Please note that a fix pack contains both features and APAR fixes.
WAS 8002 & other offerings download --> http://www-01.ibm.com/support/docview.wss?uid=swg24031368
Recommended Fixes Page --> http://www-01.ibm.com/support/docview.wss?rs=180&context=SSEQTP&uid=swg27004980#ver80
Readme --> http://www-01.ibm.com/support/docview.wss?uid=swg27023570
FixList 8002 -> http://www-01.ibm.com/support/docview.wss?uid=swg27022958
z/OS --> http://www-01.ibm.com/support/docview.wss?rs=404&uid=swg27006970

Improvements in WAS 8002 =>

Education on ALL these new features is provided by the IBM Education Assistant here
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.was_v8/was/WASv8.0.0.2.html

Customized access logging for WAS
Before V8.0.0.2 the access log only offered two options for format, common and combined. With the new access log enhancements added in V8.0.0.2 you can customize how much information, and what information, is displayed. 17 directives were added to display different pieces of information in the access log. These directives, allow for any order the user wants and each directive can be quoted. These access log enhancements provide greater customization and more information on display to the user.

Example:
%h %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" "%C"
Remote host, URL path not including query string, start time of request (NCSA format), method, path, query-string, and protocol, .....

- Directives must be separated by spaces

- Individual directives can be surrounded by quotes

- Directives can be ordered however the user wishes

- Unless noted, all directives will print - if no value is found

� Information on access log functionality:
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/utrb_httperrlogs.html
� Used as a base for the new functionality:
http://publib.boulder.ibm.com/httpserv/manual70//mod/mod_log_config.html

Tools from the Virtual Member Manager to clear adapter cache, manage realm default parent & retrieve the data model

The Clear Adapter Cache feature is needed to ensure that VMM always returns correct data. Changes made to the existing Get and Search commands. New control added to be used with the existing Get and Search API. Two new commands added for handling the cache of the adapters.
New commands for set, update, delete, list of realm level default parent. Simplifies configuration and management. Commands can be used in server and local mode
VMM data Model is a description of the organization of data in a manner that reflects the information structure supported by VMM. VMM allows to retrieve the different kinds of information related to its data model schema. To retrieve the VMM schema there is currently a supported API. There is no command used from the command line to retrieve the schema.This feature adds new commands to retrieve the different VMM data model that can be used in server and local mode.

http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.0.0.2/Tools.html?dmuid=20111105095219725033

Fixing the Performance problem with the Service Data Object (SDO) Repository

The feature focuses on enhancing the performance of the SDO repository. This is done by
changing the way in which the data is accessed by the entity beans. Finder methods fetches the entire objects even if you are interested in just one column in the table, these are exposed to the clients. Replacing the finder methods with the ejbSelect methods and exposing them to client interface enhances the performance by several factors.
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.0.0.2/Performance/WASv8002_SDO_Performance.pdf?dmuid=20111107132159204845

sibDDLGenerator enhancements for SIBus -
Before these enhancements, the sibDDLGenerator command created the CREATE DATABASE statement DDL and created the tablespaces listed here. If the same database had to be shared by multiple messaging engines, it was necessary to make some changes to the DDLs generated by the sibDDLGenerator command:
remove the CREATE DATABASE statement DDL, and modify the tablespace names. As part of the enhancements to the sibDDLGenerator, two new parameters (-createdbstmt, -tablespaceprefix) are added. These parameters remove the need for changes to the DDLs generated by the sibDDLGenerator command .
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.0.0.2/Architecture/WAS_8002_SIB_sibDDLGenerator.pdf?dmuid=20111107132412229085&noframes=true

Installation changes and enhancements in IBM WebSphere Application Server V8.0 for the z/OS platform
Clarification of fix pack delivery mechanism (PTFs) to ensure fixed service footprint and Support for Installation Manager console mode
https://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.0.0.2/zOS/WAS_8002_zOS_Installation.pdf?dmuid=20111116163149641261

How to create a non-transactional JDBC connection in WAS

2012-01-23T08:38:00.000-08:00

The ability to create a non-transactional datasource was introduced in WAS 7.0, and thus unavailable in any earlier release.

Answer: Courtesy of Fred Rowe =>

In the admin console, navigate to "Resources | JDBC | Data sources"
Select the desired datasource from the list.
Click on the "WebSphere Application Server data source properties" link to the right of the screen under the "Additional Properties" group.
Check the "Non-transactional data source" box.
Restart the server.

IBM JVM and JDK -Xdump and -Xtrace options for scenario based problem determination

2011-12-28T16:01:00.000-08:00

There are multiple ways of configuring the IBM JDK shipped by WebSphere Application Server to generate WebSphere and JDK trace in situations where you have no control of the application or the server runtime.

Scenario 1:
In a long running server when an exception occurs say a javax.transaction.RollbackException, we need to enable a particular WebSphere runtime trace for a period of 1 hour to collect pertinent logs. This allows for collection for logs ONLY and IF a critical incident occurs in long run and production deployment scenarios.

Using the JVM dump engine ability to run command lines based on events, you could trigger a command line on, say, the uncaught javax.transaction.RollbackException:
-Xdump:tool:events=uncaught.filter="javax/transaction/RollbackException",exec="command line to run"
Where we could then use wsadmin to change the trace settings:
exec="sh /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -lang jython -f setWASTrace.py"

For (setWASTrace.py) scripting with WebSphere Application Server see this blog post.

Scenario 2:
Print a thread stack trace whenever the method com.ibm.ws.util.ThreadPool$Worker.setStartTime method is called
-Xtrace:print=mt,methods={com/ibm/ws/util/ThreadPool$Worker.setStar tTime},trigger=method{com/ibm/ws/util/ThreadPool$Worker.setStartTime,jstack trace}

Scenario 3:
Print JVM entry and exit trace in clear text whenever any code is executed in the sun.net.www.protocol.http and sun.net.www.http packages
-Xtrace:methods={sun/net/www/protocol/http*,sun/net/www/http*},print=mt

Scenario 4
Take Javacores when CPU Starvation is detected. This will write to a file named cpu_starvation%pid%.txt in the current working directory of the JVM (i.e. the same directory as the javacores) when CPUStarvationDetectedException is thrown

-Xdump:tool:events=throw,range=1..0,filter=com/ibm/loglistener/CPUStarvationDetectedException,exec="cmd /c c:\cpu_starvation.bat end %pid"

Perl script for finding invalidations in Dynacache Trace

2011-12-20T18:40:00.000-08:00

Dynatrace.pl - Tool for finding invalidations in Dynacache Trace

This tool searches Dynacache Trace for invalidations and generates two CSV files.

Pre-req

This tool requires Perl. ActivePerl can be installed on Windows. The DateTime package is required.

The log to be parsed needs to contain Dynacache Tracing.

Usage

C:\>dynatrace.pl

usage: C:\dynatrace.pl [OPTIONS] FILE

Trace should contain this trace:
com.ibm.ws.cache.*=all:com.ibm.ws.drs.*=all

Tool looks for this line:
Cache.remove() cacheName=baseCache id=Customers cause=1 source=2

Filtering Options:

-i | --instance CACHE_INSTANCE
Tool uses the contains reg ex

-c | --cause CAUSE
Use either number of string
   0    UNSET
   1    DIRECT
   2    LRU
   3    TIMEOUT
   6    INACTIVITY
   7    DISK_GARBAGE_COLLECTOR
   8    DISK_OVERFLOW

-s | --source SOURCE
   1    MEMORY
   2    REMOTE
   3    DISK
   4    NOOP
   5    LOCAL

Output:
- FILE.inv.csv
- FILE.inv.nonum.csv

FILE.inv.nonum.csv can be used for grouping

Example Usage

C:\>dynatrace.pl --source REMOTE trace.log
-----> Grepping file: trace.log
-----> Filtering by source=REMOTE
-----> Processing: trace.log
-----> Writing trace.inv.csv
-----> Writing trace.inv.nonum.csv
-----> Log file format: Default - short
-----> Process trace.log file. Found 3443 invalidations
-----> Done

Download

Download the tool from here ---------> Dynatrace.pl

Oracle’s Newly Unveiled WebLogic 12c Server

2011-12-20T11:03:00.000-08:00

Oracle WebLogic 12 c (c stands for cloud) was announced on December 1st 2011. I would like to use this blog post to differentiate between WAS and the new launched WebLogic 12c. Please take an analytic look at the Oracle WebLogic Server 12c announcement here. This article puts to rest any confusion regarding differences between WAS and WebLogic. For a comparison of which is faster WebSphere or WebLogic see.

Defensive queuing for website surge protection

2011-12-20T10:29:00.000-08:00

Today's blog post explains a deployment and tuning methodology to protect your application server from a traffic surge.

In the figure below decreasing values are assigned to the tuning parameters for each of the displayed Web site components. This deliberate reduction in the maximum number of available resources assigned to each successive layer is called the Funnel model methodology. The benefit of the funnel tuning model is that we
want as many clients to connect to our system as possible, but without overwhelming the resources in each of the layers downstream (for example, database connections). The funnel helps us place these requests into various queues at each layer, where they will wait until the next layer has the capacity to process them. In summary, the funnel model helps us handle bursts of client

Queue tuning points in a Java EE Web site: the Funnel model

For some high level guidance and theory see. The proportion and ratios of the various queues to various another is very important in determining the queuing that occurs at load. If every incoming request requires a connection to a back-end database then the datasource connection pool should be at least equal in size to the servlet thread pool.

What is the size of the WebContainer threadpool in relation to the connection pool ?
What should be the IHS thread/process configuration in relation to the WebContainer threadpool ?

Answer from SWAT guru Kevin Grigorenko -->

The ideal proportion is 1:1 with all wait time on the network, but that assumes all layers perform and scale the same, which is rarely (if ever) the case, so only the general recommendation of 1:<=1 is made. Also, requests are not all the same, so one request may use more memory on a thread than others.
In general, I recommend customers "play it safe" -- for example, start at the max database concurrency, N, then set WAS WebContainer to 1.5N (could go more since a lot of times requests are just for static content) and then set IHS to 2N.
But a lot of stress testing is required at the boundary of the incoming queues -- the biggest problem is that customers configure the front queue (e.g. IHS) at N but then don't even create a test that sends N concurrent requests in. When that happens (spike in load, DOS attack, etc.), their JVMs or database crash/OOM.

Using Java Server Faces 2 with WebSphere Application Server 7

2011-12-20T10:03:00.000-08:00

If you want to use newer versions of Java Server Faces i.e. JSF 2.x with WebSphere Application Server 7 or WebSphere Application Server 6.1 take a look at this blog post MyFaces 2.0 and WebSphere Application Server v7

Hottest items in the WebSphere Application Server Dynacache Servlet cache

2011-11-29T10:24:00.000-08:00

Provided is a DynaCache program that leverages PMI and JMX to output the top 100 templates in the servlet cache in terms of in-memory cache hits.
You can use this client program to determine which items in the cache are the most popular.

When there are a lot of cache-entries and consequently templates in the cachespec.xml this tool will give you a good idea of what is hot or not. The utility of this tool decreases when the cache-entry names are not granular. For instance some cachespec.xml's use the single front controller, with a lot of the cache-ids hanging off a usually a single cache-entry. For now the program/tool only looks at the in-memory cache hits and not disk hits. Java src is included in the jar if you want to modify it :-)

Attached jar contains the src and the class files:
http://dl.dropbox.com/u/12872475/Dynacache-WXS/dc-pmi.jar

This program requires the template servlet cache PMI group to be enabled.

Usage:
C:\WebSphere7\AppServer\java\bin\java -Djava.ext.dirs=C:\WebSphere7\AppServer\runtimes;c:\WebSphere7\AppServer\lib;C:\WebSphere7\AppServer\java\jre\lib;C:\WebSphere7\AppServer\java\jre\lib\ext;C:\WebSphere7\AppServer\lib com.ibm.ws.sample.DynaCachePMIClient localhost 8881-standalone smitaNode01 server1 baseCache

Replace C:\WebSphere7\AppServer with your $WAS_HOME &rerun.
Program arguments are in orange. 
There is also a test.properties that can be modified inside the jar for more granular control, if you do not want to provide arguments.

Output:

host=localhost , port=8881 , connector=SOAP , distributed=false
main: create the adminclient
getAdminClient: hostStr=localhost , portStr=8881
getAdminClient: WasClientPath=C:/WebSphere7/AppServer/profiles/base
*****************************************
DynaCache Automated Tests
Host     : smita.raleigh.ibm.com
Port     : 9081
PMI Port : 8881 connector: SOAP
Cell     : smitaNode01Cell
Node     : smitaNode01
Server   : server1
BaseCacheSize: 2000
Offload to Disk: false
IhsEnabled: true
globalSecurityEnabled: false
IHS Host    : localhost
IHS Port    : 80
Distributed Caching: false
zOSMode: false
*****************************************
Usage: DynaCachePMIClient [  <-distributed|-standalone>   ]
Arguments passed in to com.ibm.ws.sample.DynaCachePMIClient
args[0] localhost
args[1] 8881
args[2] -standalone
args[3] smitaNode01
args[4] server1
args[5] baseCache

Found 1
...node:smitaNode01 match: true
...process:server1 match: true
my perf MBean: WebSphere:name=PerfMBean,process=server1,platform=dynamicproxy,node=smitaNode01,version=7.0.0.19,type=Perf,mbeanIdentifier=PerfMBean,cell=smitaNode01Cell,spec=1.0

Get statistics from all cacheModules

stats name=cacheModule.template
    HitsInMemoryCount[21]=23
    HitsOnDiskCount[22]=0
    ExplicitInvalidationCount[23]=0
    LruInvalidationCount[24]=0
    TimeoutInvalidationCount[25]=0
    InMemoryAndDiskCacheEntryCount[26]=4
    RemoteHitCount[27]=0
    MissCount[28]=4
    ClientRequestCount[29]=27
    DistributedRequestCount[30]=0
    ExplicitMemoryInvalidationCount[31]=0
    ExplicitDiskInvalidationCount[32]=0
    LocalExplicitInvalidationCount[34]=0
    RemoteExplicitInvalidationCount[35]=0
    RemoteCreationCount[36]=0

    stats name=/dynacachetests/TimeStamp
        HitsInMemoryCount[21]=11
        HitsOnDiskCount[22]=0
        ExplicitInvalidationCount[23]=0
        LruInvalidationCount[24]=0
        TimeoutInvalidationCount[25]=0
        InMemoryAndDiskCacheEntryCount[26]=2
        RemoteHitCount[27]=0
        MissCount[28]=2
        ClientRequestCount[29]=13
        DistributedRequestCount[30]=0
        ExplicitMemoryInvalidationCount[31]=0
        ExplicitDiskInvalidationCount[32]=0
        LocalExplicitInvalidationCount[34]=0
        RemoteExplicitInvalidationCount[35]=0
        RemoteCreationCount[36]=0

    stats name=/dynacachetests/TimeStamp1
        HitsInMemoryCount[21]=12
        HitsOnDiskCount[22]=0
        ExplicitInvalidationCount[23]=0
        LruInvalidationCount[24]=0
        TimeoutInvalidationCount[25]=0
        InMemoryAndDiskCacheEntryCount[26]=2
        RemoteHitCount[27]=0
        MissCount[28]=2
        ClientRequestCount[29]=14
        DistributedRequestCount[30]=0
        ExplicitMemoryInvalidationCount[31]=0
        ExplicitDiskInvalidationCount[32]=0
        LocalExplicitInvalidationCount[34]=0
        RemoteExplicitInvalidationCount[35]=0
        RemoteCreationCount[36]=0

TOP 100 TEMPLATES IN THE CACHE baseCache
 /dynacachetests/TimeStamp1=12
 /dynacachetests/TimeStamp=11

Dynacache PMI structure:

How to tell if a response from WebSphere Application Server has been cached

2011-11-09T16:05:00.000-08:00

An application that uses Dynacache servlet caching in the past had no indication in the servlet response if data came from the cache or not.

Starting with WAS version 7.0.0.11 in APAR PM01156, Dynacache adds a header to the response that can be used to determine if the servlet response has been returned from Dynacache. The name of this header is "CACHED_RESPONSE"

An application may need to take different actions depending on if the data has been cached.
The header CACHED_RESPONSE has been added to the header of the servlet response and set to true when the servlet output is obtained from the cache.

ApacheCon 2011 Web Technologies in Java EE6 : Servlet 3.0 and JSF 2.0 Whats New ?

2011-11-09T15:24:00.000-08:00

Here are the slides from my ApacheCon 2011 talk on web EE technologies JSF 2.0 and Servlet 3.0

http://dl.dropbox.com/u/12872475/Servlet_JSF_apachecon_2011.pdf

Servlet 3.0 is a new specification that is part of the Java EE 6 technologies. This session will introduce you to the new features of Servlet 3.0 and explain how you can leverage them in your applications. The session will focus on two major themes: ease of development, and improving application scalability. This session is intended for Java EE developers, administrators, and architects.

JSF 2.0 is a new specification that is part of the Java EE 6 technologies. This session will introduce you to the new features of JSF 2.0 and explain how you can leverage JSF in your Java EE and Portlet applications. The session will focus on three major themes: ease of development, performance improvements, and open source technology adoption. You will learn how the tools for the Application Developer make developing UI easier, as well as the JSF-Dojo component library. This session is intended for Java EE developers, administrators, and architects.

Dynacache Servlet caching cachespec.xml example

2011-11-06T14:38:00.000-08:00

Often customers ask me for complicated and non trivial examples of a servlet cache specification using Dyncache servlet caching. I have uploaded one on github that we use for our testing.

You can copy paste this cachespec.xml in your application's WEB-INF directory to enable servlet caching of your responses in WebSphere Application Server.

How to get Contexts & Dependency Injection (JSR 299) running on WebSphere Application Server 7.0.0.x

2011-11-03T05:37:00.000-07:00

Please understand that even with these changes you are losing a lot of WAS value-add and integration of Dependency Integration with Java EE resources in WAS i.e. pojo injection in a servlet/EJB or vice versa. IBM cannot help with support calls on WAS 7 when open web beans or JSF 2 is bundled as part of the application. For dependency injection in WAS we recommend upgrading to WAS 8 which has CDI and JSF2 support baked in.

1. OpenWebBeans (OWB) Download

Register for JIRA and watch OpenWebBeans project here https://issues.apache.org/jira/browse/OWB
Versions https://issues.apache.org/jira/browse /OWB#selectedTab=com.atlassian.jira.plugin.system.project%3Aversions-panel
Latest version 1.1.2 Release Date: 21/Oct/11 OpenWebBeans-1.1.2
Distribution: https://repository.apache.org/content/repositories/releases/org/apache/openwebbeans/openwebbeans-distribution/1.1.2/
Download and unzip https://repository.apache.org/content/repositories/releases/org/apache/openwebbeans/openwebbeans-distribution/1.1.2/openwebbeans-distribution-1.1.2-binary.tar.gz
to C:\WebSphere7\AppServer\temp where C:\WebSphere7\AppServer is my WAS_INSTALL_ROOT
C:\WebSphere7\AppServer\temp was chosen as an arbitrary location for download. You can do this in any directory.
After unzip you should see C:\WebSphere7\AppServer\temp\openwebbeans-distribution-1.1.2

2. Build a sample that comes with the OWB Distribution

In this scenario I will be building the jsf2sample application.
This examples shows the usage of MyFaces JSF2 + OpenWebBeans .
Before building the application add org.apache.myfaces.webapp.StartupServletContextListener
to the web.xml of your application if you intend to use JSF2.0. This is how the listener stanza should look like

<listener>
        <listener-class>org.apache.webbeans.servlet.WebBeansConfigurationListener</listener-class>
        <listener-class>org.apache.myfaces.webapp.StartupServletContextListener</listener-class>
    </listener>

For this to work you will need to add
<profiles>
    <profile>
      <id>apache</id>
      <pluginRepositories>
        <pluginRepository>
          <id>apache.snapshots</id>
          <name>Maven Plugin Snapshots</name>
          <url>http://repository.apache.org/snapshots/</url>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
</profiles>

<activeProfiles>
    <activeProfile>apache</activeProfile>
</activeProfiles>

to your global settings in M2_INSTALL_HOME\conf\settings.xml or user settings from M2_USER_HOME\.m2\settings.xml . To determine these settings locations type "mvn -X"

You may need to change the pom.xml of the sample application
Replace ../pom.xml in with
https://repository.apache.org/content/repositories/releases/org/apache/openwebbeans/openwebbeans/1.1.2/openwebbeans-1.1.2.pom/pom.xml
You may also need to delete myfaces.version xml element from   org.apache.myfaces.core ...
After building the jsf2sample CDI you should see C:\WebSphere7\AppServer\temp\openwebbeans-distribution-1.1.2\samples\jsf2sample\target\jsf2sample.war

3. Configure WAS 7 to use JSF 2.0
Because WAS 7 comes with its own implementation of JSF we have to follow the steps below for the jsf2sample to leverage JSF 2.0 myfaces correctly.

3.1 Extract
commons-beanutils-1.8.3.jar,
commons-codec-1.3.jar,
commons-collections-3.2.jar,
commons-digester-1.8.jar,
commons-logging-1.1.1.jar,
myfaces-api-2.0.3.jar,
myfaces-impl-2.0.3.jar from the WEB-INF/lib dir of the application to a common place on your server where you want to store them.
    Say ${WAS_INSTALL_ROOT}\optionalLibraries\Apache\MyFaces..

3.2 Delete the 7 jar files listed above from the WEB-INF/lib dir of the app
3.3 Open up the myfaces-impl-2.0.x.jar file and remove the files in the META-INF\services\ directory.
3.4 Install js2sample.war application on WAS 7

3.5 Configuring WAS v7 shared libraries
Next, we configure the JAR files as Isolated Shared libraries.

Add the following to Environment > Shared Libraries > New

Name
OWB

Classpath
${WAS_INSTALL_ROOT}/optionalLibraries/IBM/JSFProviders/WebSphere-MyFaces20-annotation-provider.jar (this is needed and only available on & after 7.0.0.19)
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/commons-beanutils-1.8.3.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/commons-codec-1.3.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/commons-collections-3.2.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/commons-digester-1.8.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/commons-logging-1.1.1.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/myfaces-api-2.0.3.jar
${WAS_INSTALL_ROOT}/optionalLibraries/Apache/owb/myfaces-impl-2.0.3.jar

Click on the check box for ‘Use an isolated class loader for this shared library’. Click Apply and Save.

3.6 Associate the shared library with your
Go to Enterprise Applications > jsf2sample_war > Shared library references and add the OWB shared library reference
to both web module

3.7 If you want at this point you can enable trace for *=info:org.apache.webbeans*=all to get more information about the OWB
and WAS interaction

3.8 Restart the application server

4. Confirmation

After the server has started look for these messages to validate that your CDI application has started
[11/2/11 15:53:57:141 EDT] 00000008 WebContainerL 2 org.apache.webbeans.web.lifecycle.WebContainerLifecycle debug Deploying scanned beans.
[11/2/11 15:53:57:250 EDT] 00000008 BeansDeployer 2 org.apache.webbeans.config.BeansDeployer debug Found Managed Bean with class name : [org.apache.webbeans.jsf2.HelloJsfBean]
[11/2/11 15:53:57:391 EDT] 00000008 WebContainerL I org.apache.webbeans.web.lifecycle.WebContainerLifecycle info OpenWebBeans Container has started, it took [313] ms.
[11/2/11 15:53:59:860 EDT] 00000008 servlet I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: [jsf2sample_war] [/jsf2sample] [Faces Servlet]: Initialization successful.

5. Final Validation

Hit
   - http://localhost:9081/jsf2sample/counter.xhtml
     Shows the AJAX functionality of JSF2 specification.

   - http://localhost:9081/jsf2sample/conversation.xhtml
     Show the usage of JSF conversations.

where 9081 is the WC_defaulthost port.

WebSphere Application Server 8.5 alpha overview session at ApacheCon 2011 in Vancouver BC Thursday November 10

2011-10-31T13:47:00.000-07:00

If you are in Vancouver on Thursday November 10 and interested in listening about the new faster and lighter footprint version of WebSphere Application Server 8.5 please come to MacKenzie Room (1st Floor) at 7PM at theThe Westin Bayshore Vancouver 1601 Bayshore Drive, Vancouver, BC, V6G 2V4.

Please note that this session is open to ALL users and customers not just ApacheCon attendees.

See announcement here
http://na11.apachecon.com/posts/32726152
https://twitter.com/#!/ApacheCon/status/131098166429892609

This will be a presentation and hands-on demo followed by a Q&A session run by Eric Covener and Rohit Kelapure.

Hope to see you there!!

WebSphere Application Server Mobile, dependency injection, migration and coexistence strategy

2011-10-30T09:50:00.000-07:00

From a mobile perspective , the new WebSphere Application Server (WAS) Feature Pack for Web 2.0 and Mobile V1.1.0 provides developers with ready-to-use components, simplified, standards based development of mobile web applications, rich Ajax websites, and REST web services. This feature pack can be installed on WAS 7 as well as WAS 8 and contains an implementation of the JAX-RS specification that provides for building RESTful webservices. Feature packs are fully supported by IBM. See whitepaper http://www-01.ibm.com/common/ssi/rep_ca/3/897/ENUS211-143/ENUS211-143.PDF for more details.

For dependency injection we strongly recommend using the Contexts and Dependency Injection (CDI) specification , part of the web profile in Java EE 6. CDI provides type safety and loose coupling between POJOs and other EE resources using dependency injection. CDI (JSR 299) provides context and semantics to the lower level Dependency Injection Specification in Java EE6 defined by JSR 330. CDI also provides a way to extend the Java EE platform via extensions which allow customers to build custom components and modify EE in a specification compliant fashion. For a detailed overview and a presentation on CDI see http://dl.dropbox.com/u/12872475/CDI_Impact_2011_RohitKelapure.pdf

The CDI specification is currently ONLY available in WebSphere Application Server v8. A less desirable option would be to bundle a CDI implementation such as from the Apache Open WebBeans project in your own application on WAS 7. In this scenario when the CDI is bundled inside the application, IBM will not be able to fix CDI issues as it is a part of the customer application.
For an overview of features in WAS 8 see http://www-01.ibm.com/common/ssi/rep_ca/9/897/ENUS211-139/ENUS211-139.PDF

For migration to WAS 8 see the following resources
http://www-01.ibm.com/support/docview.wss?uid=swg27008724
http://www.ibm.com/developerworks/websphere/downloads/migtoolkit/vtov.html

WAS 8 can coexist with WAS 7. The links below will explain mixed cell coexistence, where you can add servers at a later version of WAS for the new features and keep the production environment stable on an older version WAS.
Running multiple application server versions
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.migration.nd.doc/info/ae/ae/tins_coexistep.html
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.migration.nd.doc/info/ae/ae/cmig_coexist.html
Setting up Version 8.0 coexistence
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.migration.nd.doc/info/ae/ae/tmig_70coexist70.html
Setting up Version 6.x, Version 7.0 and Version 8.0 coexistence
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.migration.nd.doc/info/ae/ae/tmig_6coexist70.html

Excessive DirectByteBuffer WebSphere Application Server native allocations

2011-10-25T18:03:00.000-07:00

Things that we recommend to customers who are dealing java and native heap issues due to excessive DirectByteBuffer allocations.

Switching AIO to NIO
http://www-01.ibm.com/support/docview.wss?uid=swg21366862

Setting the WebSphere Application Server WebContainer to synchronous mode
http://www.ibm.com/support/docview.wss?uid=swg21317658

Introspection of system dumps generated on an OOM with IBM Product Extensions into Memory Analyzer Tool: Run the following query
Open Query Browser → IBM Extensions → Java SE Runtime → DirectByteBuffers

Fixed size Major thread pools (WebContainer, etc.) (min=max)
http://www-01.ibm.com/support/docview.wss?uid=swg21368248

IBM Health Center has native memory monitoring
Generic JVM argument -Xhealthcenter
-Xhealthcenter:level=low to not gather profiling data
Support Assistant -Xhealthcenter:level=headless for writing to an HCD file

Javacore memory analysis script
get_memory_use.pl
http://www.ibm.com/developerworks/aix/library/j-nativememory-aix/index.html

Enable explicit GC
If ExplicitGC has been disabled please enable it

Dynacache JSPs that wil force the creation of a cache instance

2011-10-10T15:29:00.000-07:00

The JSPs below force the creation of a cache instance and allow for invalidation of the cache instance. Drop these JSPs in the /installedApps/Dynamic Cache Monitor.ear/CacheMonitor.war/WEB-INF directory.

https://github.com/kelapure/dynacache/blob/master/scripts/Invaliddatereqs.jsp
https://github.com/kelapure/dynacache/blob/master/scripts/invalidateIT.jsp

The primary purpose of these JSPs is to force the creation of the cache instance and make the cache instance show up in the list of cache instances in the CacheMonitor or the Extended Cache Monitor. The JSPs also allow for a specific cache ID to be invalidated.

Migrating to WebSphere Application Server 8

2011-10-10T09:36:00.000-07:00

Excellent Session by WebSphere Application Server 8 Release Architect Dana Price on WebSphere application Server 8 Migration.

Another excellent resource on Migration is the Migration Toolkit. The version-to-version Migration Toolkit information is here. Documentation and the tool itself can be downloaded from that page.

See Migration resources for WAS http://www-01.ibm.com/support/docview.wss?uid=swg27008724

New lightweight and faster version of WebSphere Application Server now available

2011-10-09T08:21:00.000-07:00

Today, just slightly over 100 days since we eGA'ed WAS V8, we officially introduce the WAS 8.5 Open Alpha to the world. This Alpha introduces for the first time a brand new lightweight tooling and runtime environment that is easier to setup than your phone, starts faster than your TV, and requires less disk space than your digital camera.

Based on OSGi standards, it is also completely modularized so that it can extended with only the functions and programming models that you require. This new environment gives users the freedom to 'customize' their infrastructure for their specific industry demands.

WAS 8.5 is lighter, faster and has more features than Tomcat.Check it out, it *will* surprise you what we've done :o) http://www.ibm.com/wasdev

IBM WebSphere Application Server JVM free and low overhead profiling tool

2011-10-07T12:31:00.000-07:00

The IBM Monitoring and Diagnostic Tools for Java - Health Center is a lightweight tool that monitors active IBM Virtual Machines for Java with minimal performance overhead. The Health Center suggests live tuning recommendations for garbage collection, profiles methods including call stacks, and highlights contended locks.

Please see demo's of IBM Health Center recorded here:

1. Overview of Health Center features http://www.youtube.com/watch?v=5Tcktcl0qxs&feature=related

2. Install IBM Support Assistant and Health Center http://www.youtube.com/watch?v=6WjE9U0jvEk

3. How to enable a Java application for live monitoring by the Health Center http://www.youtube.com/watch?v=Hdp0mJ13NLQ&feature=related

4. What's new in version 1.2 of the Health Center http://www.youtube.com/watch?v=Hdp0mJ13NLQ&feature=related

5. What's new in version 1.3 of the Health Centerhttp://www.youtube.com/watch?v=rZrVVCdv7l0

For the IBM Health Center 1.3 Infocenter see http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

For starting a Java application with the Health Center agent enabled see http://publib.boulder.ibm.com/infocenter/hctool/v1r0/topic/com.ibm.java.diagnostics.healthcenter.doc/com.ibm.java.diagnostics.healthcenter.gui/docs/connecting.html

For downloading and installing the server side agent i.e. healthcenter.jar use -
http://publib.boulder.ibm.com/infocenter/hctool/v1r0/topic/com.ibm.java.diagnostics.healthcenter.doc/com.ibm.java.diagnostics.healthcenter.gui/docs/installingagent.html

JavaOne 2011 - Java EE 6 vs Spring 3.1 Framework Shootout

2011-10-05T11:59:00.000-07:00

This slide deck will compare Spring and Java EE stack in terms of web frameworks. It will re-examine the motivations behind the Spring framework and explore the emergence of the Java EE programming model to meet the challenges posed. This session will provide insight into when Spring and/or Java EE is appropriate for a building web applications and if they can coexist.

Presentation can be downloaded from http://db.tt/1q9us2JH

Java EE vs Spring Framework

View more presentations from kelapure

JavaOne 2011 - Server Resiliency Production Debugging session

2011-10-05T11:50:00.001-07:00

Troubleshooting issues such as OutOfMemoryErrors, Performance problems and various exceptions are a common task for anyone developing or deploying an application. This session will explain techniques to make the server more resilient and using open source IBM tools like the Monitoring and Diagnostic Tools for Java, Extended Memory Analyzer Tool and the Support Assistant to learn how to diagnose these common problems types.

Debugging java deployments_2

View more presentations from kelapure.

WebSphere Application Server 8.0.0.1 is now OUT

2011-09-27T11:39:00.000-07:00

In case you missed it WebSphere Application Server V8.0 Fix Pack 1 NOW AVAILABLE!
http://t.co/3w9K7x0A

This is an absolute MUST-HAVE if you are deploying your application to WAS 8.
List of fixes is found http://www-01.ibm.com/support/docview.wss?uid=swg27022958

There are a lot of fixes and enhancements in multiple component areas.
This is absolutely critical if you are having any issues with your WAS 8.0.0.0 install

Some of the fixes to skip directory and zip archive scanning will speedup app deploy and reduce footprint.
There are a lot of enhancements and ease of use fixes like these in 8.0.0.1.

Go get it today!

WebSphere Application Server SIBus tuning for production deployments

2011-09-27T11:30:00.000-07:00

SIBus Tuning for production WebSphere Application Server

View more presentations from kelapure

Increasing memory for wsadmin task

2011-09-27T11:08:00.000-07:00

Customers use the wsadmin task (ant XML or programatically) to execute the WebSphere command-line administration tool with the specified arguments.

To use this task, add the following to your Ant build.xml
<taskdef name="wsadmin" classname="com.ibm.websphere.ant.tasks.WsAdmin"/>

WsAdmin ant task supported arbitrary -X arguments to the task's forked JVM by directly adding them to the build.xml :

<wsadmin script="${basedir}/script.py"> ...

   <arg value="blah" />

   <jvmarg value="-XX:MaxPermSize=128m"/>

</wsadmin>

You can also do this programatically directly by using WsAdmin task. You will need to setup classpath similar to how it is set for the wsadmin ant task. Modify WAS_HOME/bin/ws_ant.bat/sh script to setup the classpath for your custom task.

WsAdmin task = new WsAdmin();
if (forceConnType != null) {
    System.out.println("forcing connection type to: "+forceConnType);
    task.setConntype(this.forceConnType);
} else {
    task.setConntype(this.conntype);
}             
task.setLang(this.lang);
task.setUser(this.getWasuser());
task.setPassword(this.getWaspassword());
task.setScript(jFile.getAbsolutePath());
if (com.ibm.ws.ant.utils.OsUtils.isSolaris()
           || com.ibm.ws.ant.utils.OsUtils.isHPUX()) {
     //increase permgen space on solaris/hp-ux
     task.createJvmarg().setValue("-XX:MaxPermSize=512M");
} 
task.setJvmMaxMemory("1024M");
task.execute();

FFDC browser tool by David Cohen

2011-09-16T08:28:00.001-07:00

http://www.ibm.com/developerworks/websphere/downloads/ffdc_browser.html

Provides greater accessibility to FFDC logs in an organized and easy to use fashion.

Simplifies the process of locating, accessing and displaying FFDC log contents.
Works in both distributed and zOS environments.
Works in Base, ND, Clustered and Secure topologies.
Direct access from any web browser, provides remote access to the WAS log files-

No need to incur the overhead of more heavyweight tools to view logs, e.g. admin console

Realtime tool: dynamically accesses content from live WAS system
Very small application footprint: solely servlet based

Can be run during stress with minimal or no impact

Developed out of a need to get to FFDC and system log files without the use of the admin console, FTP clients or complicated file system navigation.

Dynacache Utility script to clear cache instances

2011-09-16T08:05:00.000-07:00

I have uploaded an utility Dynacache management script to github that runs on the Deployment Manager, and for each application server on each node, calls clearCache for a set of cache instances. This script can probably run at midnight when you want to flush your entire cache.

Kevin Grigorenko from the WAS SWAT team wrote this script. Thanks Kevin!

Good documentation on Installing/Configuring the sample applications for WAS8.0

2011-09-14T08:59:00.000-07:00

Accessing the samples -->
http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.samples.doc/info/ae/ae/welcome_samples.html

The samples currently available in the Samples section of the WAS V8 Information Center demonstrate the following technologies: Service Component Architecture (SCA), Communications Enabled Applications (CEA), OSGi applications, XML, and Internationalization service

For instance servlet 3.0 sample can be found here

http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.samples.doc/info/ae/ae/sample_servlet30.html

Samples available

Here is a nice article by Philippa Rhodes that gives an overview of the v8 samples:

http://www.ibm.com/developerworks/websphere/library/techarticles/1107_rhodes/1107_rhodes.html?ca=drs-

Here is the v8.0 link to samples (you can access this by the above link as well):

http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp?topic=/com.ibm.websphere.samples.doc/info/ae/ae/welcome_samples.html

Sample code for exercising Dynacache DistributedMap

2011-09-14T08:50:00.000-07:00

A lot of times customers ask me for sample code to drive a Dynacache DistributedMap.

I have put up one such sample on github that illustrates the common and basic operations with Dynacache DistributedMap i.e. the WebSphere Application Server's default POJO cache.
https://github.com/kelapure/dynacache/blob/master/com/ibm/ws/cache/sample/TestDistributedMap.java

Configuring a Dynacache distributed map is covered under Using the DistributedMap and DistributedObjectCache interfaces for the dynamic cache

Best practices and tuning of Dynacache and DRS for replication have been covered in a previous blog post here http://wasdynacache.blogspot.com/2011/07/tuning-dynacache-and-drs.html

IBM WebSphere security questions

2011-09-13T12:51:00.000-07:00

Today's blog post is a gem from WAS Senior Software Engineer Fred Rowe. We often get questions related to malicious data source access. Some these include how users with wadmin can access be prevented from gathering or pulling datasource credentials for database access. How to segment the access from intentional and un-intential access to customer data.

There are two primary ways to secure WebSphere datasources, each method has a user name and password associated with it. In both cases, if displayed from either the admin console or wsadmin, the password value is either:

obscured by dots
obscured by asterisks
displayed in encoded form

1. Using datasource custom properties:

This method is vendor-specific based on the JDBC provider associated with the datasource. Typically, JDBC drivers support the custom properties userand password. If a custom property named password is created, it's value will be obscured in the WAS admin console Data sources > datasource_name > Custom properties list page as follows:

Note that the property named pwd is not obscured. Similarly, on the WAS admin console Data sources > datasource_name > Custom properties > password details page, the XOR-encoded version of the password is displayed:

And finally, if the password property is examined from wsadmin, the XOR-encoded version of the password is displayed:

wsadmin>pwd = AdminConfig.getid("/DataSource:datasource_name/J2EEResourcePropertySet:/J2EEResourceProperty:password/")

wsadmin>print AdminConfig.showAttribute(pwd, "value")

{xor}MiYvPiwsKDAtOw==

2. Using JAAS - J2C authentication aliases:

If a JAAS auth alias is created for the datasource, the password of the alias is obscured on the WAS admin console Data sources > datasource_name > JAAS - J2C authentication data > myAlias details page:

Similarly, if the value of the password attribute of an auth alias object is examined from wsadmin:

wsadmin>print AdminConfig.getid("/JAASAuthData:/")

wsadmin>print AdminConfig.showAttribute(authAlias, "password")

*****

Resource Workload Routing in WebSphere Application Server 8

2011-09-06T16:11:00.000-07:00

WebSphere Application Server V8 adds improved high availability (HA) support in several scenarios, including:

Resource failover and retry logic for relational data sources and JCA connection factories, where the administrator can tailor the data sources and connection factory configuration based on application need to specify:
- Number of connection retries.
- Alternate/fail-over resource.
- Pre-population of alternate/fail-over resource connection pool.

Resource Workload Routing is a new feature exclusive to the WebSphere Application Server that enables applications to easily recover from resourceoutages, such as database failures, without requiring you to embedalternate resource and configuration information.

Resource workload routing includes data source and connection factory fail over andsubsequent fail back from a predefined alternate or backup resource. You can tailor theresource fail over and failback flexible configuration options tomeet your environment-specific and application needs.new

On distributed systems, you can enable resource routing by configuring an alternate resource.
A data source and connection factory can fail over and fail back automatically when a specified or default failure threshold value is reached. When fail over occurs, the application switches from using the primary resource to using the alternate resource. Fail back occurs when the application switches back from the alternate resource to the primary resource.

Implementers of Resource Adapters take note - Fail over works with non-relationalresource adapters that have a ManagedConnection object that implementsa testConnection method.

Instead of rolling your own architecture and policy for outbound resource fail over please leverage this in-built resource routing High Availability function to prevent disruption to end users.

WebSphere Application Server- Windows native memory leak

2011-09-02T14:58:00.000-07:00

Please look at this excellent WSTE session by Kevin Grigorenko that covers native memory issues for WebSphere Application Server, particularly native OutOfMemoryErrors and leaks on Windows.

It covers theory, detecting and monitoring, isolating and avoiding, and finally analyzing leaks with UMDH, LeakDiag, DebugDiag, VMMap and windbg.

Windows Native Memory Problem Determination Techniques and Tools for WebSphere Application Server - http://www-01.ibm.com/support/docview.wss?uid=swg27021906

To replay the recording of this 46 minutes Webcast, click on Download Audio (5.0MB - audio in MP3 format) then open the PDF file

WebSphere Application Server - How to display SQL statements executed from J2EE applications using minimal tracing

2011-09-02T14:51:00.000-07:00

Is there a minimal trace string to view SQL statements being executed by an application running on WebSphere Application Server?

Answer

If the application uses java.sql.PreparedStatement objects, then there is a relatively lightweight way to obtain the SQL statements being executed. PreparedStatement objects are often used in applications because of their performance over regular java.sql.Statement objects.

To view the SQL statements being executed using a PreparedStatement object, enable WebSphere tracing for the class that implements java.sql.PreparedStatement:com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement. The corresponding trace specification is: com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement=all

Examples:
Consider the following sample SQL INSERT statement being executed from the application using a java.sql.PreparedStatement:

String insertCoffeeSaleString
= "INSERT INTO SALES(ITEM_NAME,QUANTITY,PRICE) values('Coffee',5,1.25)";
PreparedStatement insertCoffeeSale
= connection.prepareStatement(insertCoffeeSaleString);
insertCoffeeSale.executeUpdate();

With com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement=all trace enabled, the following output can be seen when the Prepared Statement is initialized:

Trace: 2011/04/07 14:37:12.100 01 t=6C9020 c=UNK key=P8 (13007002)
ThreadId: 0000002b
FunctionName:
SourceId: com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement
Category: FINER
ExtendedMessage: Entry; org.apache.derby.impl.jdbc.EmbedPreparedStatement30@401c401c, com.ibm.ws.rsadapter.jdbc.WSJdbcConnection@3
c4a3c4a, HOLD CURSORS OVER COMMIT (1), PSTMT: INSERT INTO SALES (ITEM_NAME,QUANTITY,PRICE) values('Coffee',5,1.25) 1003 1007 1 0 0

wsadmin script to configure a data source for WebSphere Application Server

2011-08-31T12:07:00.000-07:00

Do you know how to configure JDBC, DataSources, etc using wsadminlib.py library ?

See: https://cwiki.apache.org/ARCHIVA/archiva-on-websphere-61.html

Specifically, the file, createArchivaServer.py, from: https://cwiki.apache.org/ARCHIVA/archiva-on-websphere-61.data/createArchivaServer.py

And this is the relevant sections from the file:
    # Specify the DB2 details.
    db2DriverPath = "/usr/opt/db2_08_01/java"
    db2Alias = "DB2ArchivaUserId"
    db2UserId = "db2inst1"
    db2Password = "db2inst1"
    # JNDI details
    db2DatabaseName = "ARCHIVA"
    db2DriverType = "4"
    db2ServerName = "localhost"
    db2PortNumber = "50000"

and

# Set the WebSpere Variable needed to for the universal driver. In this case, we just set it for
# the individual server. To set it for the entire node, just remove the serverName variable completely.
print "Setting WebSphere Variable"
setWebSphereVariable( "DB2UNIVERSAL_JDBC_DRIVER_PATH", db2DriverPath, nodeName, serverName)

# Create a JAAS entry for the userid/password for the DB2 datasource.
# If the entry already exists, the function simply returns.
createJAAS(db2Alias, db2UserId, db2Password)

# Create the DB2 Universal Driver
print "Creating the DB2 Universal Driver Provider"
serverId = getServerId(nodeName,serverName)
jdbcProvider = createJdbcProvider ( serverId, "DB2 Universal JDBC Driver Provider", "${DB2UNIVERSAL_JDBC_DRIVER_PATH}/db2jcc.jar;${UNIVERSAL_JDBC_DRIVER_PATH}/db2cc_license_cu.jar;${DB2UNIVERSAL_JDBC_DRIVER_PATH}/db2jcc_license_cisuz.jar", "", "com.ibm.db2.jcc.DB2ConnectionPoolDataSource", "Non-XA DB2 Universal JDBC Driver-compliant Provider. Datasources created under this provider support only 1-phase commit processing except in the case where driver type 2 is used under WAS z/OS. On WAS z/OS, driver type 2 uses RRS and supports 2-phase commit processing." )
print "Created: jdbcProvider = " + jdbcProvider

# Create the JNDI Data Sources
# First, jdbc/users
print "Creating the Users Database Data Source"
usersDataSource = createDataSource( jdbcProvider, "Users Database", "jdbc/users", "jdbc/users", 10, db2Alias, "com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper" )
print "Created: usersDataSource = " + usersDataSource
setJ2eeResourceProperty( usersDataSource, "databaseName", "java.lang.String", db2DatabaseName, "The name of the database." )
setJ2eeResourceProperty( usersDataSource, "driverType", "java.lang.Integer", db2DriverType, "The JDBC connectivity-type of a data source." )
setJ2eeResourceProperty( usersDataSource, "serverName", "java.lang.String", db2ServerName, "The TCP/IP address or host name for the DRDA server." )
setJ2eeResourceProperty( usersDataSource, "portNumber", "java.lang.Integer", db2PortNumber, "The TCP/IP port number where the DRDA server resides." )

# Then, jdbc/archiva
print "Creating the Archiva Database Data Source"
archivaDataSource = createDataSource( jdbcProvider, "Archiva Database", "jdbc/archiva", "jdbc/archiva", 10, db2Alias, "com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper" )
print "Created: archivaDataSource = " + archivaDataSource
setJ2eeResourceProperty( archivaDataSource, "databaseName", "java.lang.String", db2DatabaseName, "The name of the database." )
setJ2eeResourceProperty( archivaDataSource, "driverType", "java.lang.Integer", db2DriverType, "The JDBC connectivity-type of a data source." )
setJ2eeResourceProperty( archivaDataSource, "serverName", "java.lang.String", db2ServerName, "The TCP/IP address or host name for the DRDA server." )
setJ2eeResourceProperty( archivaDataSource, "portNumber", "java.lang.Integer", db2PortNumber, "The TCP/IP port number where the DRDA server resides." )

Courtesy IBMer Chris Graham from down under.

Non standard performance tuning for the WebSphere Application Server

2011-08-30T13:15:00.000-07:00

Links for some non standard and standard performance tuning of the WebSphere Application Server

http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html
http://wasdynacache.blogspot.com/2011/08/tune-your-websphere-application-server.html
http://wasdynacache.blogspot.com/2011/05/websphere-application-server-top-10.html
http://wasdynacache.blogspot.com/2010/03/pimp-my-websphere-reduce-websphere.html
http://wasdynacache.blogspot.com/2011/04/websphere-application-server-walk-in.html
http://wasdynacache.blogspot.com/2011/05/websphere-application-server.html
http://webspherecommunity.blogspot.com/2011/05/tuning-for-idleness-service.html

For a paid offering ask your IBM services representative about the Performance Tuning Configurator (PTC) tool for improving and tuning WAS performance

IBM Extensions to the Eclipse memory analyzer

2011-08-30T09:28:00.000-07:00

The IBM extended version of Eclipse Memory Analyzer contains a number of plugins that help with determining the memory usage of WAS components like Dynacache, HTTPSession, WebContainer etc .. The IBM Extensions for Memory Analyzer currently offers extensions covering: Java SE Runtime, & WebSphere Application Server

Snapshot of what this look like in the tool ...

IBM WebSphere Application Server Extensions to Eclipse Memory Analyzer

IBM Java SE Runtime Extensions to Eclipse Memory Analyzer

If you have a complete core dump this tool can also show you thread details, thread stacks, a classloader explorer etc ...

Java Thread exploration with Eclipse Memory Analyzer

My version of the IBM extended Eclipse Memory Analyzer

Happy Debugging!

WebSphere Session Persistence Best Practices ... The 10/10 Rule

2011-08-18T10:45:00.000-07:00

10 Seconds write frequency /10 KB session size

The most important factor in session replication performance is the write frequency setting.
This setting should be carefully considered to determine if the environment needs full reliability and failover support (End of Servlet service), or if a time based frequency is sufficient. In testing done for this study, a value larger than 10 seconds rarely provided much additional benefit. Therefore, as a best practice, if the time based frequency is chosen, we would recommend a value of 10 seconds.

The cost of replication or persistence increases as the size of the session object increases.
Therefore, as a best practice, we would recommend keeping session sizes smaller than 10 KB.

Database session persistence performs better than Memory- to- Memory session persistence. This is because in the Database persistence mode, the application servers are not having to act as both an application server servicing requests and a replication backup, as was the case in the Memory-to-Memory scenarios. For larger environments, Database persistence is almost always the better choice as it scales a lot better than Memory-to-Memory.

There is an extra cost associated with a database server, license, and database administrator. Most environments require database backups / mirrors. The performance of the database can also be a bottleneck as the network latency between database and app server, and the speed of the database machine and disk drives are all factors. From a cost perspective Memory-to-Memory session persistence will always beat database persistence.

See this Whitepaper WebSphere Application Server V8 Replication Performance for a complete discussion of this topic and DayTrader benchmark results.

Diagnostic tools for WebSphere Application Server

2011-08-17T14:02:00.000-07:00

There are a number of publicly available tools and scripts that WebSphere Application Server component and service teams use for assisting and problem determination of PMRs.

WebSphere Plugin tools - Perl Script for parsing WebSphere Plugin logs
https://github.com/covener/plugin-tools

HTTPSession - Session tracker debug Servlet
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=pix&product=was-express-dist&topic=rtrb_httpsesncomp
Servlet name: com.ibm.ws.webcontainer.httpsession.IBMTrackerDebug shipped with WAS.

Dynacache: Extended Cache Monitor and Flight Recorder
http://wasdynacache.blogspot.com/2010/03/dynacache-flight-recorder-wsadmin.html
http://www.ibm.com/developerworks/websphere/downloads/cache_monitor.html

HAManager: Awk scripts for analyzing HAManager logs
https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en_US#/person/0600007J6A/file/2499709b-2511-44ac-a38a-fa62b3e3e7aa

SIBus –Explorer & Destination Handler

SIB Explorer: http://alphaworks.ibm.com/tech/sibexplorer
SIB Performance Tool: http://www.alphaworks.ibm.com/tech/sibperf
A tool to view, move, copy, delete, save and restore messages in the default messaging provider: http://www-01.ibm.com/support/docview.wss?uid=swg24021439

LDAP Tools

Apache Directory Studio Very handy LDAP tool based on Eclipse http://directory.apache.org/studio/index.html

ldapSearch - LDAP search tool http://linux.die.net/man/1/ldapsearch

Sun JVM ONLY Tools- jps, jstatd, jstack, jmap, jinfo, jhat visualVM & jConsole

Name space dumper $WAS_HOME/bin/dumpNameSpace tool

OSGI Bundle Explorer and debug utility $WAS_HOME/bin/osgiConsole.sh/bat

ISA Tools

Generating system dumps and thread dumps automatically

2011-08-11T12:06:00.000-07:00

Generating a system dump instead of a heap dump automatically on an OOM
To generate a system dump as default on an OOM error or when the JVM receives the SIGQUIT.
Please set the following JVM generic custom property
-Xdump:java+system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,priority=999,request=exclusive+compact+prepwalk
Please also ensure that there is adequate space for the system dump by following this technote .

Generating a thread dump automatically on a thread hang
To configure a hang detection policy that dumps a javacore when a thread is hung by setting the following property

From the administrative console, click Servers > Application Servers > server_name OR System Administration > Deployment Manager or System Administration > NodeAgent > node_agent_name
Under Server Infrastructure, click Administration > Custom Properties
Click New.
Add the following property:

Name com.ibm.websphere.threadmonitor.dump.java
Value true

This causes a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.

Table of where to set a JVM Custom property

Application server	Servers > Server Types > WebSphere application servers > server_name > Java and Process Management > Process Definition > Java Virtual Machine > Custom Properties
Deployment manager	System Administration > Deployment Manager > Java and Process Management > Process definition > > Java Virtual Machine > Custom Properties
Node agent	System Administration >Node Agent > nodeagent > Java and Process Management > Process definition > Java Virtual Machine > Custom Properties

References:
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Ftools%2Fdiag_collector.html

http:http//publib.boulder.ibm.com/infocenter/javasdk v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Ftools%2Fdiag_collector.html

JAX-RS implementations in WebSphere Application Server

2011-08-09T11:17:00.000-07:00

Jersey is the open source JAX-RS (JSR 311) Reference Implementation for building RESTful Web services. Jersey provides an API so that developers may extend Jersey to suit their needs. See blog post on how to integrate Jersey with WebSphere Application Server 6.1. If the first suggestion does not work look at this post.

Gotcha To get Jersey working with WAS v7 you will need to set the webcontainer custom property.
Name: com.ibm.ws.webcontainer.invokeFiltersCompatibility
Value: true
This workaround is NOT needed in WAS v8. See APAR PK56247 for details on why you need to do this in WAS v7.

If your application that leverages Jersey throws the following error during deployment

The URI scheme wsjar of the URI wsjar:file:/C:/rad802/workspace/myCompany/TestJerseyProvider/WebContent/WEB-INF/lib/Providers.jar!/com/mycompany/commons/web/rest/providers is not supported. Package scanning deployment is not supported for such URIs.

This occurs because your classes in web-inf/lib are not being recognized, because the Jersey URI Scheme scanner which scans WEB-INF/lib only works, when the protocol has 3 digits (like jar:) but WebSphere uses wsjar: as protocol.

We ran into a similar issue when using Open Web Beans see https://issues.apache.org/jira/browse/OWB-577
You can also look at the svn change history to see how we fixed it in Open Web Beans.

This is fixable in Jersey by adding your own com.sun.jersey.core.spi.scanning.uri.UriSchemeScanner which is the interface for scanning URI-based resources and reporting those resources to a scanning listener. Jersey's code add's additional UriSchemeScanner's using the com.sun.jersey.spi.service.ServiceFinder. Looking at existing scanners like the the com.sun.jersey.core.spi.scanning.uri.JarZipSchemeScanner & com.sun.jersey.core.spi.scanning.uri.FileSchemeScanner you will be able to write a scanner that is able to scan the jars in the WEB-INF/lib directory of your WAS application.

Another option is to do what the warning says Try using a different deployment mechanism such as explicitly declaring root resource and provider classes using an extension of javax.ws.rs.core.Application.

Please note that WAS ships with its own JAX-RS implementation based off the Apache Wink Project. This is available by default in WAS version 8 and with the Web 2.0 feature pack in WAS v7.

Tune your WebSphere Application Server deployment for Production or Development

2011-08-06T07:03:00.000-07:00

Do you know that WebSphere Application Server install contains scripts to tune the performance of the server ? These can be found in $WAS_HOME/scriptLibraries/perfTuning/V70

This wsadmin script can be run as follows ....

$WAS_HOME/bin/wsadmin -f applyPerfTuningTemplate.py [-nodeName <node> -serverName <server>] [-clusterName <cluster>] -templateFile <file> where templateFile could be one of "peak.props", "development.props" or "default.props"

#-----------------------------------------------------------------------------

# File Name:
#      ApplyPerfTuning.py
# Purpose:
#      Apply pre-defined performance tuning templates to a specific
#      application server instance or all instances in a cluster. Three
#      templates are pre-defined (default, peak, and development);
#      however, these files can easily be modified to create your own custom
#      tuning template.
#-----------------------------------------------------------------------------

You can tune WAS either for development (faster application restarts, smaller footprint, dynamic threadpool sizes) or production (increased throughput, smaller garbage collection pauses, Fixed threadpool sizes). Correctly tuning WAS can vastly improve your developer and customer experience.

For a complete case study in basic and advanced tuning see http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html . The article discussed this core set of tunables plus other option resulting in performance gains in excess of 100%.

Configure WebSphere Application Server Dynacache and Session persistence to use WebSphere eXtreme Scale

2011-08-05T13:05:00.000-07:00

Dynacache
Please see instructions below on how to configure Dynacache to leverage WXS instead of the default caching engine. I have also provided links for detailed instructions if you get confused or stuck/
Video eXtreme Scale Wiki Configuring cache integration
Redbook chapter on integrating WXS with Dynacache in the context of WebSphere Commerce

Configure WebSphere Commerce to use WebSphere eXtreme Scale for dynamic cache to improve performance and scale

HTTPSession
You will also be interested in configuring session persistence with WXS.
Infocenter Configuring the HTTP session manager with WebSphere Application Server
Attention: With Version 7.1.0.3 and later, you can persist sessions that use URL rewriting or cookies as a session tracking mechanism. Before Version 7.1.0.3, you can save only sessions that use cookies as the session tracking mechanism to the data grid.

Cheatsheet

1. Install & Set up an WXS 7 grid (Catalog server and other XS containers that host the cache) . The latest WXS version has an admin console for monitoring. Please install the latest WXS 7 fix pack before proceeding.

2. Augment the WAS profile with the WXS client

3. Configure the dmap to use extreme scale

Basically add the following properties to the map where we create the dmap

com.ibm.ws.cache.CacheConfig.enableCacheReplication = true

com.ibm.ws.cache.CacheConfig.cacheProviderName = com.ibm.ws.objectgrid.dynacache.CacheProviderImpl

com.ibm.websphere.xs.dynacache.topology = remote

com.ibm.websphere.xs.dynacache.num_initial_containers = 1

4. [optional] XS does not place replica and primary shards on containers that have the same IP address (unless the configuration is in development mode). If ALL your XS containers are on one physical machine or node you will need to set development=true. In the deployment policy descriptor XML file after WXS client is installed i.e. $WAS-HOME/optionalLibraries/ObjectGrid/dynacache/etc/dynacache-remote-deployment.xml set developmentMode to true

<mapSet name="DYNACACHE_REMOTE"
numberOfPartitions="103"
minSyncReplicas="0"
maxSyncReplicas="0"
maxAsyncReplicas="1"
numInitialContainers="1"
developmentMode="false" --> true
replicaReadEnabled="false">
<map ref="IBM_DC_PARTITIONED_.*" />

WebSphere Application Server Defibrillator for stalled software

2011-08-03T09:46:00.000-07:00

Food for thought ...

I have a lot of interest in Java serviceability and JVM egregious condition recovery. If you share the same interest, please check out the following research paper http://groups.csail.mit.edu/pac/jolt/paper.pdf that talks about Detecting and Escaping In nite Loops with Java or the more easily digestible http://web.mit.edu/newsoffice/2011/infinite-loops-interrupt-0802.html

One thought that did resonate in the article is from Westley Weimer, an assistant professor of computer science at the University of Virginia who says "The vast majority of software engineering or programming-language researchers are shackled to this notion of full correctness or full soundness: You can’t change anything in the program if there’s even the possibility of getting a slightly wrong answer,”

The article goes on to say that “One of the things that’s really characterized Martin’s research over the last 10 or 15 years is casting off the shackles of soundness in favor of approaches that are probably correct but dramatically more useful in real life.”

This is true in order to achieve resiliency you will need to sacrifice some program consistency and correctness.

WebSphere Application Server SWAT BLOG

2011-08-03T05:38:00.000-07:00

For all those interested in WebSphere Application Server serviceability and problem determination please follow the WAS Swat blog run by Kevin Grigorenko WAS troubleshooting extraordinaire https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/?lang=en

How to analyze IBM JVM core dumps

2011-08-02T20:36:00.000-07:00

Today's blog post is advice, I provided to a customer for analyzing IBM JDK core dumps

Question ...

We have a problem where our Linux-based build machines for the portal drivers, occasionally core dump in the middle of the build process. The only way I know of to invoke the Java team to shoot these, is to open a WAS PMR. I'm not sure that's appropriate, given that it's unclear exactly what JRE is running at the time (it could be the one from the WAS or from ant)

Response

In order to analyze a coredump you will need to run the Dump Analyzer plugin from IBM Support Assistant (ISA). The dump report will tell you the details of the JVM that dumped core, the heap contents and the javacore. This will help you determine the culprit component faster.

First see these instructions to run the dump analyzer outside the ISA

cd $ISA_HOME\applications\eclipse\plugins\com.ibm.java.diagnostics.dbda_2.2.2.20090926232659
java -Xmx1400M -cp dumpAnalyzer.jar;lib\dtfj.jar;..\com.ibm.java.diagnostics.dbda.was_2.1.3.20090427213613\wasAnalyzers.jar;..\com.ibm.java.diagnostics.dbda.core_2.2.2.20090926232659\coreAnalyzers.jar com.ibm.dtfj.analyzer.base.DumpAnalyzer core1.sdff com.ibm.dtfj.analyzer.jvm.DefaultDumpReport_Extended > DumpAnalyzer_DefaultDumpReport_Extended.txt

OR

You basically need to copy the 6 required jars from IBM Support Assistant and then run the command
set CP="dumpAnalyzer.jar;coreAnalyzers.jar"
set BCP="lib/j9/dtfj.jar;lib/j9/dtfj-interface.jar;lib/sov/dtfjsu.jar;lib/sov/dtfj.jar"
java -cp $CP -Xbootclasspath/p:$BCP
com.ibm.dtfj.analyzer.base.DumpAnalyzer (dumpName) (options)

You need to put this in a script and run it whenever a core dump is generated.

Happy Debugging!

Solving an application performance slowdown or regression with WebSphere Application Server

2011-08-01T14:37:00.000-07:00

Recently I had the chance to troubleshoot and fix a performance regression a component of the WebSphere Application Server. The objective was to determine performance and locking hotspots in our server code.

1. Beg, borrow steal or write a benchmark. We already have a pretty good Dynacache performance benchmark. This is critical to draw a line in the sand and determine a before and after performance #s.

2. Enable IBM Health Center profiling. Using IBM Health Center which is baked into the WAS JVM, is the easiest and lightest way of doing profiling with WebSphere Application Server. No expensive profiling tools and no instrumentation of src. This is best thing since baked bread for performance in WAS.

3. So you need two things for this exercise. On the client side you need an IBM Health Center Client that comes as a free product plugin to the IBM Support Assistant. On the server side you need IBM Health Center server library which comes preinstalled with the JVM which is bundled with WAS. You will find one WAS_HOME/java/jre/lib/ext/healthcenter.jar

4. Configure WAS to use the IBM Health Center. See details in Infocenter for older versions of java.
Servers ->Server Types -> WebSphere application servers. Select the server name and then select Java and Process Management -> Process definition -> Java Virtual Machine -> Generic JVM Arguments.
For Java 5 SR10 and later, or Java 6 SR5 and later, use:-Xhealthcenter

5. After restaring WAS, start the ISA and configure a connection to your WAS server. This connection is established typically on port 1872. If you want to control the server port see infocenter. After the JVM is started with the agent enabled, you see a message detailing the port for the Health Center agent. For example:
05-Mar-2009 09:49:57 com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgentINFO: Health Center agent started on port 1972.

6. Run the benchmark or the test.

7. After the test save the profile as a *.hcd file. You can then pass this profile around and analyze it offline. See technote.

8. Interpret the method profile. The Method Profile table shows which methods are using the most processing resource.

Methods with a higher Self (%) value are described as "hot", and are good candidates for optimization. Small improvements to the efficiency of these methods might have a large effect on performance. Methods near the bottom of the table are poor candidates for optimization. Even large improvements to their efficiency are unlikely to affect performance, because they do not use as much processing resource.
You can optimize methods by reducing the amount of work that they do or by reducing the number of times that they are called. Highlighting a method in the table populates the call hierarchy views.Filter the contents of the method profile table using the text box above the table. See the filtering help topic for more information.
Additionally, when you select the Hide low sample entries, the table does not list any entries that have a sample count of less than 2. Use this option if your table contains many entries that are not obvious candidates for optimization to improve the performance of the table.

9. Make java source code changes to improve performance. Scrub and repeat. We used the process above to take before and after profiles to figure out the delta i.e .where the performance had slipped.

Happy profiling!

Java Hung Thread Detection & Handling

2011-07-25T08:05:00.000-07:00

Have you ever wondered how hung thread detection works ?
See this article for an excellent approach for hung thread detection in a Java application -
http://www.javaworld.com/community/print/8051

WebSphere Application Server is the ONLY application server that offers hung thread detection (since version 6.1.0.19). The hang detection option for WebSphere Application Serveris turned on by default. You can configure a hang detection policyto accommodate your applications and environment so that potentialhangs can be reported, providing earlier detection of failing servers.When a hung thread is detected, WebSphere Application Server notifiesyou so that you can troubleshoot the problem. For details see Infocenter.

WSVR0605W is the message when there is a hung thread.

In WAS 7, WSVR0605W is accompanied by a thread dump of hung thread:
[11/16/09 12:41:03:296 PST] 00000020 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (00000021) has been active for 655546 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
  at java.lang.Thread.sleep(Native Method)
  at java.lang.Thread.sleep(Thread.java:851)
  at com.ibm.Sleep.doSleep(Sleep.java:55)
  at com.ibm.Sleep.service(Sleep.java:35)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
   ...

WSVR0606W is the message when a thread that was seen as hung is no longer hung.

Hang detection only monitors managed threads, such as the WebContainer thread pool.
Any native threads, or threads spawned by an application are not monitored.

Happy Debugging!

Java Serialization performance improvement

2011-07-22T14:40:00.000-07:00

If your application or application server spends a lot of time during serialization and deserialization of java objects say due to disk caching or replication then improving the performance of java serialization pays a lot of dividend in terms of reduced CPU% and improved performance. It is worth spending the time optimizing serialization performance.

The best performance in serializing or deserializing data comes by owning the process by implementing Externalizable. Take control over writing and reading your own state. That will give the best performance.

See these articles that expound on the topic:
http://www.javaperformancetuning.com/tips/rawtips.shtml
http://www.onjava.com/pub/a/onjava/excerpt/JavaRMI_10/index.html

Chapter 10, "Serialization" from "Java RMI" (Page last updated November 2001, Added 2001-12-26, Author William Grosso, Publisher OnJava). Tips:

Use transient to avoid sending data that doesn't need to be serialized.
Serialization is a generic marshalling mechanism, and generic mechanisms tend to be slow.
Serialization uses reflection extensively, and this also makes it slow.
Serialization tends to generate many bytes even for small amounts of data.
The Externalizable interface is provided to solve Serialization's performance problems.
Externalizable objects do not have their superclass state serialized, even if the superclass is Serializable. This can be used to reduce the data written out during serialization.
Use Serializable by default, then make classes Externalizable on a case-by-case basis to improve performance.

Record Data Replication statistics in WebSphere Application Server

2011-07-22T13:36:00.000-07:00

If any point you need to record the DRS activity in your WebSphere Applications Server cluster you can set the JVM system property ENABLE_DRS_STATS_LOGGING to 2 to record Data Replication Service (DRS) traffic statistics
See technote http://www-01.ibm.com/support/docview.wss?uid=swg1PK23862&wv=1

Please note, as the technote says Stats logging is enhanced by providing summary and detailed statistics on outgoing message transmission. Thesestatistics are used for servicability and are not generallycollected unless instructed by IBM service.

Migrating WebSphere Application Server

2011-07-22T13:31:00.000-07:00

If you are considering migrating to WebSphere Application Server or upgrading your current version of WAS please take a look at the IBM® WebSphere® Application Server Migration Toolkit v3.0

Quickly and cost-effectively migrate to WebSphere Application Server V8, whether from a previous version of WebSphere Application Server (5.1, 6.0, 6.1, 7.0) or competitive application servers including Oracle® Weblogic Server, Oracle Application Server, and JBoss Server.
Simple remediation tools utilize Rational® source code scanning technology to identify and resolve known application migration issues. All while providing developers complete control over the process.
Simplified developer migration experience using Application Migration Tools
Support for migrating JEE applications to WebSphere Application Server Version 8.0
Improved rule sets which scan application source code for known migration problems. New depreciated and removed APIs, Behavior changes and enforcements for JRE and Java EE specifications
Enhanced platform support – Supports the latest versions of Rational Application Developer and Eclipse.

Detailed information PDF Doc
This toolkit will reduce the traditional cost associated with migration by 40%.

Flushing classloader leaks in your application in WebSphere Application Server

2011-07-21T17:51:00.000-07:00

WebSphere Application Server has a hidden serviceability gem in JVM custom property com.ibm.ws.runtime.detectAppCLLeaks described in this technote

So what does setting this custom property do ?
This custom property enables the classloader leak detection feature in the application server.When the server is shutting down WAS determines the classloaders that have leaked and thereby held references to all their loaded classes and objects. If a classloader leak is detected a heapdump is triggered. You will also see a warning message in the SystemOut.log like so

[7/21/11 20:50:24:917 EDT] 00000017 ApplicationMg W   Application class loader leak: 
com.ibm.ws.classloader.CompoundClassLoader@21b79f0[war:leak_war/leak.war]
   Local ClassPath: 
   Parent: com.ibm.ws.classloader.CompoundClassLoader@218fc48[app:leak_war]
   Delegation Mode: PARENT_FIRST

This provides an excellent way to flush out all the classloader leaks in your application. I advise setting this property in development environments. Please note that not all suspect leaks are of concern particularly the ones that belong to WebSphere Application Server system applications.

Happy Debugging!

Classloader leak reference

Windows style Task Manager for WebSphere Application Server

2011-07-17T18:02:00.000-07:00

Ever wondered what your application server was up to ? If you want to know what all the threads of the application server are doing or what an individual thread is upto take a look at the two JSPs provided below which simulate a windows task manager for all the threads of the Application Server

These JSPs leverge the ThreadMXBean API in the JDK to display thread information. Caution: This may not work on zOS. These JSPs allow a user to see all threads in a table view, and then click on one of the threads to get a detailed view.
https://github.com/kelapure/dynacache/blob/master/scripts/AllThreads.jsp
https://github.com/kelapure/dynacache/blob/master/scripts/ViewThread.jsp

All Thread Views

Thread Detail View

JVM -Xtrace the holy grail of WebSphere Application Server tracing

2011-07-12T09:46:00.000-07:00

Today's objective is to generate -Xtrace for ALL methods on the WebSphere PMI interfaces that our PMI modules implement. These PMI interfaces are called by the WAS container to compute PMI statistics. To do this we use IBM JVM's method trace facility.

Method trace is a powerful tool for tracing methods inany Java code. Method trace provides a comprehensive and detailed diagnosis ofcode paths inside your application, and also inside the system classes.You do not have to add any hooks or calls to existing code. You canfocus on interesting code by using wild cards and filtering to controlmethod trace. Method trace can trace method entry, exit and parameters.

Use method trace to debug and trace application code and the systemclasses provided with the JVM.

While method trace is powerful, it also has a cost. Applicationthroughput is affected by method trace. The impact is proportion tothe number of methods traced. Additionally, trace output is reasonablylarge and might require a large amount of drive space. For instance,a full method trace of a "Hello World" application is over 10MB
You can control method trace by using the command-line option -Xtrace:. This -Xtrace is set as a JVM generic argument in the WebSphere Application Server admin console.

Please see below for the JVM generic argument for capturing all PMI Event points as JVM trace.
-Xtrace:methods={com/ibm/ws/pmi/server/modules/*.*()},print=mt

This will put the output in native_stderr.log.
You can then write post processors to process the trace or alternately you may be able to suck the output in IBM Health Center to visualize it. I have not tried this yet.

More details on -Xtrace here
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/topic/com.ibm.java.doc.diagnostics.50/diag/tools/trace_options_method.html
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.50%2Fdiag%2Ftools%2Ftrace_options_output.html

Unfortunately this will generate the JVM trace for ALL methods of the PMI modules not just the ones on the PMI interface. This limitation we will need to live with for now.Thanks to WAS SWAT Team guru Kevin Grigorenko for sounding me out as we discussed options.

Happy Debugging!

Uneven WebSphere Application Server Plugin load balancing and routing

2011-07-09T07:56:00.000-07:00

If your WebSphere Application Server deployment is experiencing an imbalance in HTTP traffic routing from the WebSphere Plugin please take the following configuration and recommendations into consideration

RANDOM load balancing is better than ROUND_ROBIN for load balancing. With RANDOM for all non-affinity requests, the server is “picked” randomly.The affinity requests are sent to the matching clone id. LoadBalance “weight” is not used at all in the Random case.
An affinity request is one where session affinity is established by the first request. All subsequent requests with the same sesssion ID are termed as "affinity requests". Plugin ALWAYS obeys session affinity when routing requests and routes to the appserver targeted on the first request that established affnity.
In Round Robin load balancing, the first request is routed randomly. After that, the plug-in will use the next server in the list and so on… (member1, member2, member3, member4). Each time a server is “picked”, the load balance weight is decremented by one. Any server with a weight below 0 will NO longer be used for new requests. But, affinity requests will still go through.When all servers have a weight of less than 0, the plug-in will “reset” the weights to the starting values. By default affinity requests do NOT cause the plugin to decrement the weight of the server. This can lead to an uneven distribution when there are many affinity requests.

One options you have is to set the property IgnoreAffinityRequests="false". If IgnoreAffinityRequests="false" then the weight IS lowered by each affinity request, leading to a more balanced routing in a Round Robin environment. This can have some adverse consequences if you get a burst of non-affinity requests, they could all be targeted to the servers with the lowest weight and can cause issues with that server. To protect against this, you can defined the maxconnection setting in conjunction, which would avoid having too many non-affinity requests sent to a single server at any one time.

If you already have plugin traces or are willing to enable debug to triage this issue further WAS IBM HTTP Server guru and WebSphere plugin expert Eric Covener has the following scripts that will help analyze your plugin trace and provide a summary of your plugin HTTP routing statistics https://github.com/covener/plugin-tools

For recommended values of the plugin configuration please look at http://www-01.ibm.com/support/docview.wss?uid=swg21318463
To understand HTTP Plugin load balancing see http://www-01.ibm.com/support/docview.wss?uid=swg21219567
Demystifying the WebSphere Plugin http://www-01.ibm.com/support/docview.wss?uid=swg27014498&aid=1

How do you determine if this is an "affinity" request
) Check for SSLJSESSION cookie
2) Check for JSESSIONID cookie
3) Check for URL rewrite (jsessionid) in the URL
4) Check partitionID against partition table if any

TRACE: ws_common: websphereHandleSessionAffinity: Checking for session affinity
TRACE: ws_common: websphereHandleSessionAffinity: Checking the SSL cookie affinity: SSLJSESSION
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'SSLJSESSION'
TRACE: lib_htrequest: htrequestGetCookieValue: No cookie found for: 'SSLJSESSION'
TRACE: ws_common: websphereHandleSessionAffinity: Checking the cookie affinity: JSESSIONID
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'JSESSIONID'
TRACE: lib_htrequest: htrequestGetCookieValue: name='JSESSIONID',
value='0000nFKj2sPjmtT6hh5QlWyQ1rU:12n7511dh'
TRACE: ws_common: websphereHandleSessionAffinity: Checking the JSESSIONID in cookie:
0000nFKj2sPjmtT6hh5QlWyQ1rU:12n7511dh
DEBUG: ws_common: websphereParseCloneID: Parsing clone ids from
'0000nFKj2sPjmtT6hh5QlWyQ1rU:12n7511dh'
TRACE: ws_common: websphereParseCloneID: Adding clone id '12n7511dh'
TRACE: ws_common: websphereParseCloneID: Returning list of clone ids

How do you know which server was picked by the WebSphere Plugin ?
The trace shows which server was “picked”. Search the trace file (http_plugin.log) for the string “picked, weight” to quickly see which servers are being chosen in what order.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
lockedServerGroupUseServer: Server GAME3Node01_member4 picked, weight 6.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - TRACE: ws_server_group:
serverGroupIncrementConnectionCount: Server GAME3Node01_member4 picked,
pendingConnectionCount 1 totalConnectionsCount 1.
[Wed Sep 10 17:26:29 2008] 00001f18 00000cb0 - DEBUG: ws_server_group:
serverGroupNextRoundRobinServer: use server GAME3Node01_member4

First Failure Data Capture for your enterprise application with WebSphere Application Server

2011-07-08T10:50:00.000-07:00

Since WebSphere Application Server version 7, we have exposed the First Failure Data Capture (FFDC) API to customer applications. Unfortunately we have not publicized this much in the Infocenter.

One of the key points for FFDC that differentiates it from log and trace is that it is a snapshot and not history. Log and Trace tends to give a narrow view, but a history that can be highly valuable. FFDC tries to provide a wider view at a point in time.

See Javadoc
http://publib.boulder.ibm.com/infocenter/wasinfo/fep/topic/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/ffdc/config/package-summary.html
http://publib.boulder.ibm.com/infocenter/wasinfo/fep/topic/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/ffdc/package-summary.html

What is FFDC and Why do I care ?
FFDC is capturing the state of the application server and/or the application when an unexpected error or exception occurs the first time. All subsequent iterations of the same error/exception are ignored. This lightweight diagnostic collector of information is designed to be ALWAYS on in a production environment. WAS code leverages FFDC extensively to collect information and state for a component when an exception or catastrophic event occurs. You can now do the same for your enterprise application.

What differentiates it from logging ?

An exception has already occurred. This means that performance has been impacted and the code is in a failure path
FFDC is in the category of dump or snapshot type tools. These tools aim to provide a broad view of the system at a particular point in time. Logging keeps a narrow view, over a period of time
Each FFDC statement executes only once (some rare caveats). If the statement is executed again, it will know that it has already been executed, and it will simply update summary information. This means that FFDC processing can focus more on capturing all needed information, and less on performance.
The FFDC infrastructure provides many points where developers can plug in code that will be called when appropriate. The class or method experiencing a failure rarely knows all of the needed context to resolve the problem. FFDC provides extensions so that more focused serviceability code can take a more holistic view and gather a much broader context.

How do I use it ?

try { 

// Application code here 


} catch (Exception e) { 

    Ffdc.log(e, myClass, myClassName+myMethodName, “lineNumber”, cde1, cde2, …) ; 

}
// Args:  Exception, reporting class, “sourceId”, “probeId”, context data elements
// where sourceId and probeId are any strings, but this pattern is common

package howto_ffdc._1_simple;
import static com.ibm.ffdc.Manager.Ffdc;
// import com.ibm.ffdc.Ffdc; // Used if alternate call is done below
public class SimpleTest extends TestCase {
 
   protected void setUp() throws Exception {
     System.setProperty("com.ibm.ffdc.log", “/opt/IBM/WebSphere/logs/ffdc/");
    }
    
    public void testWithoutFormatter(){
  try {
         // ... do work
  } catch (Exception e) {
         Ffdc.log(e, this, getClass().getName(),"24", customer);
         /* alternate if generating the parms for the call is expensive */
         // Ffdc ffdc = Ffdc.getFfdc(e, this, getClass().getName(),"24") ;
         //if (ffdc.isLoggable())
         //   MyData myData = expensiveCallToGetData() ;
         //   ffdc.log(customer, myData) ;
   }
 }
}

An excellent deep dive into FFDC by Michael Casile, WebSphere Serviceability Team Lead shows how you can use FFDC in your Java Enterprise application with WebSphere Application Server. http://slidesha.re/pWODOZ

In summary FFDC is a simple WebSphere application server facility to improve the serviceability of you enterprise software. It is low cost to implement, extensible, can grow with you (Data Collectors, Formatters, Providers, and Incident Forwarders) and the extensions do not impact core code i.e. no change needed to Ffdc.log statements to affect improved information.

... credits to Mike Casile and the WAS Serviceability team

On an intel linux platform, what are the default java parameters that are used when starting WebSphere Application Server ? java -Xmx… etc ?

2011-07-08T09:09:00.000-07:00

You will find below ALL the user arguments used with the WAS 8 JVM.

This was taken for a javacore triggered on WAS8 on a Windows 2003 Server 64 bit WAS install.

2CIUSERARG -Xms50m

2CIUSERARG -Xmx256m

How to take a javacore

wsadmin>set objectName [$AdminControl queryNames WebSphere:type=JVM,process=server1,*]

wsadmin>$AdminControl invoke $objectName dumpThreads

Look for Javacore filename:   C:\WebSphere8\m1116.12\profiles\AppSrv01\javacore.20110708.115112.4824.0001.txt

Complete List

1CIUSERARGS UserArgs:

2CIUSERARG -Xoptionsfile=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs\options.default

2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry

2CIUSERARG -XXgc:numaCommonThreadClass=java/lang/UNIXProcess$*

2CIUSERARG -Xjcl:jclscar_26

2CIUSERARG -Dcom.ibm.oti.vm.bootstrap.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin

2CIUSERARG -Dsun.boot.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin

2CIUSERARG -Djava.library.path=C:\WebSphere8\m1116.12\java\jre\bin\compressedrefs;C:\WebSphere8\m1116.12\java\jre\bin;.;C:\WebSphere8\m1116.12\lib\native\win\x86_64;C:\WebSphere8\m1116.12\bin;C:\WebSphere8\m1116.12\java\bin;C:\WebSphere8\m1116.12\java\jre\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\development\tools\bin;C:\development\nt_java\java\bin;C:\development\tools\apache-maven-3.0.3\bin;C:\development\tools\SlikSvn\bin\;%M2%;%JAVA_HOME%\bin;C:\cygwin\bin;c:\vslick\win;C:\development\tools\apache-maven-3.0.3\bin;C:\cygwin\bin;C:\development\tools\bin;

2CIUSERARG -Djava.home=C:\WebSphere8\m1116.12\java\jre

2CIUSERARG -Djava.ext.dirs=C:\WebSphere8\m1116.12\java\jre\lib\ext

2CIUSERARG -Duser.dir=C:\WebSphere8\m1116.12\profiles\AppSrv01

2CIUSERARG _j2se_j9=1119744 0x000007FFFF237160

2CIUSERARG -Dconsole.encoding=Cp850

2CIUSERARG -Djava.class.path=C:\WebSphere8\m1116.12\profiles\AppSrv01\properties;C:\WebSphere8\m1116.12\properties;C:\WebSphere8\m1116.12\lib\startup.jar;C:\WebSphere8\m1116.12\lib\bootstrap.jar;C:\WebSphere8\m1116.12/lib/lmproxy.jar;C:\WebSphere8\m1116.12/lib/urlprotocols.jar;C:\WebSphere8\m1116.12\java\lib\tools.jar

2CIUSERARG -Declipse.security

2CIUSERARG -Dwas.status.socket=2121

2CIUSERARG -Dosgi.install.area=C:\WebSphere8\m1116.12

2CIUSERARG -Dosgi.configuration.area=C:\WebSphere8\m1116.12\profiles\AppSrv01/servers/server1/configuration

2CIUSERARG -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors

2CIUSERARG -Xshareclasses:name=webspherev80,nonFatal

2CIUSERARG -Dsun.reflect.inflationThreshold=250

2CIUSERARG -Xbootclasspath/p:C:\WebSphere8\m1116.12/java/jre/lib/ext/ibmorb.jar;C:\WebSphere8\m1116.12/java/jre/lib/ext/ibmext.jar

2CIUSERARG -Djava.class.path=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties;C:\WebSphere8\m1116.12/properties;C:\WebSphere8\m1116.12/lib/startup.jar;C:\WebSphere8\m1116.12/lib/bootstrap.jar;C:\WebSphere8\m1116.12/lib/jsf-nls.jar;C:\WebSphere8\m1116.12/lib/lmproxy.jar;C:\WebSphere8\m1116.12/lib/urlprotocols.jar;C:\WebSphere8\m1116.12/deploytool/itp/batchboot.jar;C:\WebSphere8\m1116.12/deploytool/itp/batch2.jar;C:\WebSphere8\m1116.12/java/lib/tools.jar

2CIUSERARG -Dibm.websphere.internalClassAccessMode=allow

2CIUSERARG -Xms50m

2CIUSERARG -Xmx256m

2CIUSERARG -Xcompressedrefs

2CIUSERARG -Xscmaxaot4M

2CIUSERARG -Xscmx60M

2CIUSERARG -Dws.ext.dirs=C:\WebSphere8\m1116.12/java/lib;C:\WebSphere8\m1116.12\profiles\AppSrv01/classes;C:\WebSphere8\m1116.12/classes;C:\WebSphere8\m1116.12/lib;C:\WebSphere8\m1116.12/installedChannels;C:\WebSphere8\m1116.12/lib/ext;C:\WebSphere8\m1116.12/web/help;C:\WebSphere8\m1116.12/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime

2CIUSERARG -Dderby.system.home=C:\WebSphere8\m1116.12/derby

2CIUSERARG -Dcom.ibm.itp.location=C:\WebSphere8\m1116.12/bin

2CIUSERARG -Djava.util.logging.configureByServer=true

2CIUSERARG -Duser.install.root=C:\WebSphere8\m1116.12\profiles\AppSrv01

2CIUSERARG -Djava.ext.dirs=C:\WebSphere8\m1116.12/tivoli/tam;C:\WebSphere8\m1116.12/java/jre/lib/ext

2CIUSERARG -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder

2CIUSERARG -Dpython.cachedir=C:\WebSphere8\m1116.12\profiles\AppSrv01/temp/cachedir

2CIUSERARG -Dwas.install.root=C:\WebSphere8\m1116.12

2CIUSERARG -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager

2CIUSERARG -Dserver.root=C:\WebSphere8\m1116.12\profiles\AppSrv01

2CIUSERARG -Dcom.ibm.security.jgss.debug=off

2CIUSERARG -Dcom.ibm.security.krb5.Krb5Debug=off

2CIUSERARG -Djava.library.path=C:\WebSphere8\m1116.12/lib/native/win/x86_64/;C:\WebSphere8\m1116.12\java\jre\bin\default;C:\WebSphere8\m1116.12\java\jre\bin;.;C:\WebSphere8\m1116.12\lib\native\win\x86_64;C:\WebSphere8\m1116.12\bin;C:\WebSphere8\m1116.12\java\bin;C:\WebSphere8\m1116.12\java\jre\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\development\tools\bin;C:\development\nt_java\java\bin;C:\development\tools\apache-maven-3.0.3\bin;C:\development\tools\SlikSvn\bin\;%M2%;%JAVA_HOME%\bin;C:\cygwin\bin;c:\vslick\win;C:\development\tools\apache-maven-3.0.3\bin;C:\cygwin\bin;C:\development\tools\bin;

2CIUSERARG -Djava.endorsed.dirs=C:\WebSphere8\m1116.12/endorsed_apis;C:\WebSphere8\m1116.12/java/jre/lib/endorsed;C:\WebSphere8\m1116.12\endorsed_apis;C:\WebSphere8\m1116.12\java\jre\lib\endorsed

2CIUSERARG -Djava.security.auth.login.config=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties/wsjaas.conf

2CIUSERARG -Djava.security.policy=C:\WebSphere8\m1116.12\profiles\AppSrv01/properties/server.policy

2CIUSERARG -Dsun.java.command=com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer C:\WebSphere8\m1116.12\profiles\AppSrv01\config smitaNode03Cell smitaNode03 server1

2CIUSERARG -Dsun.java.launcher=SUN_STANDARD

2CIUSERARG _port_library 0x000007FFFF235F20

2CIUSERARG _bfu_java 0x000007FFFF238230

2CIUSERARG _org.apache.harmony.vmi.portlib 0x0000000000151140

Tuning WebSphere Application Server Dynamic Cache Service (Dynacache) and Data Replication Service (DRS) for replication

2011-07-06T10:14:00.000-07:00

If you want to improve the consistency & performance of your WebSphere Application Server DistributedMap replicated cache please follow these best practices

Dynacache Tuning

1. Change replication type of the replication domain to either NOT_SHARED or PUSH depending on your replication needs. NOT_SHARED scales better than PUSH because it only replicates invalidations not cache data.

2. Ensure that the API calls that are putting objects in the cache are NOT overriding the sharing type of the replication domain

3. Set the following as JVM custom properties or as properties on your servlet/object cache instance to reduce redundant DRS traffic, improve performance & force creation of cache instances

com.ibm.ws.cache.CacheConfig.filterLRUInvalidation = true
com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation = true
com.ibm.ws.cache.CacheConfig.filterInactivityInvalidation= true * requires a relatively recent APAR. check with support
com.ibm.ws.cache.CacheConfig.ignoreValueInInvalidationEvent = true
com.ibm.ws.cache.CacheConfig.useServerClassLoader = true
com.ibm.ws.cache.CacheConfig.createCacheAtServerStartup = true * since WAS v7

4. [Optional] In the PUSH replication mode to eliminate the bootstrap of cache data, the application code that gets an instance of distributedMap should call distributedMap.setDRSBootstrap(false). Please note this will prevent the synchronization of cache instance across JVMs in the replication domain at server startup.

Data Replication Service (DRS) Tuning

Configure DRS to use its own threadpool *default since 7.0.0.15

http://www-01.ibm.com/support/docview.wss?uid=swg1PM16027

High Availability Manager (HAManager) & Data Consistency Services (DCS) Tuning

http://www-01.ibm.com/support/docview.wss?uid=swg21251873

HAManager Protocol Versions
Bump up the lower level wire format (IBM_CS_WIRE_FORMAT_VERSIONcore group custom property) and the high availability manager (IBM_CS_HAM_PROTOCOL_VERSION core group custom property) coregroup protocol versions to the latest one supported. See the following article in the Infocenter. New protocol versions are added to the product if new messages,or new algorithms are required to support new product features, orto improve core group performance. Please note that because the new messages, or newalgorithm might not be compatible with the existing messages or algorithm,a new protocol might not be able to interoperate with the old versionof the protocol.

Please also revisit my earlier FAQ and technote on cache replication
Please also read this olden goldie on Dynacache and DRS Best Practices.

Network issues

2011-07-05T14:59:00.000-07:00

If you are having consistent network disruption issues with the JVM like UnknownHostExceptions, hostname DNS lookup failures etc you can try setting the following JVM option on the IBM JREs
-Xtrace:methods={java/net/*},print=mt
to print all the activities happening over the net package giving you sufficient information to find out what is happening.

JVM networking APIs are a thin veneer over the OS APIs. There are a couple of flags that control network JVM settings i.e. preferring IPV4 or IPV6, caching the result of a local host lookup to avoid trips to the DNS & caching of lookup results. Generally speaking, all the interesting stuff happens in the OS.

If you are having network issues with HAManager and DCS WebSphere components you can configure the IBM_CS_IP_REFRESH_MINUTES custom property that controls how frequently the core group IP cache is cleared. By default the value is 60 seconds.

Java Networking Custom Properties

java.net.preferIPv4Stack (default: false)
If IPv6 is available on the operating system the underlying native socket will be an IPv6 socket. This allows Java(tm) applications to connect too, and accept connections from, both IPv4 and IPv6 hosts.If an application has a preference to only use IPv4 sockets then this property can be set to true. The implication is that the application will not be able to communicate with IPv6 hosts.

java.net.preferIPv6Addresses (default: false)
If IPv6 is available on the operating system the default preference is to prefer an IPv4-mapped address over an IPv6 address. This is for backward compatibility reasons - for example applications that depend on access to an IPv4 only service or applications that depend on the %d.%d.%d.%d representation of an IP address. This property can be set to try to change the preferences to use IPv6 addresses over IPv4 addresses. This allows applications to be tested and deployed in environments where the application is expected to connect to IPv6 services.

networkaddress.cache.ttl (default: -1)
Specified in java.security to indicate the caching policy for successful name lookups from the name service.. The value is specified as as integer to indicate the number of seconds to cache the successful lookup.A value of -1 indicates "cache forever".

networkaddress.cache.negative.ttl (default: 10)
Specified in java.security to indicate the caching policy for un-successful name lookups from the name service.. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups.A value of 0 indicates "never cache". A value of -1 indicates "cache forever".

WebSphere Application Server stopped automatically after network cable was unplugged

2011-07-01T06:34:00.000-07:00

If you find that your application server stopped automatically after a network cable was unplugged or some sort of adverse network event, please look for the following stack trace in the SystemOut.log

[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O Panic:component requested panic from isAlive
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O java.lang.RuntimeException: emergencyShutdown called: 
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.runtime.component.ServerImpl.emergencyShutdown(ServerImpl.java:633)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.runtime.RuntimeProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.coordinator.impl.JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.HAGroupImpl.doIsAlive(HAGroupImpl.java:866)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.HAGroupImpl$HAGroupUserCallback.doCallback(HAGroupImpl.java:1364)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.hamanager.impl.Worker.run(Worker.java:64)
[6/30/11 9:50:19:339 EDT] 00000018 SystemOut     O     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)

Look for any HMGR informational messages like HMGR0130I that indicate why the JVM was terminated.
If you using SIBus read this technote http://www-01.ibm.com/support/docview.wss?uid=swg21496900

From technote
"WebSphere Application Server uses database tables to store Java Message Service (JMS) messages. Anytime the messaging engine loses its connection to the data store, the JVM shuts down in an orderly fashion to prevent data loss. If the messaging engine were to continue to run and accept work, results might be unpredictable and your messaging engine might be in an inconsistent state when the data store connection is restored.

If you continually see this problem, consider modifying the Retry Interval for your data source. This value specifies the length of time, in seconds, that the application server waits before retrying to make a connection if the initial attempt fails. By default, this value is 0. If you set the value to 3, the connection is retried instead of holding onto a bad connection. You can find more information on datasource properties in the WebSphere Application Server data source properties topic within the WebSphere Application Server Information Center. "

Happy Debugging :)

Need a cheatsheat or guide to setting up ITCAM with WebSphere Application Server (WAS) ?

2011-07-01T06:19:00.000-07:00

If you wanted one place to review ALL easy to read documentation for setting up ITCAM with WebSphere Application Server please go to -

The ITCAM for AD Deployment Guide is a wiki available at https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/ITCAM+for+Application+Diagnostics+Deployment+Guide

This wiki guides you through the process of successfully deploying and maintaining ITCAM for Application Diagnostics; it includes links to the Application Diagnostics InfoCenter, and to some product technotes.

If you have time please go through this video for a deep-dive into ITCAM for WAS

Happy monitoring & debuggging !

How Mainframe customers can cut cost through FRCA caching in WebSphere Application Server for zOS

2011-06-30T08:18:00.000-07:00

If you deploy WAS on zOS and serve static (images, css files, static HTML) and dynamic content (servlets, JSPs) from your application please stop and read this blog post.

Starting WAS 7 WebSphere Application Server for z/OS can be configured to use the Fast Response Cache Accelerator facility of the z/OS Communications

Server TCP/IP FRCA has been used for years inside the IBM HTTP Server to cache static contents like pictures or HTML files.

The high speed-cache can now be used to cache static and dynamic contents, such as servlets and JSP files, instead of using the WebSphere Application Server Dynamic Cache.

ROI Benefits of using FRCA to cache static and dynamic content in a customer retail application to zOS customers can be seen in the two charts below:

Exploitation of FRCA is really as an extension to the existing Dynamic Cache (“DynaCache”) capability of each Application Server.FRCA is defined as an “External Cache Group,”and the “Adapter Bean”is what provides the function to access FRCA. Application servers > [server] > Dynamic cache service > External cache group.

In addition to enabling FRCA you will also need to write a cachespec.xml that tells Dynacache what to cache in the FRCA cache and drop it in the application's WEB-INF directory. A sample of this cachespec.xml for caching ALL static content can be found here.

Detailed description on HOW to configure FRCA for WAS can be found in the Chapter 7 of WebSphere Application Server V7 Administration and Configuration Guide, SG24-7615.

Need to troubleshoot slow http requests ?

2011-06-30T05:48:00.000-07:00

Today's post is an answer from WebSphere Application Server SWAT Team member Kevin Grigorenko in response to the question

Given the problem of a slow HTTP interaction (as measured by a load generation client for example), how would you go about gathering more detail on where the time is being spent?

1. Leverage the Trace Request Analyzer plugin from ISA with the detection gap set to 1ms. Trace and Request Analyzer for WebSphere Application Server allows you to find delays and possible hangs from WebSphere trace files and HTTP plug-in traces by parsing call trees of methods and traces and calculating delays in each method and trace.

2. Take three or four javacores spread 20 seconds apart to see if threads are hanging in particular java operation or code paths. Javacores are the poor man's profiler to debug performance issues.

3. Use Request Metrics (you can start off at Hops and then, if necessary, Performance_debug). This will print a line to SystemOut.log for each request, information like the URL and user IP, and how long it took (and if you do component level, how long each component took, e.g. a servlet forward, etc. -- I think Hops will show database times since it's a boundary). This functionality is essentially what monitoring tools use (just as an agent instead of SystemOut.log). Often, ARM to SystemOut.log has a huge overhead, so you can use a filter if you have some idea of what might be causing the slow request. http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.nd.doc/info/ae/ae/uprf_rrequestmetrics.html

4. If IHS or an ODR or some other web server fronts WAS, then you can use something like LogFormat %D or %T to print the time each request takes to access.log and then dive into that slow URL from there. You can also use this to then filter ARM in #3. http://publib.boulder.ibm.com/httpserv/manual70//mod/mod_log_config.html

5. Use a custom Request Metrics agent, that can do, for example, a javacore when a request takes over a threshold, like 1 second -- this can essentially be extended to do what a monitoring product does -- i.e. avoiding the overhead of SystemOut.log -- let me know if you need the code.

6. Use a monitoring product like ITCAM or Wily Introscope or HP Open View.

Dynacache cache policy for caching all static content in an Application

2011-06-20T14:10:00.000-07:00

Copy this into a cachespec.xml file and put this in the WEB-INF directory of your application. Please enable servlet caching on the WebContainer.


<?xml version="1.0" ?>
<!DOCTYPE cache SYSTEM "cachespec.dtd">
<cache>
       <!-- Cache Entry for the File Serving Servlet
        Will cache all static content in the web application -->
 <cache-entry>
  <class>static</class>
  <name>com.ibm.ws.webcontainer.servlet.SimpleFileServlet.class</name>
  <cache-id>
   <component id="" type="pathinfo">
    <required>true</required>
   </component> 
   <timeout>300</timeout>
  </cache-id>
 </cache-entry>
</cache>

Static HTTP Caching in relation to Dynacache

2011-06-20T13:50:00.000-07:00

There are different kinds of HTTP caches like browser caches, gateway caches and proxy caches.All these can be generically be termed as web caches. They are primarily used to reduce latency and network traffic.

There are two models of managing these HTTP caches i.e. Expiration and Validation. In expiration you specify how long a response should be considered “fresh” by including either or both of the Cache-Control: max-age=N or Expires headers. Caches that understand expiration will not make the same request until the cached version reaches its expiration time and becomes “stale”. The Validation model is used for dynamic web apps where changes in resource state occur frequently and unpredictably. Validation is usedby servers and caches to communicate when an representation has changed. Byusing it, caches avoid having to download the entire representation when theyalready have a copy locally, but they’re not sure if it’s still fresh. This is done through the use of validators. The two most common validators are 1. the time that the document last changed, as communicated in Last-Modified header and 2. ETags which are unique identifiers that are generated by the server and changed every time the representation does. Because the server controls how the ETag is generated, caches can be surer that if the ETag matches when they make a If-None-Match request, the representation really is the same.

Dynacache does NOT provide support for HTTP Cache-Control headers. This support is provided by the webcontainer on the server side and the WebSphere Proxy Server, ODR and IHS on the edge tier. Dynacache has been build ground up for page fragment caching which does not fit well with the principles of HTTP caching. The Cache-Control headers that control HTTP Caches apply to the entire response and NOT fragments. Dynacache will not look in the cache and automatically set Cache-Control headers, nor do a conditional GET to return a 304.

Static HTTP Cache-Control caching works really well for static resources like static HTML, jpg, gif, etc. These static resources can be cached in DynaCache using the static caching policy class. On WAS7 zOS the FRCA cache is an excellent place to stash your static content. The real advantage of caching using HTTP Cache-Control headers is that the cache is distributed across tiers i.e. between one or more Web servers; however the disadvantage is that the cache cannot be purged effectively on demand.

In summary, you can use DynaCache for caching both dynamic and static content; however Dynacache does NOT provide support for HTTP Cache-Control headers. Caching of static content in Dynacache is done by configuring the right cachespec.xml and leveraging FRCA technology on zOS. It is better to cache static content (images, html, css, etc) on the edge tier.

WebSphere Application Server Beta 8 has ENDED.

2011-06-17T10:50:00.000-07:00

WAS 8 Beta has ended and WAS 8.0 is now generally available for production and support.
Thank you for your participation and support.

Today June 17 2011, we eGA three critical products containing key new function, innovation, and performance enhancements. WAS V8, Compute Grid V8, and the Web 2.0 and Mobile Feature Pack 1.1 all deliver key function at a critical time for customers seeking to transform their business models for increasing growth and agility.Please note that since the IBM® WebSphere® Application Server Version 8.0 has now been officially released. You may obtain a trial version, free of charge, from here. You will also find information on how to purchase this product, as well as free trial support.

WAS V8 is the fastest, highest quality application server that is the easiest to migrate to, install and service.
See technical announcement on http://webspherecommunity.blogspot.com/
For production and support please see http://www-01.ibm.com/software/webservers/appserv/was/features
For a free developer version please see http://www.ibm.com/developerworks/downloads/ws/wasdevelopers/index.html
For a trial version please see http://www.ibm.com/developerworks/downloads/ws/was/

Reducing WebSphere Application Server I/O overhead

2011-05-30T06:19:00.000-07:00

Best practices for reducing I/O overhead of WebSphere Application Server:

If you applications on WAS are I/O intensive i.e do a lot of logging then following the best practices below will help reduce the stress on the OS I/O sub-system. If you run into issues with incomplete logging or with truncated javacores then reducing the amount of information logged to the system may help reduce the severity of the problem.

1. Disable the WebSphere Application Server service log/activity log
The service log is more commonly known as the activity.log and is found in the/profiles//logs directory. There is only oneactivity.log for each node. WebSphere Application Server runtime events arelogged to the activity.log. It is written in binary format, so it cannot be viewed in a text editor.

The main purpose of the activity.log is that it can be viewed with theLog Analyzer tool, is a graphical user interface that displays the events from theactivity.log and uses a symptom database to analyze the events and diagnoseproblems.This service is not essential to WAS and very few system administrator make use of the activity / service log.

You can configure properties of the activity.log in the administrative console:
1. Select Troubleshooting → Logs and Trace.
2. Select the WebSphere Application Server process.
3. Select IBM® Service Logs. Disable the activity.log.

Dynacache vs Memcached

2011-05-24T05:02:00.000-07:00

This is a series of posts comparing caching technologies. The first in this series is Dynacache vs Memcached.

Caching technology comparison

View more presentations from kelapure

WAS GenCon Garbage Collection Policy

2011-05-19T07:58:00.000-07:00

In WebSphere Application Server v8 that will be available from June 17 the default garbage collection policy has been changed to gencon.
You may wonder why this policy is now the default ?

■ GenCon is good for transactional workloads: when the transaction is done, most of the objects are thrown away
■ GenCon is good for interactive workloads: short GC pauses mean high responsiveness
■ Many applications fit one of these models
■ Many common Java idioms create short-lived helper objects:
– e.g. StringBuffer / StringWriter,
– e.g. Enumerator / Iterator
■ A smaller collection area means a smaller working set, improving cache utilization
■ The young generation collector compacts as it collects, reducing fragmentation
■ It also tends to relocate objects so that related objects move closer together
– For example: a String and its char[] array, or a HashTable$Entry and its key.
– This can bring significant cache benefits

Things to be aware of when using gencon
■ Tenured objects are rarely collected (by design!) -- this can lead to objects living longer than
expected. A common case of this is class unloading
■ Similar problems exist for finalization and reference objects that survive long enough to be
tenured
■ Newspace overhead: since the young generation is divided into semi-spaces, there is
always a small amount of unusable heap

Please look at this presentation on Generational Garbage Collection: Theory and Best Practices on the UK WebSphere Users Group http://www.websphereusergroup.org.uk/wug/files/presentations/31/Chris_Bailey_-_Generational_GC.pdf

WebSphere Application Server Top 10 Tuning Recommendations

2011-05-18T05:38:00.000-07:00

This blog post is based on a talk given at IMPACT 2011 by WebSphere Chief Performance Architect Surya Duggirala

#10 - Properly Tune the Operating System

Operating System is consistently overlooked for functional tuning as well as performance tuning.
Understand the hardware infrastructure backing your OS. Processor counts, speed, shared/unshared, etc
ulimit values need to be set correctly. Main player here is the number of open file handles (ulimit –n). Other process size and memory ones may need to be set based on application
Make sure NICs are set to full duplex and correct speeds
Large pages need to be enabled to take advantage of –Xlp JDK parametes
If enabled by default check RAS settings on OS and tune them down
Configure TCP/IP timeouts correctly for your applications needs
Depending on the load being placed on the system look into advanced tuning techniques such as pinning WAS processes via RSET or TASKSET as well as pinning IRQ interrupts

WAS Throughput with processor pinning

#9 – Keep Application Logging to a Minimum

Never should there be information outside of error cases being written to SystemOut.log
If using logging build your log messages only when needed

Good

if(loggingEnabled==true){ errorMsg = “This is a bad error” + “ “ + failingObject.printError();}

Bad

errorMsg = “This is a bad error” + “ “ + failingObject.printError();
If(loggingEnabled==true){ System.out.println(errorMsg); }

Keep error and log messages to the point and easy to debug
If using Apache Commons, Log4J, or other frameworks ensure performance on your system is as expected
Ensure if you must log information for audit purposes or other reasons that you are writing to a fast disk

#8 – Understand and Tune Infrastructure (databases & other interactive server systems)

WebSphere Application Server and the system it runs on is typically only one part of the datacenter infrastructure and it has a good deal of reliance on other areas performing properly.Think of your infrastructure as a plumbing system. Optimal drain performance only occurs when no pipes are clogged.
On the WAS system itself you need to be vary aware of

What other WAS instances (JVMs) are doing and their CPU / IO profiles
How much memory other WAS instance (or other OS’s in a virtualized case) are using
Network utilization of other applications coexisting on the same hardware

In the supporting infrastructure

Varying Network Latency can drastically effect split cell topologies, cross site data replication and database query latency

Ensure network infrastructure is repeatable and robust
Don’t take for granted bandwidth or latency before going into production always test as labs vary

Firewalls can cause issues with data transfer latencies between systems

On the database system

Ensure that proper indexes and tuning is done for the applications request patterns
Ensure that the database supports the number of connected clients your WAS runtime will have
Understand the CPU load and impacts of other applications (batch, OLTP, etc all competing with your applications)

On the database system

Ensure that proper indexes and tuning is done for the applications request patterns
Ensure that the database supports the number of connected clients your WAS runtime will have
Understand the CPU load and impacts of other applications (batch, OLTP, etc all competing with your applications)

On other application server systems or interactive server systems

Ensure performance of connected applications is up for the load being requested of it by the WAS system
Verify that developers have coded specific handling mechanisms for when connected applications go down (You need to avoid storm drain scenarios)

#7 – Minimize HTTP Session Content

High performance data replication for application availability depends on correctly sized session data

Keep it under 1MB in all cases if possible

Only should be storing information critical to that users specific interaction with the server
If composite data is required build it progressively as the interaction occurs

Configure Session Replication in WAS to meet your needs
Use different configuration options (async vs. synch) to give you the availability your application needs without compromising response time.
Select the replication topology that works best for you (DB, M2M, M2M Server)
Keep replication domains small and/or partition where possible

#6 – Correctly Tune Thread Pools

Thread pools and their corresponding threads control all execution on the hardware threads.
Understand which thread pools your application uses and size all of them appropriately based on utilization you see in tuning exercises

Thread dumps, PMI metrics, etc will give you this data
Thread Dump Memory Analyzer and Tivoli Performance viewer (TPV) will help in viewing this data.

Think of the thread pool as a queuing mechanism to throttle how many active requests you will have running at any one time in your application.

Apply the funnel based approach to sizing these pools

Example IHS (1000) -> WAS ( 50) -> WAS DB connection pool (30) ->
Thread numbers above vary based on application characteristics

Since you can throttle active threads you can control concurrency through your codebase

Thread pools needs to be sized with the total number of hardware processor cores in mind

If sharing a hardware system with other WAS instances thread pools have to be tuned with that in mind.
You need to more than likely cut back on the number of threads active in the system to ensure good performance for all applications due to context switching at OS layer for each thread in the system
Sizing or restricting the max number of threads a application can have can sometimes be used to prevent rouge applications for impacting others.

Default sizes for WAS thread pools on v6.1 and above are actually a little to high for best performance

Two to one ratio (threads to cores) typically yields the best performance but this varies drastically between applications and access patterns

TPV & TDMA tool snapshots

#5 –Tune JDBC Data Sources

Correct database connection pool tuning can yield significant gains in performance
This pool is highly contended in heavily multithreaded applications so ensuring significant available connections are in the pool leads to superior performance.
Monitor PMI metrics via TPV or others tools to watch for threads waiting on connections to the database as well as their wait time.

If threads are waiting increase the number of pooled connections in conjunction with your DBA OR decrease the number of active threads in the system
In some cases, a one-to-one mapping between DB connections and threads may be ideal

Frequently database deadlocks or bottlenecks first manifest themselves as a large number of threads from your thread pool waiting for connections
Always use the latest database driver for the database you are running as performance optimization in this space between versions are significant
Tune the Prepared Statement Cache Size for each JDBC data source

Can also be monitored via PMI/TPV to determine ideal value

#4 –Create Cells To Group Like Applications

Create Cells and Clusters of application servers with an express purpose that groups them in some manner
Large Cells (400-500-1000 members) for the most part while supported don’t make sense
Group applications that need to replicate data to each other or talk to each other via RMI, etc and create cells and clusters around those commonalities.
Keeping cell size smaller leads to more efficient resource utilization due to less network traffic for configuration changes, DRS, HAManager, etc.

For example, core groups should be limited to no more than 40 to 50 instances

Smaller cells and logic grouping make migration forward to newer versions of products easier and more compartmentalized.

#3 – Ensure Uniform Configuration Across Like Servers

Uniform configuration of software parameters and even operating systems is a common stumbling block
Most times manifests itself as a single machine or process that is burning more CPU, Memory or garbage collecting more frequently
Easiest way to manage this is to have a “dump configuration” script that runs periodically
Store the scripts results off and after each configuration change or application upgrade track differences
Leverage the Visual Configuration Explorer (VCE) tool available within ISA

Visual Configuration Explorer (VCE)

#2 – Correctly Tune The JVM

Correctly tuning the JVM in most cases will get you nearly 80% of the possible max performance of your application
The big area to focus on for JVM tuning is heap size

Monitor verbose:gc and target GCing no more than once every 10 seconds with a max GC pause of a second or less.
Incremental testing is required to get this area right running with expected customer load on the system
Only after you have the above boundary layers met for GC do you want to start to experiment with differing garbage collection policies

Beyond the Heap Size settings most other parameters are to extract out max possible performance OR ensure that the JVM cooperates nicely on the system it is running on with other JVMs
The Garbage Collector Memory Visualizer is an excellent tool tool for diagnosing GC issues or refining JVM performance tuning.

Provided as a downloadable plug-in within the IBM Support Assistant

Garbage Collection Memory Visualizer (GCMV)

#1 – Perform Proper Load Testing

Properly load testing your application is the most critical thing you can do to ensure a rock solid runtime in production.
Replicating your production environment isn’t always 100% necessary as most times you can get the same bang for your buck with a single representative machine in the environment

Calculate expected load across the cluster and divide down to single machine load
Drive load and perform the usual tuning loop to resolve the parameter set you need to tweak and tune.
Look at load on the database system, network, etc and extrapolate if it will support the full systems load and if not of if there are questions test

Performance testing needs to be representative of patterns that your application will actually be executing
Proper performance testing keeps track of and records key system level metrics as well as throughput metrics for reference later when changes to hardware or application are needed.
Always over stress your system. Push the hardware and software to the max and find the breaking points.
Only once you have done real world performance testing can you accurately size the complete set of hardware required to execute your application to meet your demand.

Additional Links

WebSphere Application Server Performance site
DeveloperWorks Article: Performance Tuning Case Study based on DayTrader - Step-by-step approach to tuning the application server based on a sample application
WebSphere Application Server Sample Performance Tuning Script - Can be used to adjust common tuning parameters based on predefined templates or customized to support additional fine tuning. Now available within v7.0.0

Dynacache cache entry priority questions

2011-05-12T05:50:00.000-07:00

Smaller the size of the cache and more tuned to your working set, the better performance you will get. (Smaller footprint, better concurrency etc ... )

Q Should we still avoid timeouts as our invalidation policy if at all possible?
Absolutely. Explicitly invalidating the cache-entries brings down the cache size immediately and prevents LRU activity.

Q Our application's usage is currently assigning a fixed default "Priority" for each-and-every item placed in the Cache using the DynaCache API (using timeouts as our invalidation policy).
This is fairly normal. Priorities are typically used when there are different classes of cached items some more important than others. Lower priority cache entries are considered first when it comes to making room in the cache.

Q Default priority for each Cache item. Admin versus API... what wins?
Dynacache API priority overrides priority set on the admin console. Priority determines how many cycles in the clock algorithm must pass before an unused entry is chosen as a victim. Each entry's clock starts with the default or value set programatically/cachespec,xml and is decremented each clock cycle. A clock value of <= 0 implies a victim candidate. The default is 1.
I would keep the default priority of entries as 1 and only set higher priority for those entries that are more important than the rest.

Dynacache Questions and Answers on Data Replication Service (DRS) replication best practices

2011-05-10T14:25:00.001-07:00

Questions on Dynacache best practices vis a vis replication ...

Q We plan to use one or more replication domains that span a large number of servers with a number of object cache instances, and to use sharing policy PUSH.
Unfortunately PUSH does not scale. Go with NOT_SHARED.

Q We have observed that it takes a little more than one second for the replication to begin after a cache is being looked up the first time. We want to lookup all cache instances early at the server start up process, in order to fill the caches with existing data in the cluster, making it available to the application as soon as the data is requested.
In WAS v7 Dynacache has a JVM generic customer property called "com.ibm.ws.cache.CacheConfig.createCacheAtServerStartup" which when set to "true" will create cache instances automatically at server startup instead of the default on-demand behavior. This definitely should be set if deploying to WAS7. If on an earlier version of WAS look at technote http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21313480 .
See http://wasdynacache.blogspot.com/2010/04/dynacache-and-drs-replication-faq.html .

Q We are concerned about the push mechanism and whether the initial push of say 100-150 Mb existing data distributed over perhaps 30-40 cache instances into a new-started JVM could have (temporary) performance implications on one or multiple JVM:s that are up and running.
Your concern is warranted and that is why I said PUSH does not scale. When a new JVM starts up *ALL* the other JVMs send it cached data. So the same cached data is sent by ALL the running JVMs to the new one. If this is a lot of cache data, I have often seen the older JVMs that are pushing the data die due to OOMs. If you so desire you can completely disable bootstrap by calling DistributedMap.setDRSBootstrap(false). This will prevent any JVM from sending data to the upcoming JVM.

Q When data is pushed into a newborn cache, will all cached element come from the same JVM, or could some elements be pushed from JVM 1 and others from JVM 2
Unfortunately there is no workload distribution when sending data by the existing JVMs to the new one. We don't even optimize and have one JVM send the data, instead ALL the JVMs send all the cache data in that object cache instance to the new JVM.
Some customers write cache population scripts to populate the cache for each JVM after it comes up and configure the cache in the NOT_SHARED mode which only replicates invalidations to keep the cache consistent. Other option is to use PUSH with drs bootstrap disabled.

How to detect a WebSphere Java classloader memory leak via the Eclipse Memory Analyzer tool

2011-05-10T13:40:00.001-07:00

A presentation on Class Loading and debugging Class Loader memory leaks in WebSphere Application Server by Ian is an excellent treatise on this topic. Short summary follows:
1. Run "classloader explorer" query
2. Sort classloaders by name - identify the WAS CompoundClassLoaders
3. Examine each CompoundClassLoader object - according to WAS, those whose localClassPath field is set to the empty string should be eligible for collection.
4. For each CompoundClassloader whose localClassPath is the empty, string, run "Classloader->Path to GC roots->Exclude all weak/soft/phantom etc. references."
5. Review the reference chains for each classloader and analyse for possible culprits/owners.

Debugging WAS from Dumps: Diagnose more than leaks with Memory Analyzer and its IBM Extensions pleas is another great presentation to understand how to do JVM debugging

For bonus you can also look at On the Move? WebSphere JDK Migrations, Past, Present and Future... and Beyond

These links and presentations are from the WebSphere UK User Group
http://www.websphereusergroup.org.uk/wug/downloads/31/

WebSphere Application Server - Performance considerations for deploying very large Java Enterprise Edition (JEE) applications

2011-05-10T07:15:00.000-07:00

Please consider the following when developing and deploying very large applications (> 30,000 classes) bundled in one ear file.

The cost of having a large number of classes depends strongly on details of how the classes are present in the application, and whether Java EE5 features are enabled, and whether scanning is left enabled on all of the classes.

1) If the classes are exposed in a web module archive "WEB-INF/classes" directory, that will cause very large slowdowns. The converse is for the classes to be in a JAR, either in a simple utility JAR directly contained by the application archive (EAR file), or for the classes to be contained within an EJBJar, or for the classes to be contained in a JAR beneath a web module archive "WEB-INF/lib" directory.

2) If javaEE5 processing is enabled for the classes, either because the classes are present in a javaEE5 enabled EJBJar or WAR, that will cause large slowdowns. The converse is for the classes to be present in a location which will not be scanned, for example, if there is a "META-INF/application.xml" for the EAR, and if all of the modules use a pre-javaEE5 version, or if all of the modules are marked as metadata-complete.

3) Even if javaEE5 processing is enabled, if a large subset of the classes do not require annotation scanning, that can be disabled using special properties in the application and module "META-INF/MANIFEST.MF" manifest files.

4) If using WAS version 7, please upgrade to fixpack 7.0.0.13 or later. We put in some optimizations in 7.0.0.13 to make the reading and writing to disk of applications much more efficient. Large apps (like Commerce) that are over 100 MB in size deployed 30-40% faster.

5) Check if the application is EE5 (in the application.xml, look for the version= tag). If so, check if there are any WAR files inside the EAR that contain numerous JARs under WEB-INF/lib. If those are present they can increase deployment time significantly. There is a tuning option available to help, and it's documented here PK87053: JAVA ENTERPRISE EDITION VERSION 5 APPLICATIONS TAKE A LONG TIME TO DEPLOY http://www-01.ibm.com/support/docview.wss?uid=swg1PK87053

When deploying the application, there will be an unavoidable cost of copying the application files during deployment, which will be proportional to the total size of the classes (but not proportional to the number of classes, unless these are exposed in a WEB-INF/classes directory). If the average class size is, say, 3K, and if there are 30K classes, then that will put the application size at at least 90M, which will take time to expand and to copy into the deployment location. However, that cost is only during deployment.

Note of thanks to Thomas Bitonti and David W. Hare for providing this information

IMPACT 2011 Sessions

2011-04-28T14:55:00.000-07:00

Recently I had an opportunity to present at IMPACT 2011 WebSphere Software group conference on various Java EE6 specifications that we have been working on in WebSphere Application Server 8. I also threw in my WebSphere eXtreme Scale pitch for free :). All session slides are available here http://www.websphereusergroup.org/kelapure/blog/documents.one

Integrating WebSphere eXtreme Scale to Boost the Value of...

This session presents the process, challenges, benefits and best practicesin integrating WebSphere eXtreme Scale (WXS) and the WebSphere DataPower XC10appliance with WebSphere stack middleware products includingWebSphere Application Server, WebSphere Commerce, WebSphere PortalServer, WebSphere Business Events and Rational Jazz-based products. WXS provides a powerful, elastic, high-performance solution for scalability issues through caching and grid technology. The session will also provide in-depth coverage of WXS infrastructure concepts including grid clients and servers, the grid catalog service, zone support, and scalability sizing considerations.

Getting started with Java Contexts and Dependency...

Contexts and Dependency Injection for the Java EE Platform (CDI) is one of key Java EE 6 features that help to knit together the web tier and the transactional tier of the Java EE platform. It is a set of services that, used together, makes it easy for developers to use enterprise beans along with JSF technology in web applications. This session will introduce you to fundamental concepts of CDI and explain how to leverage CDI in your application. The session is targeted to Java EE application developers, administrators and architects. The audience will walk away with an understanding of how to leverage CDI in a Java EE application, how to write a portable extension to CDI, and how to enable third party frameworks for CDI.

All things JSF 2.0: Features, Tooling, JSF-Dojo...

JSF 2.0 is a new specification that is part of the Java EE 6 technologies. This session will introduce you to the new features of JSF 2.0 and explain how you can leverage JSF in your Java EE and Portlet applications. The session will focus on three major themes: ease of development, performance improvements, and open source technology adoption. You will learn how the tools provided by Rational Application Developer make developing UI easier, as well as the JSF-Dojo component library that IBM is working on. This session is intended for Java EE developers, administrators, and architects.

Servlet 3.0: What's New

Servlet 3.0 is a new specification that is part of the Java EE 6 technologies. This session will introduce you to the new features of Servlet 3.0 and explain how you can leverage them in your applications. The session will focus on two major themes: ease of development, and improving application scalability. This session is intended for Java EE developers, administrators, and architects.

WebSphere Application Server A walk in the clouds

2011-04-28T09:14:00.000-07:00

Now that cloud and its enabling technologies like application, platform, software virtualization are hot and sexy, I am jumping on the bandwagon too :-) with some specific advice regarding WAS deployments in the cloud.

In order to leverage the full awesomeness of your WAS virtualized deployment, it is critical to do the following :

Tuning WAS for idleness: WAS system components can sometimes be more chatty than teenagers aka in corporate speak WebSphere schedules timed events to check for system updates and the availability of other servers; keeping the entire infrastructure up-to-date and running flawlessly. There are several configurable timers and options which affect the amount of background processing during idle windows. It is critical to tune these timer threads and system daemons to reduce background processing within the application server. This helps the hypevervisor technology manage the guests better. It allows virtual images to swap in and out, thereby leading to better resource utilization and less over-commit. Follow the recommendations in the document below for tuning WAS, for idleness in dense virtual environments: http://bitly.com/lC0YQe
http://webspherecommunity.blogspot.com/2011/05/tuning-for-idleness-service.html

Tune the JVM: In order to use Java effectively in the cloud you should avoid over commit, use less and share more.
– Avoid over-commit
• Matching scenarios: sandbox, legacy systems
• Limits cost savings
– Use Less
   • Trade off some performance for memory
   • Classic speed/space trade-off
   • Easy - Just some Java command line tuning
– Share More
   • Requires multiple JVMs per Guest OS
   • Easy – one additional Java command line option -Xshareclasses

Enable class sharing between JVMs using the -Xshareclasses JVM argument. This reduces the physical RAM usage when running multiple JVMs and improved start-up time. This option is ONLY available on the IBM JDKs. The shared cache contains Application classes, Metadata (JIT data, class file locations, debug, ...) and Ahead-of-time (AOT) compiled code which is shared across JVMs on the same guest OS. Tuning the shared class cache.

Select the right garbage collection policy i.e. optimize for throughput, pause time or short lived objects/web. Correctly set the heap sizes for your JVMs. Remember the maximum heap size setting should be 43% larger than the maximum occupancy of the application. Generally speaking, tune and restrict the JVM heap size to as small a value you can live with.

Problem Determination tools for WebSphere Application Server

2011-04-27T10:30:00.000-07:00

The five recommended tools that every WebSphere Application Server sysadmin and developer should have in his arsenal to debug and resolve problems are

IBM Monitoring and Diagnostic Tools for Java

Dump Analyzer
Garbage Collection and Memory Visualizer (GCMV)
Memory Analyzer (MAT)
Health Center

IBM Thread and Monitor Dump Analyzer for Java (TMDA) [Tech Preview]

When to apply these tools is summarized by the table below:

These tools can be obtained as free and fully supported add-ons to the IBM Support Assistant

See complete list of IBM Support Assistant tools: http://ibm.co/eVZa68

How to determine the version of WebSphere Application Server programatically

2011-04-26T12:25:00.001-07:00

Often times it becomes critical in operational scripting or even in your application code to determine the current runtime version of WAS. The Server mbean provides exactly this facility.
Below is a set of wsadmin script commands using AdminControl that you can use to query the Server mbean.

wsadmin>set server [$AdminControl completeObjectName cell=smitaNode02Cell,node=smitaNode02,name=server1,type=Server,*]
WebSphere:name=server1,process=server1,platform=proxy,node=smitaNode02,j2eeType=J2EEServer,version=8.0.0.0,type=Server,mbeanIdentifier=cells/smitaNode02Cell/nodes/smitaNode02/servers/server1/server.xml#Server_1183122130078,cell=smitaNode02Cell,spec=1.0,processType=UnManagedProcess

wsadmin>$AdminControl getAttribute $server state
STARTED

wsadmin>$AdminControl getAttribute $server platformName
ND

wsadmin>$AdminControl getAttribute $server platformVersion
8.0.0.0

wsadmin>$AdminControl getAttribute $server serverVersion
--------------------------------------------------------------------------------
IBM WebSphere Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time April 26, 2011 3:09:10 PM EDT

Installation
--------------------------------------------------------------------------------
Product Directory C:\WebSphere8\f1116.06
Version Directory C:\WebSphere8\f1116.06\properties\version
DTD Directory C:\WebSphere8\f1116.06\properties\version\dtd
Log Directory C:\Documents and Settings\All Users\Application Data\IBM\Installation Manager\logs

Product List
--------------------------------------------------------------------------------
NDTRIAL installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server Network Deployment
Version 8.0.0.0
ID NDTRIAL
Build Level f1116.06
Build Date 4/19/11
Architecture x86-64 (64 bit)
Installed Features IBM 64-bit SDK for Java, Version 6

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

wsadmin>$AdminControl getAttribute $server processType
UnManagedProcess

wsadmin>

This exact same code can be written in java as well using the AdminClient interface
See http://www.ibm.com/developerworks/websphere/techjournal/0302_cundiff/cundiff.html
also http://alvinabad.wordpress.com/2009/02/15/automating-websphere-using-jmx/

With these tools in hand you can interrogate all the characteristics of your current WAS server :
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.javadoc.doc/public_html/mbeandocs/index.html

Free WebSphere Application Server for Developers

2011-04-21T05:35:00.000-07:00

Today I want to highlight a free edition of WebSphere Application Server for Developers i.e. WAS4D

WAS4D is a standalone WebSphere Application Server development runtime which is

Free offering for projects that don't warrant the expense of a priced and supported runtime on the developer desktop.

Same as the WAS server type provided in Rational Application Developer for test and debug.

Identical to the production runtime environment your applications will eventually run on.

Licensing restricted to development use only.

http://www.ibm.com/developerworks/downloads/ws/wasdevelopers/index.html
Please use WAS4D to develop your JEE applications for free :-)

WebSphere Application Server HAManager Settings

2011-04-19T09:55:00.000-07:00

Sometimes customers get conflicting advice on the WebSphere Application Server HAManager component like so ...

I would enable HA on everything........ Even if applications aren't using it or don't need it, there are components/functions of WAS that rely on it
You can disable the HA Manager if the applications do not require it

The two statements are less conflicting than they may appear. The points to consider when enabling or disabling HAManager with WebSphere are as follows:

The HA Manager will use a certain amount of resources (CPU and memory).
There are a number of WebSphere components (Memory-to-memory cache replication, Session memory persistence, Stateful Session Bean failover, Singleton failover, Workload management routing, On-demand configuration routing & transaction log failover) that use the HA Manager. These components will add their own resource usage on top of what the HA Manager uses by itself. Depending on the topology, that resource usage may be significant.
New dependencies can be created on the HA Manager at any time. These dependencies may be added in the service streams (Feature in Service). While we strive to keep the list of dependencies (in the Info Center or Large Topology White Paper) up-to-date, new dependencies can be added without our knowledge.
Customers may consider disabling the HA Manager to save resource.

Generally speaking, we don't recommend that customers disable the HA Manager. We do recommend applying the most current tuning. A number of customers have disabled the HA Manager and caused themselves problems. The most common symptom will be an application routing issue. If a customer decides to disable the HA Manager, proceed with caution. Realize that you may need to re-enable it in the future.

Updated version of "Best Practices for a Large WebSphere Topology"

2011-04-18T09:26:00.001-07:00

http://www.ibm.com/developerworks/websphere/library/techarticles/0710_largetopologies/0710_largetopologies.html

JSF 1.2 Sun Reference Implementation eats too much memory in WebSphere Application Server

2011-04-18T09:22:00.000-07:00

The com.sun.faces.util.LRUMap object can hold on to a lot of memory as this is used to hold the various JSF views in the session. There are two types of JSF Views stored in the session. Logical Views in session and Number of views in session: A logical view is a top level view that may have one or more actual views inside of it. This will be the case when you have a frameset, or an application that has multiple windows operating at the same time. The LOGICAL_VIEW_MAP map is an LRU Map which contains an entry for each logical view, up to the limit specified by the com.sun.faces.numberOfViewsInSession parameter. Each entry in the LOGICAL_VIEW_MAP is an LRU Map, configured with the com.sun.faces.numberOfLogicalViews parameter.

By default the number of views stored for each of these maps is 15. Therefore you can see how it could end up using a lot of memory. The value of com.sun.faces.numberOfViewsInSession and com.sun.faces.numberOfLogicalViews does not have to be "4", it can whatever you feel is adequate for your application.If either of these parameters are not in the application then it will store up to 15 view in the LRU Maps. Setting these values to something lower will result in lower memory usage by JSF.

The actual number depends on your application. Basically, if we can't find a JSF view in the session to restore we will create new one. In general, a complex application is one that would allow a user to move back and forth to pages (think something like a wizard), or an application that contains framesets or a lot of pop up windows. For example, if a pop up window is used to fill out some information and then click submit to go back to the original page... that would require storing more views in session.

But I've always found 15 to be a high number, especially if the views are large (contains quite a lot of JSF components and their state). One thing to remember is each LogicalView can contain the set number of Actual views. That is where the idea of a frameset comes in -- one logical view for the parent page, and the actual views are the different frames.

You can find more information at the following link, starting with, "A comment on the numberOfXXX parameters."

WebSphere Critical Security Vulnerability - Denial of Service Security Exposure with Java JRE/JDK

2011-02-16T09:44:00.000-08:00

Dear WebSphere Administrators,

Please patch your WAS deployments ASAP to fix a Denial of Service Security Exposure with Java JRE/JDK hanging when converting 2.2250738585072012e-308 number (CVE-2010-4476) (PM32387)
See http://www-01.ibm.com/support/docview.wss?uid=swg21462019#solution_dist

This vulnerability can cause the Java Runtime Environment to go into a hang, infinite loop, and/or crash resulting in a denial of service exposure. This same hang will occur if the number is written without scientific notation (324 decimal places). In addition to the Application Server being exposed to this attack, any Java program using the Double.parseDouble method is also at risk of this exposure including any customer written application or 3rd party written application.

The security vulnerability is generic and specifically its a bug in sun.misc.FloatingDecimal.doubleValue
See http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/

This can be used as a denial of service attack against app servers, by sending the server an HTTP request containing this value in a field - if the server parses the value with parseDouble(), the thread doing the parsing will go into an infinite loop.

A hacker looks up a website and submits the string value "2.2250738585072012e-308" for a form field that expects double and parses it on the server side. Other instances could include an web service hosted on WAS that take a string input from a form or a URL, and tries to convert said string to a floating point variable Any code on the server side that does parseDouble is now compromised if hacker figures out how to trigger the parsing.

And remember ...

** JDK fixes are NOT dependent on the version/release of WSAS **

Oracle recently released a fix for the same
http://blogs.oracle.com/security/2011/02/security_alert_for_cve-2010-44.html
http://www.oracle.com/technetwork/topics/security/alert-cve-2010-4476-305811.html

Reference:

http://www.reddit.com/r/programming/comments/fczjc/next_language_java_hangs_when_converting/

http://news.ycombinator.com/item?id=2164863

Should I use a database or memory-to-memory replication for session failover in WebSphere Application Server

2011-02-02T19:37:00.000-08:00

credit Tom Alcott

A: Performance does not differ significantly between database persistence and memory-to-memory replication. This is because 95% of the cost of replicating or persisting sessions is incurred in the serialization/deserialization of the session object -- which must occur regardless of how the session is distributed. Also, as the size of the session object increases, performance degrades -- again, about equally for both session distribution options.

Instead, the decision will be based partially on how the two technologies differ: With a database, you actually persist the data (to disk), so a highly available database server can survive a cascading failure, while using application servers as session stores and replicators for this purpose may not. In the case of a "gold standard" (two identical cells/domains), a highly available database can pretty much assure session failover between domains, while with memory to memory, there can only be a single replicator common to the two cells; hence, it becomes a single point of failure (SPOF).

Thus, for configurations where cross-cell session failover is a requirement, a highly available database is the only option for eliminating a SPOF. Note that while sharing sessions across cells is supported, this is not generally recommended. By sharing state between cells, it makes it significantly more difficult to independently upgrade components (application and WAS) in the two cells. In the end, the decision then becomes based on what technology you are most comfortable with and which delivers the required quality of service for your availability requirements.

With memory-to-memory replication, the amount of session information you can store is bounded by the JVM heap size of your application server(s). Even with the advent of 64-bit JVM support in WebSphere Application Server V6.01, the maximum application server heap size is going to be significantly smaller than the amount of disk space you have available on a database server that is serving as a session store. Therefore, I am still of the opinion that database persistence remains the best option, although I know that in many organizations it is more expedient to use memory-to-memory replication to avoid conflicts over roles and responsibilities between system and database administrators.

A common cause of serious problems is the use/abuse of the HTTP session for all kinds of garbage. It all starts when one programmer decides to throw a little something in there, then another, and another, until your poor session looks like one of those polluted ponds filled with old tires and washing machines.
Stuffing something into the session is a very common shortcut taken under duress of an impending deadline. The effect on JVM heap sizes and performance (particularly when using persistent sessions) is dramatic. Doing code reviews (see #3) can help to prevent this. Session sizes should be from 2-4K, and there are several ways to measure them; the Tivoli® Performance Monitor that comes with WebSphere Application Server, for example, will show you session sizes.

Also read
1. Comment lines: Erik Burckart: What you want to know about HTTP session persistence
2. Chapter 12 of WebSphere Application Server V7 Administration and Configuration Guide

Eclipse Memory analyzer

2011-02-01T05:34:00.000-08:00

From John Pape : Use the Memory Analyer Tool (MAT) to diagnose problems in your Java applications

Simplifying configuration of IBM WebSphere Application Server

2011-01-31T19:08:00.000-08:00

Credit to Andy DingsorHere is the presentation from January 25, 2011, presented by my friend & colleague Andy Dingsor to the RTP WebSphere User Group. Wsadminlib.wasug.2011 0125-0726

View more presentations from kelapure.

http://wsadminlib.blogspot.com/
http://websphereusergroup.org/go/document/index

Java plug-in for WebSphere Application Server that allows a user to interrupt a hung thread

2011-01-31T18:47:00.001-08:00

IBM WebSphere Application Server Utilities Hung Thread Interrupter is now externally available from developerWorks WebSphere downloads.

This tool is a pure-Java plug-in for WebSphere Application Server and allows a user to interrupt a hung thread separately. Attached below is the link to the download page:

http://www.ibm.com/developerworks/websphere/downloads/hungthread.html

WebSphere OutOfMemory Errors

2010-10-18T18:23:00.000-07:00

If your WAS deployment is experiencing OOM issues you can setup the JVM to produce a system dump when the OOM event occurs. This system dump can later be jextracted and sucked into Eclipse Memory Analyzer to do offline analysis.
There are a variety of ways to obtain a dump from the IBM software development kit and various formats for the dump produced. It important to understand the content of each of the dumps so that the correct one can be selected for analysis of a given problem. In essence there are three forms of dump:

A system dump which is a complete dump of all the information in a process. The Dump Analyzer requires this form of dump to perform analysis.
A heap dump which gives a more terse view of the objects in the heap but does not include thread and monitor information. This form of dump is of use to other tools for heap analysis.
A summary dump sometimes called a javacore file. This is a human readable dump designed to summarise the state of the process in high level terms; the threads, monitors etc.

Dumps are produced by the JVM either on demand (a signal from the user) or on event (something happening within the VM). When the JVM starts it registers a number of event handlers which cause dumps to be generated for a default set of events. In the case of an Out of Memory event a heap dump is generated, for a user signal a javacore is generated and for a JVM crash a system dump is generated. For detailed information on setting the JVM options that control dump production.

To generate a system dump automatically on an OOM you will need to set the following JVM generic argument:-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=nodumps+exclusive+prepwalk

In order to generate java core dump, system core dump, heap dump and a snap dump at user signal, the dump agents must be configured through JVM options as follows.-Xdump:java+heap+system+snap:events=user

You can have multiple -Xdump options on the command line.
-Xdump agents are always merged internally by the JVM, as long as none of the agent settings conflict with each other.

-Xdump:java+heap+system+snap:events=user -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=nodumps+exclusive+prepwalk

-X options are specified as generic JVM arguments on WebSphere Application Server 6.1 as follows:
In the Administration Console select Servers
Select Application Servers
Click on the name of your server
In the Server Infrastructure section, expand Java and Process Management and select Process Definition > Java Virtual Machine
Scroll down and locate the textbox for Generic JVM arguments.
Please also see Crash on AIX produces no core or a truncated core to prevent a truncated core from being generated on AIX.

Please note these events ONLY work for IBM Java5 and Java6 JVMs.
Happy debugging :-)