d8888 888 888      88888888888 888      d8b                                 888       888          888       .d8888b.           888                               
      d88888 888 888          888     888      Y8P                                 888   o   888          888      d88P  Y88b          888                               
     d88P888 888 888          888     888                                          888  d8b  888          888      Y88b.               888                               
    d88P 888 888 888          888     88888b.  888 88888b.   .d88b.  .d8888b       888 d888b 888  .d88b.  88888b.   "Y888b.   88888b.  88888b.   .d88b.  888d888 .d88b.  
   d88P  888 888 888          888     888 "88b 888 888 "88b d88P"88b 88K           888d88888b888 d8P  Y8b 888 "88b     "Y88b. 888 "88b 888 "88b d8P  Y8b 888P"  d8P  Y8b 
  d88P   888 888 888          888     888  888 888 888  888 888  888 "Y8888b.      88888P Y88888 88888888 888  888       "888 888  888 888  888 88888888 888    88888888 
 d8888888888 888 888          888     888  888 888 888  888 Y88b 888      X88      8888P   Y8888 Y8b.     888 d88P Y88b  d88P 888 d88P 888  888 Y8b.     888    Y8b.     
d88P     888 888 888          888     888  888 888 888  888  "Y88888  88888P'      888P     Y888  "Y8888  88888P"   "Y8888P"  88888P"  888  888  "Y8888  888     "Y8888  
                                                                 888                                                          888                                        
                                                            Y8b d88P                                                          888                                        
                                                             "Y88P"                                                           888   

All Things WebSphere

Concerns and issues relating to all versions of WebSphere Application Server

Wednesday, August 31, 2011


wsadmin script to configure a data source for WebSphere Application Server

Do you know how to configure JDBC, DataSources, etc using wsadminlib.py library ?

See: https://cwiki.apache.org/ARCHIVA/archiva-on-websphere-61.html

Specifically, the file, createArchivaServer.py, from: https://cwiki.apache.org/ARCHIVA/archiva-on-websphere-61.data/createArchivaServer.py

And this is the relevant sections from the file:
    # Specify the DB2 details.
    db2DriverPath = "/usr/opt/db2_08_01/java"
    db2Alias = "DB2ArchivaUserId"
    db2UserId = "db2inst1"
    db2Password = "db2inst1"
    # JNDI details
    db2DatabaseName = "ARCHIVA"
    db2DriverType = "4"
    db2ServerName = "localhost"
    db2PortNumber = "50000"


# Set the WebSpere Variable needed to for the universal driver. In this case, we just set it for
# the individual server. To set it for the entire node, just remove the serverName variable completely.
print "Setting WebSphere Variable"
setWebSphereVariable( "DB2UNIVERSAL_JDBC_DRIVER_PATH", db2DriverPath, nodeName, serverName)

# Create a JAAS entry for the userid/password for the DB2 datasource.
# If the entry already exists, the function simply returns.
createJAAS(db2Alias, db2UserId, db2Password)
# Create the DB2 Universal Driver
print "Creating the DB2 Universal Driver Provider"
serverId = getServerId(nodeName,serverName)
jdbcProvider = createJdbcProvider ( serverId, "DB2 Universal JDBC Driver Provider", "${DB2UNIVERSAL_JDBC_DRIVER_PATH}/db2jcc.jar;${UNIVERSAL_JDBC_DRIVER_PATH}/db2cc_license_cu.jar;${DB2UNIVERSAL_JDBC_DRIVER_PATH}/db2jcc_license_cisuz.jar", "", "com.ibm.db2.jcc.DB2ConnectionPoolDataSource", "Non-XA DB2 Universal JDBC Driver-compliant Provider. Datasources created under this provider support only 1-phase commit processing except in the case where driver type 2 is used under WAS z/OS. On WAS z/OS, driver type 2 uses RRS and supports 2-phase commit processing." )
print "Created: jdbcProvider = " + jdbcProvider

# Create the JNDI Data Sources
# First, jdbc/users
print "Creating the Users Database Data Source"
usersDataSource = createDataSource( jdbcProvider, "Users Database", "jdbc/users", "jdbc/users", 10, db2Alias, "com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper" )
print "Created: usersDataSource = " + usersDataSource
setJ2eeResourceProperty( usersDataSource, "databaseName", "java.lang.String", db2DatabaseName, "The name of the database." )
setJ2eeResourceProperty( usersDataSource, "driverType", "java.lang.Integer", db2DriverType, "The JDBC connectivity-type of a data source." )
setJ2eeResourceProperty( usersDataSource, "serverName", "java.lang.String", db2ServerName, "The TCP/IP address or host name for the DRDA server." )
setJ2eeResourceProperty( usersDataSource, "portNumber", "java.lang.Integer", db2PortNumber, "The TCP/IP port number where the DRDA server resides." )

# Then, jdbc/archiva
print "Creating the Archiva Database Data Source"
archivaDataSource = createDataSource( jdbcProvider, "Archiva Database", "jdbc/archiva", "jdbc/archiva", 10, db2Alias, "com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper" )
print "Created: archivaDataSource = " + archivaDataSource
setJ2eeResourceProperty( archivaDataSource, "databaseName", "java.lang.String", db2DatabaseName, "The name of the database." )
setJ2eeResourceProperty( archivaDataSource, "driverType", "java.lang.Integer", db2DriverType, "The JDBC connectivity-type of a data source." )
setJ2eeResourceProperty( archivaDataSource, "serverName", "java.lang.String", db2ServerName, "The TCP/IP address or host name for the DRDA server." )
setJ2eeResourceProperty( archivaDataSource, "portNumber", "java.lang.Integer", db2PortNumber, "The TCP/IP port number where the DRDA server resides." )

Courtesy IBMer Chris Graham from down under.


Tuesday, August 30, 2011


Non standard performance tuning for the WebSphere Application Server

Links for some non standard and standard performance tuning of the WebSphere Application Server


For a paid offering ask your IBM services representative about the Performance Tuning Configurator (PTC)  tool for improving and tuning WAS performance



IBM Extensions to the Eclipse memory analyzer

The IBM extended version of Eclipse Memory Analyzer contains a number of plugins that help with determining the memory usage of WAS components like Dynacache, HTTPSession, WebContainer etc .. The IBM Extensions for Memory Analyzer currently offers extensions covering:  Java SE Runtime, & WebSphere Application Server

Snapshot of what this look like in the tool ...
IBM WebSphere Application Server Extensions to Eclipse Memory Analyzer
IBM Java SE Runtime Extensions to Eclipse Memory Analyzer

If you have a complete core dump this tool can also show you thread details, thread stacks, a classloader explorer etc ...
Java Thread exploration with Eclipse Memory Analyzer

My version of the IBM extended Eclipse Memory Analyzer

Happy Debugging!


Thursday, August 18, 2011


WebSphere Session Persistence Best Practices ... The 10/10 Rule

10 Seconds write frequency /10 KB session size

The most important factor in session replication performance is the write frequency setting.
This setting should be carefully considered to determine if the environment needs full reliability and failover support (End of Servlet service), or if a time based frequency is sufficient. In testing done for this study, a value larger than 10 seconds rarely provided much additional benefit. Therefore, as a best practice, if the time based frequency is chosen, we would recommend a value of 10 seconds.

 The cost of replication or persistence increases as the size of the session object increases.
Therefore, as a best practice, we would recommend keeping session sizes smaller than 10 KB.

Database session persistence performs better than Memory- to- Memory session persistence. This is because in the Database persistence mode, the application servers are not having to act as both an application server servicing requests and a replication backup, as was the case in the Memory-to-Memory scenarios. For larger environments, Database persistence is almost always the better choice as it scales a lot better than Memory-to-Memory.

There is an extra cost associated with a database server, license, and database administrator. Most environments require database backups / mirrors. The performance of the database can also be a bottleneck as the network latency between database and app server, and the speed of the database machine and disk drives are all factors. From a cost perspective Memory-to-Memory session persistence will always beat database persistence.

See this Whitepaper WebSphere Application Server V8 Replication Performance  for a complete discussion of this topic and DayTrader benchmark results.


Wednesday, August 17, 2011


Diagnostic tools for WebSphere Application Server

There are a number of  publicly available tools and scripts that WebSphere Application Server component and service teams use for assisting and problem determination of  PMRs.

WebSphere Plugin tools - Perl Script for parsing WebSphere Plugin logs

HTTPSession -  Session tracker debug Servlet
Servlet name: com.ibm.ws.webcontainer.httpsession.IBMTrackerDebug  shipped with WAS.

Dynacache: Extended Cache Monitor and Flight Recorder

HAManager: Awk scripts for analyzing  HAManager logs


SIBus –Explorer & Destination Handler
LDAP Tools

Apache Directory Studio Very handy LDAP tool based on Eclipse http://directory.apache.org/studio/index.html 
ldapSearch -  LDAP search tool http://linux.die.net/man/1/ldapsearch

Sun JVM ONLY Tools- jps, jstatd, jstack, jmap, jinfo, jhat visualVM & jConsole

Name space dumper $WAS_HOME/bin/dumpNameSpace tool

OSGI Bundle Explorer and debug utility  $WAS_HOME/bin/osgiConsole.sh/bat

ISA Tools


Thursday, August 11, 2011


Generating system dumps and thread dumps automatically

Generating a system dump instead of a heap dump automatically on an OOM
To generate a system dump as default on an OOM error or when the JVM receives the SIGQUIT.
Please set the following JVM generic custom property
Please also ensure that there is adequate space for the system dump by following this technote .

Generating a thread dump automatically on a thread hang
To configure a hang detection policy that dumps a javacore when a thread is hung by setting the following property
  1. From the administrative console, click Servers > Application Servers > server_name OR  System Administration > Deployment Manager  or  System Administration > NodeAgent > node_agent_name
  2. Under Server Infrastructure, click Administration > Custom Properties
  3. Click New.
  4. Add the following property:
Name com.ibm.websphere.threadmonitor.dump.java
Value true

This causes  a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.

 Table of where to set a JVM Custom property

Application server Servers > Server Types > WebSphere application servers > server_name > Java and Process Management > Process Definition > Java Virtual Machine > Custom Properties
Deployment manager System Administration > Deployment Manager > Java and Process Management > Process definition > > Java Virtual Machine > Custom Properties
Node agent System Administration >Node Agent > nodeagent > Java and Process Management > Process definition > Java Virtual Machine > Custom Properties


http:http//publib.boulder.ibm.com/infocenter/javasdk v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Ftools%2Fdiag_collector.html

Tuesday, August 9, 2011


JAX-RS implementations in WebSphere Application Server

Jersey is the open source JAX-RS (JSR 311) Reference Implementation for building RESTful Web services. Jersey provides an API so that developers may extend Jersey to suit their needs. See blog post on how to integrate Jersey with WebSphere Application Server 6.1. If the first suggestion does not work look at this post.

Gotcha To get Jersey working with WAS v7 you will need to set the webcontainer custom property.
Name: com.ibm.ws.webcontainer.invokeFiltersCompatibility
Value: true
This workaround is NOT needed in WAS v8. See APAR PK56247   for details on why you need to do this in WAS v7.

If your application that leverages Jersey  throws the following error during deployment
The URI scheme wsjar of the URI wsjar:file:/C:/rad802/workspace/myCompany/TestJerseyProvider/WebContent/WEB-INF/lib/Providers.jar!/com/mycompany/commons/web/rest/providers is not supported. Package scanning deployment is not supported for such URIs.

This occurs because your classes in web-inf/lib are not being recognized, because the Jersey URI Scheme scanner which scans WEB-INF/lib only works, when the protocol has 3 digits (like jar:) but WebSphere uses wsjar: as protocol.

We ran into a similar issue when using Open Web Beans see https://issues.apache.org/jira/browse/OWB-577
You can also look at the svn change history to see how we fixed it in Open Web Beans.

This is fixable in Jersey by adding your own com.sun.jersey.core.spi.scanning.uri.UriSchemeScanner  which is the interface for scanning URI-based resources and reporting those resources to a scanning listener. Jersey's code add's additional UriSchemeScanner's using the com.sun.jersey.spi.service.ServiceFinder. Looking at existing scanners like the the com.sun.jersey.core.spi.scanning.uri.JarZipSchemeScanner & com.sun.jersey.core.spi.scanning.uri.FileSchemeScanner you will be able to write a scanner that is able to scan the jars in the WEB-INF/lib directory of your WAS application.

Another option is to do what the warning says  Try using a different deployment mechanism such as explicitly declaring root resource and provider classes using an extension of javax.ws.rs.core.Application.

Please note that WAS ships with its own JAX-RS implementation based off the Apache Wink Project. This is available by default in WAS version 8 and with the Web 2.0 feature pack in WAS v7.


Saturday, August 6, 2011


Tune your WebSphere Application Server deployment for Production or Development

Do you know that WebSphere Application Server install contains scripts to tune the performance of the server ? These can be found in   $WAS_HOME/scriptLibraries/perfTuning/V70

This wsadmin script can be run as follows ....
$WAS_HOME/bin/wsadmin -f applyPerfTuningTemplate.py [-nodeName <node> -serverName <server>] [-clusterName <cluster>] -templateFile <file> where templateFile could be one of "peak.props",  "development.props" or "default.props"

# File Name:
#      ApplyPerfTuning.py
# Purpose:
#      Apply pre-defined performance tuning templates to a specific
#      application server instance or all instances in a cluster. Three
#      templates are pre-defined (default, peak, and development);
#      however, these files can easily be modified to create your own custom
#      tuning template.

You can tune WAS either for development (faster application restarts, smaller footprint, dynamic threadpool  sizes) or production (increased throughput, smaller garbage collection pauses, Fixed threadpool sizes). Correctly tuning WAS can vastly improve your developer and customer experience.

For a complete case study in  basic and advanced tuning see   http://www.ibm.com/developerworks/websphere/techjournal/0909_blythe/0909_blythe.html . The article discussed this core set of tunables plus other option resulting in performance gains in excess of 100%.


Friday, August 5, 2011


Configure WebSphere Application Server Dynacache and Session persistence to use WebSphere eXtreme Scale

Please see instructions below on how to configure Dynacache to leverage WXS instead of the default caching engine. I have also provided links for detailed instructions if you get confused or stuck/
Video  eXtreme Scale  Wiki  Configuring cache integration
Redbook chapter on integrating WXS with Dynacache in the context of WebSphere Commerce
Configure WebSphere Commerce to use WebSphere eXtreme Scale for dynamic cache to improve performance and scale

You will also be interested in configuring session persistence with WXS.
Infocenter  Configuring the HTTP session manager with WebSphere Application Server
Attention: With Version and later, you can persist sessions that use URL rewriting or cookies as a session tracking mechanism. Before Version, you can save only sessions that use cookies as the session tracking mechanism to the data grid.


1. Install & Set up an WXS 7 grid (Catalog server and other XS containers that host the cache) . The latest WXS version has an admin console for monitoring. Please install the latest WXS 7 fix pack before proceeding. 

2. Augment the WAS profile with the WXS client

3.  Configure the dmap to use extreme scale 
Basically add the following properties to the map where we create the dmap
com.ibm.ws.cache.CacheConfig.enableCacheReplication  = true
com.ibm.ws.cache.CacheConfig.cacheProviderName = com.ibm.ws.objectgrid.dynacache.CacheProviderImpl 
com.ibm.websphere.xs.dynacache.topology = remote
com.ibm.websphere.xs.dynacache.num_initial_containers = 1

4. [optional] XS does not place replica and primary shards on containers that have the same IP address (unless the configuration is in development mode). If ALL your XS containers are on one physical machine or node you will need to set development=true. In the deployment policy descriptor XML file after WXS client is installed  i.e. $WAS-HOME/optionalLibraries/ObjectGrid/dynacache/etc/dynacache-remote-deployment.xml set developmentMode to true

developmentMode="false" --> true
  <map ref="IBM_DC_PARTITIONED_.*" />


Wednesday, August 3, 2011


WebSphere Application Server Defibrillator for stalled software

Food for thought  ...

I have a lot of interest in Java serviceability and JVM egregious condition  recovery. If you share the same interest, please check out the following research paper http://groups.csail.mit.edu/pac/jolt/paper.pdf that talks about Detecting and Escaping In nite Loops with Java or the more easily digestible http://web.mit.edu/newsoffice/2011/infinite-loops-interrupt-0802.html

One thought that did resonate in the article is from Westley Weimer, an assistant professor of computer science at the University of Virginia who says "The vast majority of software engineering or programming-language researchers are shackled to this notion of full correctness or full soundness: You can’t change anything in the program if there’s even the possibility of getting a slightly wrong answer,”

The article goes on to say that  “One of the things that’s really characterized Martin’s research over the last 10 or 15 years is casting off the shackles of soundness in favor of approaches that are probably correct but dramatically more useful in real life.”

This is true in order to achieve resiliency you will need to sacrifice some program consistency and correctness.


WebSphere Application Server SWAT BLOG

For all those interested in WebSphere Application Server serviceability and problem determination please follow the WAS Swat blog run by Kevin Grigorenko WAS troubleshooting extraordinaire https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/?lang=en

Tuesday, August 2, 2011


How to analyze IBM JVM core dumps

Today's blog post is advice, I provided to a customer for analyzing IBM JDK core dumps

Question ... 
We have a problem where our Linux-based build machines for the portal drivers, occasionally core dump in the middle of the build process. The only way I know of to invoke the Java team to shoot these, is to open a WAS PMR.  I'm not sure that's appropriate, given that it's unclear exactly what JRE is running at the time (it could be the one from the WAS or from ant)

In order to analyze a coredump you will need to run the Dump Analyzer plugin from IBM Support Assistant (ISA). The dump report will tell you the details of the JVM that dumped core, the heap contents and the javacore. This will help you determine the culprit component faster.

First see these instructions to run the dump analyzer outside the ISA

cd  $ISA_HOME\applications\eclipse\plugins\com.ibm.java.diagnostics.dbda_2.2.2.20090926232659
java -Xmx1400M -cp dumpAnalyzer.jar;lib\dtfj.jar;..\com.ibm.java.diagnostics.dbda.was_2.1.3.20090427213613\wasAnalyzers.jar;..\com.ibm.java.diagnostics.dbda.core_2.2.2.20090926232659\coreAnalyzers.jar com.ibm.dtfj.analyzer.base.DumpAnalyzer core1.sdff com.ibm.dtfj.analyzer.jvm.DefaultDumpReport_Extended > DumpAnalyzer_DefaultDumpReport_Extended.txt


You basically need to  copy the 6 required jars from IBM Support Assistant and then run the command
set CP="dumpAnalyzer.jar;coreAnalyzers.jar"
set BCP="lib/j9/dtfj.jar;lib/j9/dtfj-interface.jar;lib/sov/dtfjsu.jar;lib/sov/dtfj.jar"
java -cp $CP -Xbootclasspath/p:$BCP
        com.ibm.dtfj.analyzer.base.DumpAnalyzer (dumpName) (options)

You need to put this in a script and run it whenever a core dump is generated.

Happy Debugging!


Monday, August 1, 2011


Solving an application performance slowdown or regression with WebSphere Application Server

Recently I had the chance to troubleshoot and fix a performance regression a component of the WebSphere Application Server. The objective was to determine performance and locking hotspots in our server code.

1. Beg, borrow steal or write a benchmark. We already have a pretty good Dynacache performance benchmark. This is critical to draw a line in the sand and determine a before and after performance #s.

2. Enable IBM Health Center profiling. Using IBM Health Center which is baked into the WAS JVM, is the easiest and lightest way of doing profiling with WebSphere Application Server.  No expensive profiling tools and  no instrumentation of src. This is best thing since baked bread for performance in WAS.

3. So you need two things for this exercise. On the client side you need an IBM Health Center Client that comes as a free product plugin to the IBM Support Assistant. On the server side you need IBM Health Center server library which comes preinstalled with the JVM which is bundled with WAS. You will find one WAS_HOME/java/jre/lib/ext/healthcenter.jar

4.  Configure WAS to use the IBM Health Center. See details in Infocenter  for older versions of java.
Servers ->Server Types -> WebSphere application serversSelect the server name and then select Java and Process Management -> Process definition -> Java Virtual Machine -> Generic JVM Arguments.
For Java 5 SR10 and later, or Java 6 SR5 and later, use:-Xhealthcenter

5. After restaring WAS, start the ISA and configure a connection to your WAS server. This connection is established typically on port 1872. If you want to control the server port see infocenter. After the JVM is started with the agent enabled, you see a message detailing the port for the Health Center agent. For example:
05-Mar-2009 09:49:57 com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgentINFO: Health Center agent started on port 1972.

6. Run the benchmark or the test.

7.  After the test save the profile as a *.hcd file. You can then pass this profile around and analyze it offline. See technote.

8. Interpret the method profile.  The Method Profile table shows which methods are using the most processing resource.
  • Methods with a higher Self (%) value are described as "hot", and are good candidates for optimization. Small improvements to the efficiency of these methods might have a large effect on performance. Methods near the bottom of the table are poor candidates for optimization. Even large improvements to their efficiency are unlikely to affect performance, because they do not use as much processing resource.
  • You can optimize methods by reducing the amount of work that they do or by reducing the number of times that they are called. Highlighting a method in the table populates the call hierarchy views.Filter the contents of the method profile table using the text box above the table. See the filtering help topic for more information.
  • Additionally, when you select the Hide low sample entries, the table does not list any entries that have a sample count of less than 2. Use this option if your table contains many entries that are not obvious candidates for optimization to improve the performance of the table.

9. Make java source code changes to improve performance. Scrub and repeat. We used the process above to take before and after profiles to figure out the delta i.e .where the performance had slipped.

Happy profiling!



December 2006   September 2008   January 2009   February 2009   March 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   October 2010   January 2011   February 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   September 2012   October 2012   November 2012   January 2013   May 2013   June 2013   July 2013   September 2013   October 2013   June 2014   August 2014  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]