Today blog post is a response from JVM serviceability expert Chris Bailey
Question ?
How does one determine what code in WAS is creating files and filling up the temp directory.
*war*.tmp files are created in linux /tmp directory. Who is creating them. These files can cause issues if they aren't cleaned up. Ultimately the /tmp filesystem gets full. The files look like :
abc.war26718.tmp
xyzwar26667.tmp
The best approach is to hack and patch the JVM java.io.File class and dump the thread stack in the File.delete method if the path equals the filename / dir name you are interested in. This works best if you have the environment under your control.
Response
1. Generating a system dump and using Memory Analyzer
This will let you find the File objects that own the .tmp files, and the objects that are keeping the File objects alive. This should help find out who owns it, although if the code that creates the Files is relying on deleteOnExit() to remove the files then it may no-longer have a reference to the Files itself - although most likely there will be one of more "in-flight" File objects which is still being used by the allocating code.
2. Using HealthCenter
The method profiling view will give you the method calls that call the File constructor, so assuming we have samples that have hit the call to create the File objects we should be able to see which candidates are responsible. Note that we'll only know what Files are being created by the code, not what the file is for (ie, if its a .tmp file or not).
3. Using method trace
We could trace every call to the File constructor and generate a stack trace (I think thats the "jstack" option"). This has the highest performance impact, and like Health Center doesn't know if the allocation is for a File with a .tmp suffix or not.
The constructor methods are the initializers, eg. File.
Question ?
How does one determine what code in WAS is creating files and filling up the temp directory.
*war*.tmp files are created in linux /tmp directory. Who is creating them. These files can cause issues if they aren't cleaned up. Ultimately the /tmp filesystem gets full. The files look like :
abc.war26718.tmp
xyzwar26667.tmp
The best approach is to hack and patch the JVM java.io.File class and dump the thread stack in the File.delete method if the path equals the filename / dir name you are interested in. This works best if you have the environment under your control.
Response
There are three possible approaches to this. Its probably worth taking a combination of approaches:
1. Generating a system dump and using Memory Analyzer
This will let you find the File objects that own the .tmp files, and the objects that are keeping the File objects alive. This should help find out who owns it, although if the code that creates the Files is relying on deleteOnExit() to remove the files then it may no-longer have a reference to the Files itself - although most likely there will be one of more "in-flight" File objects which is still being used by the allocating code.
2. Using HealthCenter
The method profiling view will give you the method calls that call the File constructor, so assuming we have samples that have hit the call to create the File objects we should be able to see which candidates are responsible. Note that we'll only know what Files are being created by the code, not what the file is for (ie, if its a .tmp file or not).
3. Using method trace
We could trace every call to the File constructor and generate a stack trace (I think thats the "jstack" option"). This has the highest performance impact, and like Health Center doesn't know if the allocation is for a File with a .tmp suffix or not.
The constructor methods are the initializers, eg. File.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.