Caldarale, Charles R wrote:
From: André Warnier [mailto:a...@ice-sa.com]
Subject: Re: CLOSE_WAIT and what to do about it
Relatedly, does there exist any way to force a given
JVM process to do a full GC interactively, but from a
Linux command-line ?
Found a command line tool that will do what you want:
http://code.google.com/p/jmxsh/
I've used it to trigger a GC in Tomcat via the following steps.
1) Start Tomcat with the following options:
-Dcom.sun.management.jmxremote.port=<port>
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
(You can, of course, set the authentication and SSL options as needed.)
2) Start jmxsh from the directory its jar is in with this:
java -jar jmxsh*.jar
3) Enter the following commands (but not the bracketed bits):
jmx_connect -h localhost -p <port>
[blank line to enter browse mode]
5 [selects java.lang]
1 [selects the Memory mbean]
5 [performs a GC]
The doc for jmxsh indicates the above steps should be scriptable, but I haven't
tried that.
It is likely that you could use jmx_connect with a different kind of service
and avoid opening up an RMI port; if I figure that out, I'll let you know.
Hi.
Thanks a million for providing the above info.
That jmxsh program is really useful.
I don't really know what I'm doing here, but I can at least more or less
figure out what happens.
To recall, my original issue is that I have some Java applications
(among which a Tomcat webapp and a couple of stand-alone Java
daemon-like programs) which apparently leave an ever-increasing number
of sockets lingering in a CLOSE_WAIT state.
And I was wondering if it was possible, as one test, to force the JVM
running these applications to perform a GC, right now, from "the outside".
Well, it is.
Following is a trace of a session with jmxsh, with one of these
applications.
Initial socket situation :
r...@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6 0 0 ::ffff:127.0.0.1:48267 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 12 0 ::ffff:127.0.0.1:36936 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
tcp6 12 0 ::ffff:127.0.0.1:50322 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
r...@arthur:/home/star/xml# ps -ef | grep 7618
root 7618 1 1 14:32 pts/3 00:00:15 ./java -server
-Dcom.sun.management.jmxremote.port=11201
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false -Xms64M -Xmx64M
-Dpgm=STARWeb -jar /home/xxxx/web4/java/xyz.jar -c
/home/star/web4/config -p 11101
The above is the process which I am going to "stress", in the sense of
communicating with it, which has the result of having it itself open a
TCP connection with another server listening on port 11002, then closing
this socket (in principle), and this multiple times.
(As you see, the program was started with the "jmxremote" options
allowing later communication with jmxsh.)
Now some interactions with the application pid=7618 ...
Situation later on :
r...@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6 0 0 ::ffff:127.0.0.1:55798 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 0 0 ::ffff:127.0.0.1:57029 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 0 0 ::ffff:127.0.0.1:48267 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 0 0 ::ffff:127.0.0.1:56781 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 12 0 ::ffff:127.0.0.1:36936 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
tcp6 12 0 ::ffff:127.0.0.1:58341 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
tcp6 0 0 ::ffff:127.0.0.1:32972 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 12 0 ::ffff:127.0.0.1:50322 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
So this application indeed left a number of sockets in the CLOSE_WAIT state.
Now triggering a GC with jmxsh :
a...@arthur:~$ java -jar bin/jmxsh-R4.jar
jmxsh v1.0, Tue Jan 22 17:23:12 GMT+01:00 2008
Type 'help' for help. Give the option '-?' to any command
for usage help.
Starting up in shell mode.
% jmx_connect -h localhost -p 11201
Connected to service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi.
%
Entering browse mode.
====================================================
Available Domains:
1. java.util.logging
2. JMImplementation
3. java.lang
SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi
====================================================
Select a domain: 3
====================================================
Available MBeans:
1. java.lang:type=Compilation
2. java.lang:type=MemoryManager,name=CodeCacheManager
3. java.lang:type=GarbageCollector,name=Copy
4. java.lang:type=MemoryPool,name=Eden Space
5. java.lang:type=Runtime
6. java.lang:type=ClassLoading
7. java.lang:type=MemoryPool,name=Survivor Space
8. java.lang:type=Threading
9. java.lang:type=GarbageCollector,name=MarkSweepCompact
10. java.lang:type=OperatingSystem
11. java.lang:type=Memory
12. java.lang:type=MemoryPool,name=Code Cache
13. java.lang:type=MemoryPool,name=Tenured Gen
14. java.lang:type=MemoryPool,name=Perm Gen
SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi
DOMAIN: java.lang
====================================================
Select an mbean: 11
====================================================
Attribute List:
1. -r- javax.management.openmbean.CompositeData
NonHeapMemoryUsage
2. irw boolean Verbose
3. -r- int ObjectPendingFinalizationCount
4. -r- javax.management.openmbean.CompositeData
HeapMemoryUsage
Operation List:
5. void gc()
SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi
DOMAIN: java.lang
MBEAN: java.lang:type=Memory
====================================================
Select an attribute or operation: 5
=====================================================
Invoking Operation gc
Invoked.
Press enter to continue.
====================================================
Select an attribute or operation:
Entering shell mode.
% jmx_close
% exit
a...@arthur:~$
Socket situation after the GC :
r...@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6 0 0 ::ffff:127.0.0.1:55798 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7618/java
tcp6 12 0 ::ffff:127.0.0.1:36936 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
tcp6 12 0 ::ffff:127.0.0.1:58341 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
tcp6 12 0 ::ffff:127.0.0.1:50322 ::ffff:127.0.0.1:11002
CLOSE_WAIT 7816/java
r...@arthur:/home/star/xml#
And indeed all but one of the CLOSE_WAIT sockets of process-id 7618 have
disappeared.
=========================================
I will now risk an interpretation :
If these sockets disappear during a GC, then it must mean that they are
still being referenced by some "abandoned" objects sitting on the Heap,
which have not yet been reclaimed by the GC.
Which probably means that the objects in question have gone out of
scope, before the socket they used was properly close()'d.
Yes ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
