On 03.02.2009 16:31, Steve Cohen wrote:
We have an application that runs under Tomcat under RHEL 5.0 and is
launched by a jsvc daemon.
It chugs along seemingly fine on several servers, yet yesterday crashed
on one of them with the above exception seemingly without experiencing
any kind of abnormal load. I am trying to get a handle on this.
System is
$ uname -a
Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686
i386 GNU/Linux
Java version:
java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing)
no special JVM memory settings are used when launching tomcat, just the
default memory allocations. I have read in some places that this
particular form of OutOfMemoryError is actually made worse by increasing
the memory size.
I need to deploy some thread and memory monitoring setup to get a handle
on this.
I wouldn't put so much work into making X11 work.
You need to find out your memory situation and your thread situation.
The error says: when trying to start a new thread, native memory
allocation failed and thus the thread could not be created.
So the reason is, that you ran out of native memory in the process. The
cure is either to reduce the amount of memory needed or to reduce the
number of threads needed.
Concerning Memory:
Basically your process uses Java memory (Heap and Permanent generation)
and native memory (thread stacks and other native memory).
If you are using a 32Bit system or VM, the address space is 4GB per
process, but that space is divided in some part available for the
process, and some part used by the kernel for data associated with the
process. The part actually available for the process depends as far as i
know on kernel settings and can be either 1, 2, or 3 GB (I would expect
2GB).
Heap, Perm and native memory has to fit into this. I'm not quite sure
about the thread stacks, but I assume those must fit into the 1/2/3 GB too.
Process memory can be inspected using the proc file system. The file
/proc/MYPID/maps (MYPID is the process id of your Tomcat process)
contains a table of memory areas used by the process. The hex numbers at
the beginning of each line shows starting and ending addresses, the
differences are the sizes of the areas.
Check whether you can verify, that the process is close to one of those
magic limits.
Then check, how big your heap and your permanent generation is. You can
do this with jstat. I would sugest you find a good sizing for your app
and don't proceed with automatic 8and thus unknown) sizing.
If total memory is close to the limit and heap+perm is close to the
total memory, you know that your java memory is to big for 32bits.
If total memory is close to the limit and heap+perm is much lower, then
there's something which consumes large native memory. It could be
threads (see next) or some embedded native component, e.g. some library
included via JNI.
Threads:
Use "kill -QUIT MYPID" with MYPID the process id of your Tomcat. It will
write a thread dump (not: memory dump) to catalina.out. This dump
contains all Java threads of your process and the stacks they are in.
See how many these are and to which components they belong. With a
thread dump you can find out, whether you have extremely many Java
threads. You need to be the owner of the Tomcat process or root to be
able to send a signal to the process. You need to be able to read
catalina.out to analyse the output.
Use the "-L" flag of "ps" to look at all threads of the Tomcat process.
Check whether there are many more threads in ps, than there are in the
Java thread dump. If so, you have some native component inside, that
creates additional threads. In this case use gstack, to check, what
those are doing (to get an idea, what they are used for).
If you can verify that you run our of native memory, but you can reduce
neither memory size not thread count, you can start playing around with
the thread stack size using -Xss. But this will be tedious.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
