Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
On 03.02.2009 16:31, Steve Cohen wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version "1.5.0_16" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. I wouldn't put so much work into making X11 work. You need to find out your memory situation and your thread situation. The error says: when trying to start a new thread, native memory allocation failed and thus the thread could not be created. So the reason is, that you ran out of native memory in the process. The cure is either to reduce the amount of memory needed or to reduce the number of threads needed. Concerning Memory: Basically your process uses Java memory (Heap and Permanent generation) and native memory (thread stacks and other native memory). If you are using a 32Bit system or VM, the address space is 4GB per process, but that space is divided in some part available for the process, and some part used by the kernel for data associated with the process. The part actually available for the process depends as far as i know on kernel settings and can be either 1, 2, or 3 GB (I would expect 2GB). Heap, Perm and native memory has to fit into this. I'm not quite sure about the thread stacks, but I assume those must fit into the 1/2/3 GB too. Process memory can be inspected using the proc file system. The file /proc/MYPID/maps (MYPID is the process id of your Tomcat process) contains a table of memory areas used by the process. The hex numbers at the beginning of each line shows starting and ending addresses, the differences are the sizes of the areas. Check whether you can verify, that the process is close to one of those magic limits. Then check, how big your heap and your permanent generation is. You can do this with jstat. I would sugest you find a good sizing for your app and don't proceed with automatic 8and thus unknown) sizing. If total memory is close to the limit and heap+perm is close to the total memory, you know that your java memory is to big for 32bits. If total memory is close to the limit and heap+perm is much lower, then there's something which consumes large native memory. It could be threads (see next) or some embedded native component, e.g. some library included via JNI. Threads: Use "kill -QUIT MYPID" with MYPID the process id of your Tomcat. It will write a thread dump (not: memory dump) to catalina.out. This dump contains all Java threads of your process and the stacks they are in. See how many these are and to which components they belong. With a thread dump you can find out, whether you have extremely many Java threads. You need to be the owner of the Tomcat process or root to be able to send a signal to the process. You need to be able to read catalina.out to analyse the output. Use the "-L" flag of "ps" to look at all threads of the Tomcat process. Check whether there are many more threads in ps, than there are in the Java thread dump. If so, you have some native component inside, that creates additional threads. In this case use gstack, to check, what those are doing (to get an idea, what they are used for). If you can verify that you run our of native memory, but you can reduce neither memory size not thread count, you can start playing around with the thread stack size using -Xss. But this will be tedious. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Bill Davidson wrote: I really would like to be able to run these kind of tools across a firewall. I can't believe there are no provisions for it. Seems there is now. Haven't tried it yet, but will soon, if I can upgrade that Tomcat. Previous messages in this thread give more clues. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: What about forwarding X through the tunnel instead? I've tried that and found it to be unusably slow. Just using Cygwin/X though. I'm not sure about those fancy light weight X compression systems. I really would like to be able to run these kind of tools across a firewall. I can't believe there are no provisions for it. It has to be something that a lot of people want to do. If I can run Oracle connections over an ssh tunnel with good performance, it seems like I should be able to do something like jconsole data too. RMI's random port thing is really annoying. It seems like there ought to be some way to specify all ports involved. Whomever designed it apparently didn't give any thought to working across firewalls. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
> From: André Warnier [mailto:a...@ice-sa.com] > Christopher Schultz wrote: > > What about forwarding X through the tunnel instead? > > > You can't, because it is variable. It is the result of some internal > "negotiation" between Jconsole and the remote JVM. > Apparently, anyway. I haven't managed to make it work so far. To expand Chris' suggestion: What about forwarding the X11 protocol through the tunnel instead, such that you can run Jconsole on the same machine as the JVM? X11 has a fixed port number, and ssh has support for this. I've used this trick when needing to do something graphical on a box with ssh-only access; works a treat. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: Steve Cohen wrote: [...] Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Not really a help here, but as I have discovered JConsole wont't even work through a simple SSH tunnel, because it wants to dynamically open a second parallel connection, which of course your tunnel knows nothing about. What about forwarding X through the tunnel instead? You can't, because it is variable. It is the result of some internal "negotiation" between Jconsole and the remote JVM. Apparently, anyway. I haven't managed to make it work so far. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Steve, Steve Cohen wrote: > One additional restriction I have is a security bureaucracy from hell. You can use jmap from the command-line. That will give you a thread dump which can help you see what all your threads are doing. If you are using a system that uses POSIX threads, it might also help to see what the output of 'ulimit -u' which will tell you the process limit for your user. POSIX threads are essentially processes, so threads count against your process limit. The obvious answer is that you simply have too many threads being created. There are a number of reasons you could be running out of threads. Here are some: 1. Too many request processor threads configured in server.xml (see your element). Inspect your configuration. You may have more connectors configured than necessary (I've seen configurations where both HTTP and AJP connectors are both enabled but only one is being used). 2. Too much file I/O. If you are using background threads to read/write files independently of the request processor thread, you might be creating too many of them. I recommend using a thread pool to schedule background jobs. 3. Timer threads. If you are using java.util.Timer, you may be creating more threads than you realize. Check your use of java.util.Timer and possibly use a thread pool for timed tasks. 4. Quartz, Quartz, Quartz (this is a job scheduler that generates a lot of questions on this list). Check any uses of Quartz. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmJ2YQACgkQ9CaO5/Lv0PBH1gCgq+lrPmgTjRhEF1JEf9jBs4oe Vq4AoJ18SigCd8pkSCh4eR0qDOawyhpu =tIYg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: > Steve Cohen wrote: > [...] >> >> Which means I'm looking for command-line equivalents for what >> JConsole does, particularly the threads tab. Any ideas along that path? >> >> > Not really a help here, but as I have discovered JConsole wont't even > work through a simple SSH tunnel, because it wants to dynamically open a > second parallel connection, which of course your tunnel knows nothing > about. What about forwarding X through the tunnel instead? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmJ1s4ACgkQ9CaO5/Lv0PAIKACgtZdXVlSsic2fpO5sEBE2RBqB 8tAAnjRZcJFC0r9FMrefKI062OgXic24 =FB4k -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Dear Steve, One additional restriction I have is a security bureaucracy from hell. As far as console access I have a two-hop connection. Is there no test system that you can use to reproduce this, under less insane restrictions? Have you considered hauling your behind to the data center to sit behind the console for a few hours? In the past I found that I could spend days fighting the bizarre rules with half- working technical solutions, or just go to the actual machine and look at the display there. Of course, it was only a two hour drive in my case. Yours might well be at the other end of the world. As far as GUI access I have some crappy device to log into that lets me have really bad keyboard/mouse control of the machine, which, unless I can overcome this means that GUI-based tools are not going to work, and as for remote GUI access, see above security bureaucracy. Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? There are some JMX command line tools, but JMX is really easy to use from code. You could write a simple JSP page to give you the figures that way. Script with curl/wget and graph with Excel. :) -- Kees Jan http://java-monitor.com/forum/ kjkos...@kjkoster.org 06-51838192 The secret of success lies in the stability of the goal. -- Benjamin Disraeli - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Hi Steve, Also, a good tool for monitoring Tomcat is Lamda Probe http://www.lambdaprobe.org/d/index.htm Regards, Ovidiu On Tue, Feb 3, 2009 at 9:31 AM, Steve Cohen wrote: > We have an application that runs under Tomcat under RHEL 5.0 and is > launched by a jsvc daemon. > It chugs along seemingly fine on several servers, yet yesterday crashed on > one of them with the above exception seemingly without experiencing any kind > of abnormal load. I am trying to get a handle on this. > System is > > $ uname -a > Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 > GNU/Linux > > Java version: > java -version > java version "1.5.0_16" > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) > Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) > > no special JVM memory settings are used when launching tomcat, just the > default memory allocations. I have read in some places that this particular > form of OutOfMemoryError is actually made worse by increasing the memory > size. > > I need to deploy some thread and memory monitoring setup to get a handle on > this. > > Can someone suggest a path forward here? > > Thanks. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- ovidiu asiminei
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Steve Cohen wrote: [...] Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Not really a help here, but as I have discovered JConsole wont't even work through a simple SSH tunnel, because it wants to dynamically open a second parallel connection, which of course your tunnel knows nothing about. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
One additional restriction I have is a security bureaucracy from hell. As far as console access I have a two-hop connection. As far as GUI access I have some crappy device to log into that lets me have really bad keyboard/mouse control of the machine, which, unless I can overcome this means that GUI-based tools are not going to work, and as for remote GUI access, see above security bureaucracy. Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Pieter Temmerman wrote: Hi Steve, Some great tools to monitor your Tomcat installations are: - Jconsole (make sure you get the topthreads plugin) - VisualVM - VisualGC - Jmap Hope that helps. On Tue, 2009-02-03 at 09:31 -0600, Steve Cohen wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version "1.5.0_16" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. Can someone suggest a path forward here? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Hi Steve, Some great tools to monitor your Tomcat installations are: - Jconsole (make sure you get the topthreads plugin) - VisualVM - VisualGC - Jmap Hope that helps. On Tue, 2009-02-03 at 09:31 -0600, Steve Cohen wrote: > We have an application that runs under Tomcat under RHEL 5.0 and is > launched by a jsvc daemon. > It chugs along seemingly fine on several servers, yet yesterday crashed > on one of them with the above exception seemingly without experiencing > any kind of abnormal load. I am trying to get a handle on this. > System is > > $ uname -a > Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 > i386 GNU/Linux > > Java version: > java -version > java version "1.5.0_16" > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) > Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) > > no special JVM memory settings are used when launching tomcat, just the > default memory allocations. I have read in some places that this > particular form of OutOfMemoryError is actually made worse by increasing > the memory size. > > I need to deploy some thread and memory monitoring setup to get a handle > on this. > > Can someone suggest a path forward here? > > Thanks. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Pieter Temmerman email: ptemmerman@sadiel.es skype: ptemmerman.sadiel SADIEL TECNOLOGÍAS DE LA INFORMACIÓN, S.A. http://www.sadiel.es. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
