Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Bill Davidson wrote: I really would like to be able to run these kind of tools across a firewall. I can't believe there are no provisions for it. Seems there is now. Haven't tried it yet, but will soon, if I can upgrade that Tomcat. Previous messages in this thread give more clues. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: I have been put right about that by someone else already. Yeah, I'm about 2 days behind on all the drivel on the list. ;) Not everyone agrees with that solution by the way, because the JConsole may interfere with the JVM on the same system (specially on mine, seen it has less RAM than Chuck's portable phone). Well, the solution /will/ work, but may not be a good one. If you thrash your production server, it's not a really good solution, now, is it? Assuming Chuck has an iPhone, which has 128MB of RAM, I'm not sure I'd want to run a JVM on that server in the first place. Something tells me we've been over this before so I won't beat a dead horse. There's an interesting other solution here : http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html Seems one /can/ now fix that second port, which would allow it setting 2 tunnels beforehand, if I understand this correctly. Cool. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmMTdYACgkQ9CaO5/Lv0PCFjgCdEor1GVBKDPgxbt2oihsnMclm sJoAnRJ7zO+cq+a8+skQyYQfi24FkVEP =ZBek -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
On 03.02.2009 16:31, Steve Cohen wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. I wouldn't put so much work into making X11 work. You need to find out your memory situation and your thread situation. The error says: when trying to start a new thread, native memory allocation failed and thus the thread could not be created. So the reason is, that you ran out of native memory in the process. The cure is either to reduce the amount of memory needed or to reduce the number of threads needed. Concerning Memory: Basically your process uses Java memory (Heap and Permanent generation) and native memory (thread stacks and other native memory). If you are using a 32Bit system or VM, the address space is 4GB per process, but that space is divided in some part available for the process, and some part used by the kernel for data associated with the process. The part actually available for the process depends as far as i know on kernel settings and can be either 1, 2, or 3 GB (I would expect 2GB). Heap, Perm and native memory has to fit into this. I'm not quite sure about the thread stacks, but I assume those must fit into the 1/2/3 GB too. Process memory can be inspected using the proc file system. The file /proc/MYPID/maps (MYPID is the process id of your Tomcat process) contains a table of memory areas used by the process. The hex numbers at the beginning of each line shows starting and ending addresses, the differences are the sizes of the areas. Check whether you can verify, that the process is close to one of those magic limits. Then check, how big your heap and your permanent generation is. You can do this with jstat. I would sugest you find a good sizing for your app and don't proceed with automatic 8and thus unknown) sizing. If total memory is close to the limit and heap+perm is close to the total memory, you know that your java memory is to big for 32bits. If total memory is close to the limit and heap+perm is much lower, then there's something which consumes large native memory. It could be threads (see next) or some embedded native component, e.g. some library included via JNI. Threads: Use kill -QUIT MYPID with MYPID the process id of your Tomcat. It will write a thread dump (not: memory dump) to catalina.out. This dump contains all Java threads of your process and the stacks they are in. See how many these are and to which components they belong. With a thread dump you can find out, whether you have extremely many Java threads. You need to be the owner of the Tomcat process or root to be able to send a signal to the process. You need to be able to read catalina.out to analyse the output. Use the -L flag of ps to look at all threads of the Tomcat process. Check whether there are many more threads in ps, than there are in the Java thread dump. If so, you have some native component inside, that creates additional threads. In this case use gstack, to check, what those are doing (to get an idea, what they are used for). If you can verify that you run our of native memory, but you can reduce neither memory size not thread count, you can start playing around with the thread stack size using -Xss. But this will be tedious. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: [...] Assuming Chuck has an iPhone, which has 128MB of RAM, I'm not sure I'd want to run a JVM on that server in the first place. Something tells me we've been over this before so I won't beat a dead horse. Chuck assured us, on this forum, that his phone had waaay more memory than that. So either it's not an iPhone, or it's a hacked one. Probably has a JVM on it too. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Peter Crowther wrote: From: André Warnier [mailto:a...@ice-sa.com] Christopher Schultz wrote: What about forwarding X through the tunnel instead? You can't, because it is variable. It is the result of some internal negotiation between Jconsole and the remote JVM. Apparently, anyway. I haven't managed to make it work so far. To expand Chris' suggestion: What about forwarding the X11 protocol through the tunnel instead, such that you can run Jconsole on the same machine as the JVM? X11 has a fixed port number, and ssh has support for this. I've used this trick when needing to do something graphical on a box with ssh-only access; works a treat. Sorry to be so obtuse. X is not something I use regularly, so when Chris mentioned X, I thought he was talking about the secondary connection/port that the JVM/Jconsole agree on, not about X-terminal and so. So thanks for the tip, but could you expand even more ? I realise this isn't really a Tomcat matter, but maybe this can help someone else in a similar situation. At which level would X11 kick in ? Layout is as follows : Office A : workstation (me) -- firewall/router - internet modem -- internet Office B : internet - internet modem - firewall/router - Tomcat host There is no VPN setup between A and B. My workstation is a Windows laptop. I don't presently have an X emulator on it. It has Java JDK 6, an SSH terminal-like program (putty), WinSCP, etc.. I'd rather avoid installing Cygwin on the laptop, because it interferes with other Unix-like things I have on it. The Tomcat host has a fixed internal IP address. I could also fix the internal IP address of my workstation. I have full control of the routers/firewalls/hosts. The firewall/router at B automatically tunnels ports 22,80,8080 and some others to the Tomcat host. I can add more tunnels. I can also build on-demand tunnels from outside to any inside host/port via the firewall and SSH. Doing something in the other direction is more tricky , because both Office A and B have variable internet IP addresses which change once per 24h (both with dynamic DNS though). I'm usually at A. B is my home office, and when my laptop is there, I have no problem running JConsole on it and connecting to the Tomcat host, because both are then on the same LAN. But I haven't managed that yet from A. ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
http://solaris.reys.net/english/2006/04/x11_forwarding Works pretty much the same on any other Linux Rgds Gregor -- just because your paranoid, doesn't mean they're not after you... gpgp-fp: 79A84FA526807026795E4209D3B3FE028B3170B2 gpgp-key available @ http://pgpkeys.pca.dfn.de:11371 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
From: André Warnier [mailto:a...@ice-sa.com] X is not something I use regularly, so when Chris mentioned X, I thought he was talking about the secondary connection/port that the JVM/Jconsole agree on, not about X-terminal and so. So thanks for the tip, but could you expand even more ? I realise this isn't really a Tomcat matter, but maybe this can help someone else in a similar situation. At which level would X11 kick in ? It ships the stream of commands to display JConsole on your local desktop. Rough sequence of events: - Start an X11 server on your workstation. Find out its DISPLAY variable. - If using PuTTY, tell it about your X11 DISPLAY variable. It can't guess! - Use ssh/PuTTY with X11 tunnelling enabled to remote to your Tomcat host. - echo $DISPLAY to make sure it's come across OK. - Start Jconsole on the Tomcat host. It will display on your local desktop, via the ssh tunnel. The network part of this is no more complex than you already have. Layout is as follows : Office A : workstation (me) -- firewall/router - internet modem -- internet Office B : internet - internet modem - firewall/router - Tomcat host There is no VPN setup between A and B. My workstation is a Windows laptop. I don't presently have an X emulator on it. It has Java JDK 6, an SSH terminal-like program (putty), WinSCP, etc.. I'd rather avoid installing Cygwin on the laptop, because it interferes with other Unix-like things I have on it. OK. If you don't want to put Cygwin on - which is certainly the easiest+cheapest way of getting an X11 display on your laptop - then you still have options. You could, for example, install something like VMware (Server is free, which would be plenty for this job). You could then run, say, Ubuntu on the VM and ssh across from Ubuntu. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
On Thu, Feb 5, 2009 at 11:11 AM, Peter Crowther peter.crowt...@melandra.com wrote: OK. If you don't want to put Cygwin on http://sourceforge.net/projects/xming Just a simple install, very small, convinient and works like charm. Rgds Gregor -- just because your paranoid, doesn't mean they're not after you... gpgp-fp: 79A84FA526807026795E4209D3B3FE028B3170B2 gpgp-key available @ http://pgpkeys.pca.dfn.de:11371 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Hi. Thanks Chris and Peter and Gregor. Gregor sent me a link to an article that looks promising, and says basically the same as Peter below, using XMing as the X11 server (client?, I can never remember..). I will try that tonight. Peter Crowther wrote: From: André Warnier [mailto:a...@ice-sa.com] X is not something I use regularly, so when Chris mentioned X, I thought he was talking about the secondary connection/port that the JVM/Jconsole agree on, not about X-terminal and so. So thanks for the tip, but could you expand even more ? I realise this isn't really a Tomcat matter, but maybe this can help someone else in a similar situation. At which level would X11 kick in ? It ships the stream of commands to display JConsole on your local desktop. Rough sequence of events: - Start an X11 server on your workstation. Find out its DISPLAY variable. - If using PuTTY, tell it about your X11 DISPLAY variable. It can't guess! - Use ssh/PuTTY with X11 tunnelling enabled to remote to your Tomcat host. - echo $DISPLAY to make sure it's come across OK. - Start Jconsole on the Tomcat host. It will display on your local desktop, via the ssh tunnel. The network part of this is no more complex than you already have. Layout is as follows : Office A : workstation (me) -- firewall/router - internet modem -- internet Office B : internet - internet modem - firewall/router - Tomcat host There is no VPN setup between A and B. My workstation is a Windows laptop. I don't presently have an X emulator on it. It has Java JDK 6, an SSH terminal-like program (putty), WinSCP, etc.. I'd rather avoid installing Cygwin on the laptop, because it interferes with other Unix-like things I have on it. OK. If you don't want to put Cygwin on - which is certainly the easiest+cheapest way of getting an X11 display on your laptop - then you still have options. You could, for example, install something like VMware (Server is free, which would be plenty for this job). You could then run, say, Ubuntu on the VM and ssh across from Ubuntu. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
From: André Warnier [mailto:a...@ice-sa.com] using XMing as the X11 server (client?, I can never remember..). Yeah, X's terminology is very counter-intuitive - I get comments of you're kidding every time I teach it. An X server serves out the keyboard, mouse and display. X11 clients connect to the X11 display server in order to display their output and get their input. All of which means that you often use an X server running on your workstation or thin client to connect to X client programs running on your application/web/... server. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
From: Gregor Schneider [mailto:rc4...@googlemail.com] http://sourceforge.net/projects/xming Just a simple install, very small, convinient and works like charm. Thanks Gregor, I wasn't aware of that one. Think I may just have a new preferred X server :-). - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Peter Crowther wrote: From: André Warnier [mailto:a...@ice-sa.com] using XMing as the X11 server (client?, I can never remember..). Yeah, X's terminology is very counter-intuitive - I get comments of you're kidding every time I teach it. An X server serves out the keyboard, mouse and display. X11 clients connect to the X11 display server in order to display their output and get their input. All of which means that you often use an X server running on your workstation or thin client to connect to X client programs running on your application/web/... server. You might also want to look at org.apache.catalina.mbeans.JMXAdaptorLifecycleListener in http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html Something similar also exists for 6.0.x Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Mark Thomas wrote: Peter Crowther wrote: From: André Warnier [mailto:a...@ice-sa.com] using XMing as the X11 server (client?, I can never remember..). Yeah, X's terminology is very counter-intuitive - I get comments of you're kidding every time I teach it. An X server serves out the keyboard, mouse and display. X11 clients connect to the X11 display server in order to display their output and get their input. All of which means that you often use an X server running on your workstation or thin client to connect to X client programs running on your application/web/... server. You might also want to look at org.apache.catalina.mbeans.JMXAdaptorLifecycleListener in http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html Something similar also exists for 6.0.x Thanks Mark. That page is a bit obscure for someone like me not making Tomcat and Java their main focus, but do I understand it correctly in that it would mean that the variable second port connection would no longer be variable, thus allowing JConsole to connect to it through 2 pre-created SSH tunnels (as long as one inserts the indicated Listener element in the server.xml of a Tomcat 5.5.25+ ) ? (admittedly, some people might consider this question a bit obscure too) As a secondary question, does the fact of adding such a Listener have any discernable impact on Tomcat's performance (when JConsole is not in use) ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
The disadvantage of running these tools on the server that also runs Tomcat is that these tools generate certain load, which under some circumstances is not wanted as it might influence the stats. On Thu, 2009-02-05 at 14:15 +0100, André Warnier wrote: Mark Thomas wrote: Peter Crowther wrote: From: André Warnier [mailto:a...@ice-sa.com] using XMing as the X11 server (client?, I can never remember..). Yeah, X's terminology is very counter-intuitive - I get comments of you're kidding every time I teach it. An X server serves out the keyboard, mouse and display. X11 clients connect to the X11 display server in order to display their output and get their input. All of which means that you often use an X server running on your workstation or thin client to connect to X client programs running on your application/web/... server. You might also want to look at org.apache.catalina.mbeans.JMXAdaptorLifecycleListener in http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html Something similar also exists for 6.0.x Thanks Mark. That page is a bit obscure for someone like me not making Tomcat and Java their main focus, but do I understand it correctly in that it would mean that the variable second port connection would no longer be variable, thus allowing JConsole to connect to it through 2 pre-created SSH tunnels (as long as one inserts the indicated Listener element in the server.xml of a Tomcat 5.5.25+ ) ? (admittedly, some people might consider this question a bit obscure too) As a secondary question, does the fact of adding such a Listener have any discernable impact on Tomcat's performance (when JConsole is not in use) ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Pieter Temmerman email: ptemmerman@sadiel.es skype: ptemmerman.sadiel SADIEL TECNOLOGÍAS DE LA INFORMACIÓN, S.A. http://www.sadiel.es. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: Christopher Schultz wrote: What about forwarding X through the tunnel instead? You can't, because it is variable. If you forward the X connection, then Jconsole is running locally on the Tomcat server. So there should be no problem with tunneling because you are only tunneling the X connection, not the JMX ones. Maybe I'm missing something. Yes. What you missed is that I understood your first X for being the mysterious second unpredictable connection port which JConsole/JVM agree on. Well, X sounded like that, didn't it ? :-) I have been put right about that by someone else already. Not everyone agrees with that solution by the way, because the JConsole may interfere with the JVM on the same system (specially on mine, seen it has less RAM than Chuck's portable phone). I'm giving the X-way a try though, using XMing (Gregor's suggestion). There's an interesting other solution here : http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html Seems one /can/ now fix that second port, which would allow it setting 2 tunnels beforehand, if I understand this correctly. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: What about forwarding X through the tunnel instead? I've tried that and found it to be unusably slow. Just using Cygwin/X though. I'm not sure about those fancy light weight X compression systems. I really would like to be able to run these kind of tools across a firewall. I can't believe there are no provisions for it. It has to be something that a lot of people want to do. If I can run Oracle connections over an ssh tunnel with good performance, it seems like I should be able to do something like jconsole data too. RMI's random port thing is really annoying. It seems like there ought to be some way to specify all ports involved. Whomever designed it apparently didn't give any thought to working across firewalls. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: Steve Cohen wrote: [...] Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Not really a help here, but as I have discovered JConsole wont't even work through a simple SSH tunnel, because it wants to dynamically open a second parallel connection, which of course your tunnel knows nothing about. What about forwarding X through the tunnel instead? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmJ1s4ACgkQ9CaO5/Lv0PAIKACgtZdXVlSsic2fpO5sEBE2RBqB 8tAAnjRZcJFC0r9FMrefKI062OgXic24 =FB4k -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Steve, Steve Cohen wrote: One additional restriction I have is a security bureaucracy from hell. You can use jmap from the command-line. That will give you a thread dump which can help you see what all your threads are doing. If you are using a system that uses POSIX threads, it might also help to see what the output of 'ulimit -u' which will tell you the process limit for your user. POSIX threads are essentially processes, so threads count against your process limit. The obvious answer is that you simply have too many threads being created. There are a number of reasons you could be running out of threads. Here are some: 1. Too many request processor threads configured in server.xml (see your Connector element). Inspect your configuration. You may have more connectors configured than necessary (I've seen configurations where both HTTP and AJP connectors are both enabled but only one is being used). 2. Too much file I/O. If you are using background threads to read/write files independently of the request processor thread, you might be creating too many of them. I recommend using a thread pool to schedule background jobs. 3. Timer threads. If you are using java.util.Timer, you may be creating more threads than you realize. Check your use of java.util.Timer and possibly use a thread pool for timed tasks. 4. Quartz, Quartz, Quartz (this is a job scheduler that generates a lot of questions on this list). Check any uses of Quartz. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmJ2YQACgkQ9CaO5/Lv0PBH1gCgq+lrPmgTjRhEF1JEf9jBs4oe Vq4AoJ18SigCd8pkSCh4eR0qDOawyhpu =tIYg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: Steve Cohen wrote: [...] Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Not really a help here, but as I have discovered JConsole wont't even work through a simple SSH tunnel, because it wants to dynamically open a second parallel connection, which of course your tunnel knows nothing about. What about forwarding X through the tunnel instead? You can't, because it is variable. It is the result of some internal negotiation between Jconsole and the remote JVM. Apparently, anyway. I haven't managed to make it work so far. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
From: André Warnier [mailto:a...@ice-sa.com] Christopher Schultz wrote: What about forwarding X through the tunnel instead? You can't, because it is variable. It is the result of some internal negotiation between Jconsole and the remote JVM. Apparently, anyway. I haven't managed to make it work so far. To expand Chris' suggestion: What about forwarding the X11 protocol through the tunnel instead, such that you can run Jconsole on the same machine as the JVM? X11 has a fixed port number, and ssh has support for this. I've used this trick when needing to do something graphical on a box with ssh-only access; works a treat. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. Can someone suggest a path forward here? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Hi Steve, Some great tools to monitor your Tomcat installations are: - Jconsole (make sure you get the topthreads plugin) - VisualVM - VisualGC - Jmap Hope that helps. On Tue, 2009-02-03 at 09:31 -0600, Steve Cohen wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. Can someone suggest a path forward here? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Pieter Temmerman email: ptemmerman@sadiel.es skype: ptemmerman.sadiel SADIEL TECNOLOGÍAS DE LA INFORMACIÓN, S.A. http://www.sadiel.es. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
One additional restriction I have is a security bureaucracy from hell. As far as console access I have a two-hop connection. As far as GUI access I have some crappy device to log into that lets me have really bad keyboard/mouse control of the machine, which, unless I can overcome this means that GUI-based tools are not going to work, and as for remote GUI access, see above security bureaucracy. Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Pieter Temmerman wrote: Hi Steve, Some great tools to monitor your Tomcat installations are: - Jconsole (make sure you get the topthreads plugin) - VisualVM - VisualGC - Jmap Hope that helps. On Tue, 2009-02-03 at 09:31 -0600, Steve Cohen wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. Can someone suggest a path forward here? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Hi Steve, Also, a good tool for monitoring Tomcat is Lamda Probe http://www.lambdaprobe.org/d/index.htm Regards, Ovidiu On Tue, Feb 3, 2009 at 9:31 AM, Steve Cohen sco...@javactivity.org wrote: We have an application that runs under Tomcat under RHEL 5.0 and is launched by a jsvc daemon. It chugs along seemingly fine on several servers, yet yesterday crashed on one of them with the above exception seemingly without experiencing any kind of abnormal load. I am trying to get a handle on this. System is $ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686 i386 GNU/Linux Java version: java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing) no special JVM memory settings are used when launching tomcat, just the default memory allocations. I have read in some places that this particular form of OutOfMemoryError is actually made worse by increasing the memory size. I need to deploy some thread and memory monitoring setup to get a handle on this. Can someone suggest a path forward here? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- ovidiu asiminei
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Dear Steve, One additional restriction I have is a security bureaucracy from hell. As far as console access I have a two-hop connection. Is there no test system that you can use to reproduce this, under less insane restrictions? Have you considered hauling your behind to the data center to sit behind the console for a few hours? In the past I found that I could spend days fighting the bizarre rules with half- working technical solutions, or just go to the actual machine and look at the display there. Of course, it was only a two hour drive in my case. Yours might well be at the other end of the world. As far as GUI access I have some crappy device to log into that lets me have really bad keyboard/mouse control of the machine, which, unless I can overcome this means that GUI-based tools are not going to work, and as for remote GUI access, see above security bureaucracy. Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? There are some JMX command line tools, but JMX is really easy to use from code. You could write a simple JSP page to give you the figures that way. Script with curl/wget and graph with Excel. :) -- Kees Jan http://java-monitor.com/forum/ kjkos...@kjkoster.org 06-51838192 The secret of success lies in the stability of the goal. -- Benjamin Disraeli - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Steve Cohen wrote: [...] Which means I'm looking for command-line equivalents for what JConsole does, particularly the threads tab. Any ideas along that path? Not really a help here, but as I have discovered JConsole wont't even work through a simple SSH tunnel, because it wants to dynamically open a second parallel connection, which of course your tunnel knows nothing about. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
