Re: Sad: Tomcat 5.5.x crashes almost every single day.
Remy Maucherat wrote: On 2/28/06, Peter Lin [EMAIL PROTECTED] wrote: honestly, besides Weblogic, most servlet and ejb containers do not provide simple and clear instructions for tracing issues. With websphere, you have to buy an expensive license of WASD and even then debugging an issue won't be better in my experience. Debugging an webapp is a difficult task and very time consuming. I sympathize with you, but the only real way to trace is to use a profiler like optimizeIt, jprofiler or yourkit. An alternative would be to run tomcat with Sun's JFluid VM which is experimental. How does BEA do that ? JRockit ? JFluid could be a way (when you're not profiling, the overhead is limited), but on production servers it's still not doable as enabling profiling would kill it. Of course, memory profiling could be low impact, and would be great already. On production, you can always use JDK jhat/jmap available in 1.5 and Mustang. This can gives you a snapshot of the heap/memory and other usefull things without paying the price of a profiler -- Jeanfrancois -- x Rémy Maucherat Developer Consultant JBoss Europe x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Peter Lin [EMAIL PROTECTED] wrote: another potential solution is to try a different JVM like Bea's JRockit. JRockit provides some built in profiling capabilities, so that is another way to get some profile data quickly. I've read about jrockit 5.0 SP1. They say it has a new feature that can help track memory leaks in just few days, not few weeks :D To find a memory leak in cocoon webaplication would probably take few years for person like me, anyway :)) -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Tomasz Nowak [EMAIL PROTECTED] wrote: Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos Any help on above would be _really_ appreciated! -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
my advice is dive in and try to debug it, or hire someone with experience debugging webapps. there's no much others can do for you at this point, since it's debugging. good luck. peter On 3/1/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Tomasz Nowak [EMAIL PROTECTED] wrote: Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos Any help on above would be _really_ appreciated! -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
This could be completely unrelated, but let me throw something at you, maybe it will help btw. I prefer to run mod_proxy over mod_jk over http, much simpler, and in *our* load tests has scaled much better during stress. Response already committed This can happen if your servlet/jsp has code like this do some stuff response.sendRedirect(url); ...do some more stuff what has happened here, is that you are trying to write to the response buffer even though you earlier indiciated (through sendRedirect or sendError) that you wouldn't. To correct this, you would have to put a return statement in your code do some stuff response.sendRedirect(url); return; ...do some more stuff java.net.SocketException: Broken pipe the apache mod_jk module has disconnected from tomcat before it was able to send the entire message. Filip Tomasz Nowak wrote: Tomasz Nowak [EMAIL PROTECTED] wrote: Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos Any help on above would be _really_ appreciated! - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Peter Lin [EMAIL PROTECTED] wrote: my advice is dive in and try to debug it, or hire someone with experience debugging webapps. there's no much others can do for you at this point, since it's debugging. good luck. I've just meant configuring the logging system: So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos Any help on above would be _really_ appreciated! Tomcat 5.5 logging docs are very brief about it.. Simple duplication of example entries does not do the job. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Filip Hanik - Dev Lists [EMAIL PROTECTED] wrote: This could be completely unrelated, but let me throw something at you, maybe it will help btw. I prefer to run mod_proxy over mod_jk over http, much simpler, and in *our* load tests has scaled much better during stress. Response already committed This can happen if your servlet/jsp has code like this do some stuff response.sendRedirect(url); ...do some more stuff what has happened here, is that you are trying to write to the response buffer even though you earlier indiciated (through sendRedirect or sendError) that you wouldn't. To correct this, you would have to put a return statement in your code do some stuff response.sendRedirect(url); return; ...do some more stuff I have little controle about it as redirect-url is on of built-in features of Apache Cocoon. I've tried to put sth like: org.apache.jk.core.MsgContext.level = SEVERE into the logging.properties but that unfortunately changes nothing -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
honestly, I really doubt logging will produce anything useful beyond the VM dump you already posted. My first hand experience, the logs won't help at this point. What needs to happen is to step through the code and see exactly what is happening. if you don't have access to a profiler, the other option is to add a bunch of system.out.println to the pages to see what is happening. If the webapp is making connection to a database or anything external, I would double check that code. Chances are, the code is leaking badly and causing the instability. peter On 3/1/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Peter Lin [EMAIL PROTECTED] wrote: my advice is dive in and try to debug it, or hire someone with experience debugging webapps. there's no much others can do for you at this point, since it's debugging. good luck. I've just meant configuring the logging system: So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos Any help on above would be _really_ appreciated! Tomcat 5.5 logging docs are very brief about it.. Simple duplication of example entries does not do the job. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Sad: Tomcat 5.5.x crashes almost every single day.
From: news [mailto:[EMAIL PROTECTED] On Behalf Of Tomasz Nowak Probably important: === [lots of technical details about kit and container config] Very important == Your application. Have you profiled it for memory leaks that would cause a OOME? Have you tested it for deadlocks that would cause hangs? Has it worked correctly for an extended period of time under another servlet container? The pain: crashing Tomcat == Everything works right until Tomcat suddenly crashes, what happens almost every single day. Sometimes it crashes with OOME / out of heap (heh, is 1GB not enought!?) Depends on your application. Cocoon can be quite heavyweight. Any ideas before abandoning Java technology? ;) Profile your application. What's eating memory? Get a thread dump when your application hangs. Is anything deadlocked? What? Where? - Peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
enableLookups=true redirectPort=8443 / Why enableLookups=true ??? Also, Cocoon can be very resource-intensive depending on which blocks you've built. JSP under Cocoon 2.1.7 and JDK 1.5 is also problematic, see for example: http://issues.apache.org/jira/browse/COCOON-1457 I have some ideas as to how to approach this, but I've not had the time to investigate. /mde/ __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Mark Eggers [EMAIL PROTECTED] wrote: enableLookups=true redirectPort=8443 / Why enableLookups=true ??? I use request.getRemoteHost in couple of places. Also, Cocoon can be very resource-intensive depending on which blocks you've built. Almost always only: - databases-block - lucene-block - xsp-block - mail-block - profiler-block Nothing more. JSP under Cocoon 2.1.7 and JDK 1.5 is also problematic, see for example: http://issues.apache.org/jira/browse/COCOON-1457 I have some ideas as to how to approach this, but I've not had the time to investigate. I don't use JSP. But on my development box JSP works fine. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Peter Crowther [EMAIL PROTECTED] wrote: Have you tested it for deadlocks that would cause hangs? [...] Profile your application. What's eating memory? Get a thread dump when your application hangs. Is anything deadlocked? What? Where? How? http://www.google.com/search?hl=plq=%22thread+dump%22+site%3Atomcat.apache.org http://www.google.com/search?hl=plq=profile+application+site%3Atomcat.apache.org -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
I'm confused - but wouldn't an enableLookups on the AJP connector return the hostname of the remote Apache server sending the request (and not the remote web client)? I don't know since I'm at work, and I always disable enableLookups since it's a potential performance hit. /mde/ --- Tomasz Nowak [EMAIL PROTECTED] wrote: Mark Eggers [EMAIL PROTECTED] wrote: enableLookups=true redirectPort=8443 / Why enableLookups=true ??? I use request.getRemoteHost in couple of places. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
[...skiped...] I've been crond-restarting Tomcat 4.1 on Java 1.4 every night for more than a year, and now I have to restart Tomcat 5.5.15 on Java 1.5 AND apache httpd also almost every single day. From my -user point of view, Tomcat doesn't change in a good direction. In fact in it is probably changing in wrong direction - at least the 4.1.x logger element has been working 'out of the box'. If you have to restart tomcat every night, your application seems to be very buggy... Come on! Am I the only one that has such problems with Tomcat? We had some serious problems with tomcat, which are solved now, however, we are on 5.0.25 (with session patch) and not on 5.5.x yet. However, when tomcat dies, what your cpu usage is looking like? 100%? Leon - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
On 2/28/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Peter Crowther [EMAIL PROTECTED] wrote: Have you tested it for deadlocks that would cause hangs? [...] Profile your application. What's eating memory? Get a thread dump when your application hangs. Is anything deadlocked? What? Where? How? http://www.google.com/search?hl=plq=%22thread+dump%22+site%3Atomcat.apache.org http://www.google.com/search?hl=plq=profile+application+site%3Atomcat.apache.org -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] kill -SIGQUIT (i think its kill -11) your tomcat-java process will cause the vm to dump out complete stack traces to the stdout (catalina.out) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Mark Eggers wrote: I'm confused - but wouldn't an enableLookups on the AJP connector return the hostname of the remote Apache server sending the request (and not the remote web client)? No. JkOptions +ForwardLocalAddress Regards, Mladen. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Tim Lucia [EMAIL PROTECTED] wrote: Send Tomcat a QUIT (11) signal on Unix, or control/break (run it interactively) on Windows. This will cause the VM to dump all thread stacks. If you don't like Tomcat, you can always shell out some big bugs for a commercial application server. For some even bigger bucks you can get commercial support. I can tell you it's unlikely you'll get a warm reception from the very people who can help you given the current tone of your postings. Your right, the tone of my postings is inproper. I've been using 'free' software for almost 10 years now and I pretty well get the rules. My only excuse is the level of my frustration, based on recent Tomcat use. For now, the only contribition to Tomcat community I can give is _pointing_out_ some real-world problems, that typical users of Tomcat may face (and face!). The problems are: 1. Poor/none default logging facility in 5.5.x. - no real help/tips on error sources - no examples how to do a decent virtual hosts logging - no tips how to switch off a lot of uneccesary trash log inputs If Tomcat is supposed to be production ready why it has no production ready logging features? 2. No real-world, step-by-step docs how to TRACE and eliminate application errors that lead to server failure. That is probably a problem lot of Tomcat users must struggle with. 3. During last years I see no actions taken by Tomcat dev team to eliminate Tomcat server failures caused by webaplications. Is it really impossible? Real world user feedback was the main point of my postings. And sorry for the tone again. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Sad: Tomcat 5.5.x crashes almost every single day.
#1 - I have a reasonable command of commons logging, log4j, juli, java logging, etc., and I can quite easily make the application logging work without a hitch. As for 'production ready logging', I would look to valves for things like access logging. The access log valve works virtually the same as Apache httpd, or at least the output is nearly so. It didn't take me all that long to learn this stuff. I will admit I haven't used Tomcat virtual hosts, so that might be an area of concern. #2 - Do any commercial application server vendors offer you assistance on how to find bugs in application code? You're much more likely to find these answers using google, or reading wikis or from community support. There are hundreds of such discussions out there. My favorite (personally) is how to avoid running out of permgen space. #3 really boils down to The application developer screwed up so badly that the entire virtual machine under which Tomcat is running is toast. That's not Tomcat's fault, really, it's more Java's. Again, any commercially available server (written in Java/sharing the VM) will likely suffer the same fate. You can always feel free to contribute documentation improvements! Tim -Original Message- From: news [mailto:[EMAIL PROTECTED] On Behalf Of Tomasz Nowak Sent: Tuesday, February 28, 2006 4:36 PM To: users@tomcat.apache.org Subject: Re: Sad: Tomcat 5.5.x crashes almost every single day. Tim Lucia [EMAIL PROTECTED] wrote: Send Tomcat a QUIT (11) signal on Unix, or control/break (run it interactively) on Windows. This will cause the VM to dump all thread stacks. If you don't like Tomcat, you can always shell out some big bugs for a commercial application server. For some even bigger bucks you can get commercial support. I can tell you it's unlikely you'll get a warm reception from the very people who can help you given the current tone of your postings. Your right, the tone of my postings is inproper. I've been using 'free' software for almost 10 years now and I pretty well get the rules. My only excuse is the level of my frustration, based on recent Tomcat use. For now, the only contribition to Tomcat community I can give is _pointing_out_ some real-world problems, that typical users of Tomcat may face (and face!). The problems are: 1. Poor/none default logging facility in 5.5.x. - no real help/tips on error sources - no examples how to do a decent virtual hosts logging - no tips how to switch off a lot of uneccesary trash log inputs If Tomcat is supposed to be production ready why it has no production ready logging features? 2. No real-world, step-by-step docs how to TRACE and eliminate application errors that lead to server failure. That is probably a problem lot of Tomcat users must struggle with. 3. During last years I see no actions taken by Tomcat dev team to eliminate Tomcat server failures caused by webaplications. Is it really impossible? Real world user feedback was the main point of my postings. And sorry for the tone again. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
On 2/28/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Your right, the tone of my postings is inproper. I've been using 'free' software for almost 10 years now and I pretty well get the rules. My only excuse is the level of my frustration, based on recent Tomcat use. For now, the only contribition to Tomcat community I can give is _pointing_out_ some real-world problems, that typical users of Tomcat may face (and face!). The problems are: The claims of regressions over 4.1 are completely bogus. 1. Poor/none default logging facility in 5.5.x. - no real help/tips on error sources - no examples how to do a decent virtual hosts logging - no tips how to switch off a lot of uneccesary trash log inputs If Tomcat is supposed to be production ready why it has no production ready logging features? In case you haven't noticed, it is extremely hard to do, because webapps have their own logging mechanism most of the time. You mention the logger element of 4.x, but it didn't actually do anything (it did put the internal logging for the specified container, as well as the logging done using the ServletContext - aka, the ugliest and most useless logging facility ever). Tomcat 5.5.15 and j.u.l use hierarchical categories for the containers so that you can easily replicate the logging that was done by the logger element. The default logging.properties does that for localhost. I do not consider logging.properties to be poor or bad in any way. You can also update to another, more full featured logger based on j.u.l, such as http://www.x4juli.org/. If you disagree with all this, you can use another technology as you were planning. 2. No real-world, step-by-step docs how to TRACE and eliminate application errors that lead to server failure. That is probably a problem lot of Tomcat users must struggle with. Right, you want an integrated profiler, ready to enable, and with no performance cost. I'd like to have it too. 3. During last years I see no actions taken by Tomcat dev team to eliminate Tomcat server failures caused by webaplications. Is it really impossible? It depends, on some OSes, it's apparently impossible to get a threaded server to run properly indeed. You're using Linux 2.4.something. Try updating to 2.6. If you are using Redhat or another Redhat based distros, always use LD_ASSUME_KERNEL. -- x Rémy Maucherat Developer Consultant JBoss Europe x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
On 2/28/06, Peter Lin [EMAIL PROTECTED] wrote: honestly, besides Weblogic, most servlet and ejb containers do not provide simple and clear instructions for tracing issues. With websphere, you have to buy an expensive license of WASD and even then debugging an issue won't be better in my experience. Debugging an webapp is a difficult task and very time consuming. I sympathize with you, but the only real way to trace is to use a profiler like optimizeIt, jprofiler or yourkit. An alternative would be to run tomcat with Sun's JFluid VM which is experimental. How does BEA do that ? JRockit ? JFluid could be a way (when you're not profiling, the overhead is limited), but on production servers it's still not doable as enabling profiling would kill it. Of course, memory profiling could be low impact, and would be great already. -- x Rémy Maucherat Developer Consultant JBoss Europe x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Sad: Tomcat 5.5.x crashes almost every single day.
I have found that running Jconsole against the VM (JMX) and watching the various memory pools can be quite informative. I have not evaluated the performance impact in a production environment (don't run Jconsole on the same production box, as per the instructions.) I have had good luck with Jprofiler as well. -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 28, 2006 5:47 PM To: Tomcat Users List Subject: Re: Sad: Tomcat 5.5.x crashes almost every single day. On 2/28/06, Peter Lin [EMAIL PROTECTED] wrote: honestly, besides Weblogic, most servlet and ejb containers do not provide simple and clear instructions for tracing issues. With websphere, you have to buy an expensive license of WASD and even then debugging an issue won't be better in my experience. Debugging an webapp is a difficult task and very time consuming. I sympathize with you, but the only real way to trace is to use a profiler like optimizeIt, jprofiler or yourkit. An alternative would be to run tomcat with Sun's JFluid VM which is experimental. How does BEA do that ? JRockit ? JFluid could be a way (when you're not profiling, the overhead is limited), but on production servers it's still not doable as enabling profiling would kill it. Of course, memory profiling could be low impact, and would be great already. -- x Rémy Maucherat Developer Consultant JBoss Europe x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Tomcat Users: We also had the dreaded, 'out of memory' errors and periodic crashes several times every day. Currently, we are stable. With help from Chuck, we were able to narrow down our problem to that of insufficient permanent generation memory. The point being that figuring out JVM memory requires a bit of homework, testing and careful evaluation. And after a lot of reading, I am only beginning to understand how all this stuff works. Patience and persistence helps. Stephen Caine CommonGround Softworks, Inc. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Hi Tomasz, you mentioned below that you have a MaxPermSize of 256m. I would suggest increasing this value to a much larger amount and even increasing your -Xms and -Xmx values. We had a problem with our application in a live environment where it would regularly get errors like the following; Exception java.lang.OutOfMemoryError: requested SIZE_FORMAT bytes for compute_new_size. Out of swap space? Possible causes: - not enough swap space left, or - kernel parameter MAXDSIZ is very small. ( 0) 0xc5bf0474 vm_exit_out_of_memory__FUlPCcb + 0x47c [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 1) 0xc5e89d78 compute_new_size__9PSPermGenFUl + 0x150 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 2) 0xc5e86024 invoke_no_policy__11PSMarkSweepSFPbb + 0x244 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 3) 0xc5e8b814 invoke__10PSScavengeSFPb + 0x154 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 4) 0xc5e51f4c failed_mem_allocate__20ParallelScavengeHeapFPbUlbT3 + 0x64 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 5) 0xc5f6a78c doit__29VM_ParallelGCFailedAllocationFv + 0x54 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 6) 0xc5f6a280 evaluate__12VM_OperationFv + 0xb0 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 7) 0xc5f691e0 evaluate_operation__8VMThreadFP12VM_Operation + 0x48 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 8) 0xc5f698a0 loop__8VMThreadFv + 0x558 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 9) 0xc5f68f24 run__8VMThreadFv + 0xd4 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] (10) 0xc5e39b20 _start__FPv + 0x148 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] (11) 0xc004b168 __pthread_body + 0x44 [/usr/lib/libpthread.1] (12) 0xc00549ec __pthread_start + 0x14 [/usr/lib/libpthread.1] In the end it was the fact that the PermGen space was all used up and JVM was trying to resize it. The PermGen may have become full because of the number of class files and JSP class files that were loaded into memory or other resources that are kept in PermGen by the JVM. I would say with 8 applications running under Tomcat you would need to make as much of that 3Gb available to the JVM to use. My 2 cents worth. Cheers. Tomasz Nowak wrote: Probably important: === - 2x Xeon, 3 GB mem - Linux 2.4.26 - Java 1.5.0_06-b05 - Apache Tomcat 5.5.15 - CATALINA_OPTS=\ -server \ -Djava.awt.headless=true \ -Dfile.encoding=UTF-8 \ -XX:MaxPermSize=256m \ -Xms1024m -Xmx1024m - Connector port=8009 protocol=AJP/1.3 maxThreads=200 minSpareThreads=25 maxSpareThreads=50 backlog=20 connectionTimeout=1 enableLookups=true redirectPort=8443 / - mod_jk 1.2.15 - Apache Httpd 1.3.34 - 8 Tomcat vhosts - 8 webapps - each webapp runs on separate vhost - each webapp is Apache Cocoon 2.1.7 - system load avarage: usually under 0.5 - http load: not more then 20-30 concurrent req The pain: crashing Tomcat == Everything works right until Tomcat suddenly crashes, what happens almost every single day. Sometimes it crashes with OOME / out of heap (heh, is 1GB not enought!?), but somtimes NOTHING wrong is logged into the logs - it just stops responding to requests (hangs). When Tomcat stops responding to requests JK eats all of 256 Apache HTTPd MaxClients so other httpd vhosts stop responding also. Disaster! Any ideas before abandoning Java technology? ;) Default Tomcat logging facility seems to be useless to me. j.u.l. logging.properties semantics looks like a programmers joke made to users (!). And even with swallowOutput it logs everything into catalina.out (?) I found none decent log4j-tomcat-virtual-hosts-logging-manual in Tomcat docs also. Is Tomcat a production ready reference j2se server or just a finish it yourself proof of concept toy? I've been crond-restarting Tomcat 4.1 on Java 1.4 every night for more than a year, and now I have to restart Tomcat 5.5.15 on Java 1.5 AND apache httpd also almost every single day. From my -user point of view, Tomcat doesn't change in a good direction. In fact in it is probably changing in wrong direction - at least the 4.1.x logger element has been working 'out of the box'. Come on! Am I the only one that has such problems with Tomcat? -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Another tool I find very useful is visualgc, which is part of the jvmstat 3.0 package available here http://java.sun.com/performance/jvmstat. You dont have to set anything up and its really lightweight. You could probably run it in a production environment with hardly any impact on performance, although I have not done that myself. You just run jps which will give you the PID's for any java executables currently running. Then run visualgc PID and it will pop up three windows showing information about heap usage, garbage collection, class file compilation, and so on. Cheers. Tim Lucia wrote: I have found that running Jconsole against the VM (JMX) and watching the various memory pools can be quite informative. I have not evaluated the performance impact in a production environment (don't run Jconsole on the same production box, as per the instructions.) I have had good luck with Jprofiler as well. -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 28, 2006 5:47 PM To: Tomcat Users List Subject: Re: Sad: Tomcat 5.5.x crashes almost every single day. On 2/28/06, Peter Lin [EMAIL PROTECTED] wrote: honestly, besides Weblogic, most servlet and ejb containers do not provide simple and clear instructions for tracing issues. With websphere, you have to buy an expensive license of WASD and even then debugging an issue won't be better in my experience. Debugging an webapp is a difficult task and very time consuming. I sympathize with you, but the only real way to trace is to use a profiler like optimizeIt, jprofiler or yourkit. An alternative would be to run tomcat with Sun's JFluid VM which is experimental. How does BEA do that ? JRockit ? JFluid could be a way (when you're not profiling, the overhead is limited), but on production servers it's still not doable as enabling profiling would kill it. Of course, memory profiling could be low impact, and would be great already. -- x Rémy Maucherat Developer Consultant JBoss Europe x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Remy Maucherat [EMAIL PROTECTED] wrote: In case you haven't noticed, it is extremely hard to do, because webapps have their own logging mechanism most of the time. You mention the logger element of 4.x, but it didn't actually do anything (it did put the internal logging for the specified container, as well as the logging done using the ServletContext - aka, the ugliest and most useless logging facility ever). Tomcat 5.5.15 and j.u.l use hierarchical categories for the containers so that you can easily replicate the logging that was done by the logger element. The default logging.properties does that for localhost. I do not consider logging.properties to be poor or bad in any way. Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos PS. Tim Lucia: I don't care about access logs, HTTPd logis it for me. Right, you want an integrated profiler, ready to enable, and with no performance cost. I'd like to have it too. OK, so what is the easiest to implement and easiest use to profiler that I can use with Tomcat? It depends, on some OSes, it's apparently impossible to get a threaded server to run properly indeed. You're using Linux 2.4.something. Try updating to 2.6. If you are using Redhat or another Redhat based distros, always use LD_ASSUME_KERNEL. Is Tomcat on 2.6 so called thread save? I've come over several warnings about Apache httpd 2.0 in 'threaded' mode on Linux kernels. People (also PHP dev's) seem to recommend sticking with 1.3. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Nathan Smith [EMAIL PROTECTED] wrote: Hi Tomasz, you mentioned below that you have a MaxPermSize of 256m. I have not seen OOME: PermGen errors on that machine but after having some of them on other machine with aditonal 10 webapps deployed I've increased it from default 64m to 256m to be absolutly sure it wont hang on PermGen error. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Leon Rosenberg [EMAIL PROTECTED] wrote: We had some serious problems with tomcat, which are solved now, however, we are on 5.0.25 (with session patch) and not on 5.5.x yet. However, when tomcat dies, what your cpu usage is looking like? 100%? I've noticed that when tomcat hangs - load avarage increases from 0.2-0.4 (normal) to about 1.0. But I'll take a close look at the cpu usage next time 5.5.x hangs. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Tim Lucia [EMAIL PROTECTED] wrote: #2 - Do any commercial application server vendors offer you assistance on how to find bugs in application code? You're much more likely to find these answers using google, or reading wikis or from community support. There are hundreds of such discussions out there. My favorite (personally) is how to avoid running out of permgen space. That problems I've already overcome :] But I honestly admit I don't have experience with Bea/Websphere etc, so I didn't know they hang/freeze/crash in the same way Tomcat does. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
you should seriously consider upgrading to 2.6.x linux. 2.4.s thread support simply sucks! If your tomcat spawns more than allowed threads it will get an outofmemoryerror and simply die... And I don't mean your 200 tomcat httpworker threads only, but all threads in the jvm, including database connection pools and so on. You should be able to count currently running threads on a 2.4.x system with ps, I think the proper switch was -H. Something like ps auxH | wc -l. You can calculate your systems thread limitation by this, very simple test program, which just creates and starts new Threads until an OOME is thrown. regards Leon On 3/1/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Leon Rosenberg [EMAIL PROTECTED] wrote: We had some serious problems with tomcat, which are solved now, however, we are on 5.0.25 (with session patch) and not on 5.5.x yet. However, when tomcat dies, what your cpu usage is looking like? 100%? I've noticed that when tomcat hangs - load avarage increases from 0.2-0.4 (normal) to about 1.0. But I'll take a close look at the cpu usage next time 5.5.x hangs. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Sad: Tomcat 5.5.x crashes almost every single day.
I use JProfiler with Tomcat all the time. -Original Message- From: news [mailto:[EMAIL PROTECTED] On Behalf Of Tomasz Nowak Sent: Tuesday, February 28, 2006 6:37 PM To: users@tomcat.apache.org Subject: Re: Sad: Tomcat 5.5.x crashes almost every single day. Remy Maucherat [EMAIL PROTECTED] wrote: In case you haven't noticed, it is extremely hard to do, because webapps have their own logging mechanism most of the time. You mention the logger element of 4.x, but it didn't actually do anything (it did put the internal logging for the specified container, as well as the logging done using the ServletContext - aka, the ugliest and most useless logging facility ever). Tomcat 5.5.15 and j.u.l use hierarchical categories for the containers so that you can easily replicate the logging that was done by the logger element. The default logging.properties does that for localhost. I do not consider logging.properties to be poor or bad in any way. Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos PS. Tim Lucia: I don't care about access logs, HTTPd logis it for me. Right, you want an integrated profiler, ready to enable, and with no performance cost. I'd like to have it too. OK, so what is the easiest to implement and easiest use to profiler that I can use with Tomcat? It depends, on some OSes, it's apparently impossible to get a threaded server to run properly indeed. You're using Linux 2.4.something. Try updating to 2.6. If you are using Redhat or another Redhat based distros, always use LD_ASSUME_KERNEL. Is Tomcat on 2.6 so called thread save? I've come over several warnings about Apache httpd 2.0 in 'threaded' mode on Linux kernels. People (also PHP dev's) seem to recommend sticking with 1.3. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
another potential solution is to try a different JVM like Bea's JRockit. JRockit provides some built in profiling capabilities, so that is another way to get some profile data quickly. peter On 2/28/06, Tim Lucia [EMAIL PROTECTED] wrote: I use JProfiler with Tomcat all the time. -Original Message- From: news [mailto:[EMAIL PROTECTED] On Behalf Of Tomasz Nowak Sent: Tuesday, February 28, 2006 6:37 PM To: users@tomcat.apache.org Subject: Re: Sad: Tomcat 5.5.x crashes almost every single day. Remy Maucherat [EMAIL PROTECTED] wrote: In case you haven't noticed, it is extremely hard to do, because webapps have their own logging mechanism most of the time. You mention the logger element of 4.x, but it didn't actually do anything (it did put the internal logging for the specified container, as well as the logging done using the ServletContext - aka, the ugliest and most useless logging facility ever). Tomcat 5.5.15 and j.u.l use hierarchical categories for the containers so that you can easily replicate the logging that was done by the logger element. The default logging.properties does that for localhost. I do not consider logging.properties to be poor or bad in any way. Hm. Here is my logging.properties: http://www.biochip.pl/logging.properties.txt The way new vhosts are added is pretty stupid for me but lets forget about the semantics now. I've added swallowOutput attibute to serveral context. How should it suppose to work? On my box it works in this way: - The only file that actually logs anything that matters is catalina.out - Most of catalina.out inputs also appear in localhost.timstamp.log - Nothing more then manager actions are logged to vhost logs - After tomcat has stared, the only input logs are: a) useless: INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action INFO: Response already committed Feb 28, 2006 10:59:22 PM org.apache.jk.core.MsgContext action (repeted serveral times every single minute (!) on my machine) b) sth like: Feb 28, 2006 11:00:03 PM org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) So how can I: - configure it to work similary to tomcat 4.1, I mean logs from different vhost go to different files - configure Tomcat not to log anything that belongs to vhost to catalina.out - turn of these useless jk infos PS. Tim Lucia: I don't care about access logs, HTTPd logis it for me. Right, you want an integrated profiler, ready to enable, and with no performance cost. I'd like to have it too. OK, so what is the easiest to implement and easiest use to profiler that I can use with Tomcat? It depends, on some OSes, it's apparently impossible to get a threaded server to run properly indeed. You're using Linux 2.4.something. Try updating to 2.6. If you are using Redhat or another Redhat based distros, always use LD_ASSUME_KERNEL. Is Tomcat on 2.6 so called thread save? I've come over several warnings about Apache httpd 2.0 in 'threaded' mode on Linux kernels. People (also PHP dev's) seem to recommend sticking with 1.3. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
On 2/28/06, Tomasz Nowak [EMAIL PROTECTED] wrote: I've not noticed more then 400-500 java threads, but I'll monitor that. I've set different ulimits form differnt users, BUT user that runs java/tomcat has no limits but one: -s 2048 (stacksize). AFAIK that one was recommended by RELEASE-NOTES.txt -- T. that seems odd. in the original message the connector settings was this Connector port=8009 protocol=AJP/1.3 maxThreads=200 minSpareThreads=25 maxSpareThreads=50 backlog=20 connectionTimeout=1 enableLookups=true redirectPort=8443 / if there's more than 400 threads, that would suggest the webapp is creating threads and not handling them correctly. If the guess is correct, a simple way to prove it is to deploy the app in Resin and use JMeter to generate 1-4 million requests. Resin should also crash, if the webapp is creating threads and letting them zombie. ultimately, the webapp needs to be profiled to make sure it behaves correctly. peter
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Don't count on it!! In a message dated 2/28/2006 7:26:33 P.M. Eastern Standard Time, [EMAIL PROTECTED] writes: it seems that using a rude tone is a good way to get many answers, your question got 26 answers, mine, with a similar problem, only 1 (and a not helpful one).
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Leon Rosenberg [EMAIL PROTECTED] wrote: kill -SIGQUIT (i think its kill -11) your tomcat-java process Ok, it has happened again ! Simply: Mar 1, 2006 2:28:34 AM org.apache.catalina.connector.CoyoteAdapter service SEVERE: An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space So I've kill -11 java process and there is an entry in catalina.out: # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x00138246, pid=17443, tid=16384 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode) # Problematic frame: # C [libpthread.so.0+0xc246] accept+0x46 # # An error report file with more information is saved as /tmp/hs_err_pid17443.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # That /tmp/hs_err_pid17443.log is here: http://www.biochip.pl/hs_err_pid17443.log.txt Got it! But does it help? -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Peter Lin [EMAIL PROTECTED] wrote: look at the dump, it looks like the permgen ran out of space PSPermGen total 50304K, used 50170K [0x445f, 0x4771, 0x545f) object space 50304K, 99% used [0x445f,0x476ee878,0x4771) try increasing your permGen to 128Mb. Another option is to use a jdk that doesn't have permGen like JRockit. Sun's PermGen is not resized once the JVM starts, so if the webapp has lots of classes, it can exceed the PermGen size. As you see at the bottom of the file I have already 256m PermSpace. Another issue is that if Tomcat 5.5.x runs out of PermSpace, it logs: java.lang.OutOfMemoryError: PermGen space in contrary to: java.lang.OutOfMemoryError: Java heap space And heap space OOME is the one that has been logged. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
On 2/28/06, Tomasz Nowak [EMAIL PROTECTED] wrote: Peter Lin [EMAIL PROTECTED] wrote: look at the dump, it looks like the permgen ran out of space PSPermGen total 50304K, used 50170K [0x445f, 0x4771, 0x545f) object space 50304K, 99% used [0x445f,0x476ee878,0x4771) try increasing your permGen to 128Mb. Another option is to use a jdk that doesn't have permGen like JRockit. Sun's PermGen is not resized once the JVM starts, so if the webapp has lots of classes, it can exceed the PermGen size. As you see at the bottom of the file I have already 256m PermSpace. Another issue is that if Tomcat 5.5.x runs out of PermSpace, it logs: java.lang.OutOfMemoryError: PermGen space in contrary to: java.lang.OutOfMemoryError: Java heap space And heap space OOME is the one that has been logged. -- T. It's not at all clear why -XX:MaxPermSize=256m isn't working and the perm size is still at the default. If the perm gen was set correctly, wouldn't the dump show something other than 99%? I agree the message is misleading, so something odd is definitely happening. only way to find out at this point is to profile the app. peter
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Hi Tomasz, this is the exact point I was making in my previous email. The error message that was returned was to do with the Java Heap Space. It may be referring to the entire heap that is allocated to the JVM, who knows. The problem can be fixed by increasing your -XX:MaxPermSize value to more than 256. In my view that really isn't enough especially if you have 8 applications running within Tomcat. It it is possible to get some sort of stack trace from the native code, you will see that this OOM error occurs while trying to resize the PermGen as below. See the line (1). ( 0) 0xc5bf0474 vm_exit_out_of_memory__FUlPCcb + 0x47c [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 1) 0xc5e89d78 compute_new_size__9PSPermGenFUl + 0x150 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 2) 0xc5e86024 invoke_no_policy__11PSMarkSweepSFPbb + 0x244 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 3) 0xc5e8b814 invoke__10PSScavengeSFPb + 0x154 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 4) 0xc5e51f4c failed_mem_allocate__20ParallelScavengeHeapFPbUlbT3 + 0x64 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 5) 0xc5f6a78c doit__29VM_ParallelGCFailedAllocationFv + 0x54 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 6) 0xc5f6a280 evaluate__12VM_OperationFv + 0xb0 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 7) 0xc5f691e0 evaluate_operation__8VMThreadFP12VM_Operation + 0x48 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 8) 0xc5f698a0 loop__8VMThreadFv + 0x558 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] ( 9) 0xc5f68f24 run__8VMThreadFv + 0xd4 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] (10) 0xc5e39b20 _start__FPv + 0x148 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl] (11) 0xc004b168 __pthread_body + 0x44 [/usr/lib/libpthread.1] (12) 0xc00549ec __pthread_start + 0x14 [/usr/lib/libpthread.1] Download and use a program called visualgc available here http://java.sun.com/performance/jvmstat. You can have it running during the day and monitor what the garbage collector is doing and how the Eden, Old and Perm Gen heap spaces are growing. If Tomcat is not throwing an OOM error then the Garbage Collector could be trying to do Full GC's causing Tomcat to hang and become unresponsive. There is a flow on affect here, which makes it look as though the problem is caused by something else. Cheers. Tomasz Nowak wrote: Peter Lin [EMAIL PROTECTED] wrote: look at the dump, it looks like the permgen ran out of space PSPermGen total 50304K, used 50170K [0x445f, 0x4771, 0x545f) object space 50304K, 99% used [0x445f,0x476ee878,0x4771) try increasing your permGen to 128Mb. Another option is to use a jdk that doesn't have permGen like JRockit. Sun's PermGen is not resized once the JVM starts, so if the webapp has lots of classes, it can exceed the PermGen size. As you see at the bottom of the file I have already 256m PermSpace. Another issue is that if Tomcat 5.5.x runs out of PermSpace, it logs: java.lang.OutOfMemoryError: PermGen space in contrary to: java.lang.OutOfMemoryError: Java heap space And heap space OOME is the one that has been logged. -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
--- Tomasz Nowak [EMAIL PROTECTED] wrote: Peter Lin [EMAIL PROTECTED] wrote: look at the dump, it looks like the permgen ran out of space PSPermGen total 50304K, used 50170K [0x445f, 0x4771, 0x545f) object space 50304K, 99% used [0x445f,0x476ee878,0x4771) try increasing your permGen to 128Mb. Another option is to use a jdk that doesn't have permGen like JRockit. Sun's PermGen is not resized once the JVM starts, so if the webapp has lots of classes, it can exceed the PermGen size. As you see at the bottom of the file I have already 256m PermSpace. Another issue is that if Tomcat 5.5.x runs out of PermSpace, it logs: java.lang.OutOfMemoryError: PermGen space in contrary to: java.lang.OutOfMemoryError: Java heap space And heap space OOME is the one that has been logged. -- T. Have you set limits on how large your heap can get? If you have limited your new size or changed the ratios any of that can have this affect as well. Sometimes changing the ratio to not have enough new space and then creating too many objects too fast can make your application eventually throw OOMEs because it can't move the objects into older spaces quick enough depending on what you application is doing. I found this happens with large applications (such as IDEs like Netbeans, but if you web app is large enough and depending on what you are doingsame thing could happen). How much memory does your system have compared to how much memory you have allocated to your heap? Asking because of the way the VM will be able to allocate memory. The total heap size switches are separate from the MaxPermSize, so if your heap size is too large and then you have bumped up your PermSize then you could have this issue when your PermSize resizes and your heap tries to grow as well. MaxPermSize doesn't set the PermSize to that size, but is merely a limit, so it will grow as you add to your permanent objects (classes and statics). Wade - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Peter Lin [EMAIL PROTECTED] wrote: It's not at all clear why -XX:MaxPermSize=256m isn't working and the perm size is still at the default. If the perm gen was set correctly, wouldn't the dump show something other than 99%? You're right, it shows PSPermGen total 50304K not the 256m I've set up. I agree the message is misleading, so something odd is definitely happening. only way to find out at this point is to profile the app. The only apps I run are Apache Cocoon 2.1.7. Profiling such webapp threatens me.. I'm not sure if my java knowledge is sufficient to handle that. -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Nathan Smith [EMAIL PROTECTED] wrote: The problem can be fixed by increasing your -XX:MaxPermSize value to more than 256. I've increased it to 512m now. I'll see. Download and use a program called visualgc available here http://java.sun.com/performance/jvmstat. You can have it running during the day and monitor what the garbage collector is doing and how the Eden, Old and Perm Gen heap spaces are growing. If Tomcat is not throwing an OOM error then the Garbage Collector could be trying to do Full GC's causing Tomcat to hang and become unresponsive. There is a flow on affect here, which makes it look as though the problem is caused by something else. Looks like no way I can run it on headless box. Setting the deamon up fails because it can't write somewhere. # /usr/lib/java5/bin/jstatd Could not create remote object access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write) java.security.AccessControlException: access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write) at java.security.AccessControlContext.checkPermission(AccessControlContext. java:264) at java.security.AccessController.checkPermission(AccessController.java:427 ) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.System.setProperty(System.java:698) at sun.tools.jstatd.Jstatd.main(Jstatd.java:122) Thanks to Java I still don't know where.. : Anyway: a) I've added -Xloggc:/home/tomcat/logs/gc.log I logs sth such inputs: 1489.487: [GC 219875K-151505K(1018496K), 0.0292590 secs] 1546.088: [GC 225745K-152608K(1031744K), 0.0268240 secs] 1591.348: [GC 241696K-152348K(1023296K), 0.0198200 secs] 1635.863: [GC 241052K-156605K(1030656K), 0.0218510 secs] 1653.314: [GC 244925K-155047K(1022272K), 0.0205590 secs] 1727.156: [GC 243047K-157627K(1029632K), 0.0215210 secs] 1730.679: [GC 245307K-163738K(1030016K), 0.0263340 secs] 1740.791: [GC 251098K-165299K(1030016K), 0.0266920 secs] 1750.912: [GC 252275K-162599K(1022592K), 0.0220800 secs] 1790.949: [GC 249255K-163285K(1028672K), 0.0179890 secs] b) These values do not mean ANYTHING to me but percheps they will be readable by someone from the list: # /usr/lib/java5/bin/jstat -gc 28923 S0CS1CS0US1U EC EUOC OU PC PUYGC YGCTFGCFGCT GCT 9984.0 12352.0 9975.6 0.0 83008.0 21940.2 932096.0 168376.4 48640.0 48550.7 822.150 0 0.0002.150 # /usr/lib/java5/bin/jstat -gcutil 28923 S0 S1 E O P YGC YGCTFGCFGCT GCT 99.92 0.00 58.27 18.06 99.82 822.150 00.000 2.150 # /usr/lib/java5/bin/jstat -gccapacity 28923 NGCMNNGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMNPGCMX PGC PC YGCFGC 116480.0 116480.0 106368.0 11840.0 4736.0 82688.0 932096.0 932096.0 932096.0 932096.0 16384.0 524288.0 48640.0 48640.0 83 0 # /usr/lib/java5/bin/jstat -gcpermcapacity 28923 PGCMN PGCMX PGC PC YGC FGCFGCT GCT 16384.0 524288.048640.048640.083 00.000 2.172 28923 is the PID of jvm I'm focused on -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Sad: Tomcat 5.5.x crashes almost every single day.
Wade Chandler [EMAIL PROTECTED] wrote: Sometimes changing the ratio to not have enough new space and then creating too many objects too fast can make your application eventually throw OOMEs because it can't move the objects into older spaces quick enough depending on what you application is doing. Gosh, it looks like don't move your mouse coursor to quickly or your operations system will fail :D How much memory does your system have compared to how much memory you have allocated to your heap? Asking because of the way the VM will be able to allocate memory. The total heap size switches are separate from the MaxPermSize, so if your heap size is too large and then you have bumped up your PermSize then you could have this issue when your PermSize resizes and your heap tries to grow as well. MaxPermSize doesn't set the PermSize to that size, but is merely a limit, so it will grow as you add to your permanent objects (classes and statics). phisical: 3gb settings: -XX:MaxPermSize=512m -Xms1024m -Xmx1024m (was 256m for Perm, 512m now) -- T. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
