Re: Tomcat dies suddenly (was JVM goes away)
David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
2010/1/14 David kerber dcker...@verizon.net: Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. I suggested memtest86+ (http://www.memtest.org/). Memtest86 (http://www.memtest86.com/) is also available; I moved to the + version when Chris Brady stopped development of the original for a period. The core tests are very similar, doing things like looking for stuck bits (always 1 or always 0) or bits whose state can be influenced by their neighbours'. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
David, I am such a dufuss... didn't even notice it cycled after it finished a test. After almost 24 hours, showing no failures. Time to call Dell. Thanks, Carl - Original Message - From: David kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 14, 2010 8:48 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
On 14/01/2010 14:36, Carl wrote: David, I am such a dufuss... didn't even notice it cycled after it finished a test. After almost 24 hours, showing no failures. Time to call Dell. If there's no memory hardware issue, then we're back to software. You were on linux right? Did you search the OS logs for evidence of an OOM kill? cat /var/log/messages | grep --ignore-case killed process p - Original Message - From: David kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 14, 2010 8:48 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Yes, Slackware, version 13, 64bit. I had done this manually (looked through each log for any evidence of a failure) but had not done it your automated way. Just did your automated way and it found nothing (I included all the messages logs)... bummer. The server I brought up Tuesday is using the same Slackware, Tomcat, JDK. This server is a Dell T105 (it was destined to be used in a smaller setting) which has an AMD processor instead of the Xeon. This server is a little slower (the users don't notice) and has yet to have any problem. Of course, the T110 ran for a week before it had a problem. Thanks, Carl - Original Message - From: Pid p...@pidster.com To: users@tomcat.apache.org Sent: Thursday, January 14, 2010 9:55 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) On 14/01/2010 14:36, Carl wrote: David, I am such a dufuss... didn't even notice it cycled after it finished a test. After almost 24 hours, showing no failures. Time to call Dell. If there's no memory hardware issue, then we're back to software. You were on linux right? Did you search the OS logs for evidence of an OOM kill? cat /var/log/messages | grep --ignore-case killed process p - Original Message - From: David kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 14, 2010 8:48 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat dies suddenly (was JVM goes away)
From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl
Re: Tomcat dies suddenly (was JVM goes away)
Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. A T110 doesn't use error-correcting memory, as I recall, so a dodgy bit could cause problems. Give it a couple of hours with memtest86+ and you'll at least know whether you've been chasing phantoms due to a hardware error. (I'm perhaps biased - I've had memory errors on three low-end servers now) - Peter 2010/1/13 Carl c...@etrak-plus.com: From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
In order to monitor java memory at chrash time you can add to JAVA_OPTS these directives -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your/tomcat/folder/memorydump.hprof In this way, if tomcat goes in out of memory, you have an image of memory (memorydump.hprof) that you can analyze by an external application like MemoryAnalyzer [ http://www.eclipse.org/mat/ ]. 2010/1/13 Carl c...@etrak-plus.com From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl
Re: Tomcat dies suddenly (was JVM goes away)
In process... thanks for the suggestion. Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 8:49 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. A T110 doesn't use error-correcting memory, as I recall, so a dodgy bit could cause problems. Give it a couple of hours with memtest86+ and you'll at least know whether you've been chasing phantoms due to a hardware error. (I'm perhaps biased - I've had memory errors on three low-end servers now) - Peter 2010/1/13 Carl c...@etrak-plus.com: From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Done. Thanks for the suggestion. Plan to place this machine back on the firing line after running the memory test suggested by Peter. Thanks, Carl - Original Message - From: Paolo Santarsiero paolo.santarsi...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 8:58 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) In order to monitor java memory at chrash time you can add to JAVA_OPTS these directives -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your/tomcat/folder/memorydump.hprof In this way, if tomcat goes in out of memory, you have an image of memory (memorydump.hprof) that you can analyze by an external application like MemoryAnalyzer [ http://www.eclipse.org/mat/ ]. 2010/1/13 Carl c...@etrak-plus.com From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peter, On 1/13/2010 8:49 AM, Peter Crowther wrote: Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. Usually, I would agree that physical memory problems are likely to be a problem, but every time I've had a physical memory problem (much more common than I'd like to admit!), the JVM has crashed in a more classic way: that is, with an hs_log file and almost always with a SIGSEGV, rather than this phantom thing described by Carl. The Linux OOM killer might be a suspect, except that the process is apparently not dying, which is very strange. Carl: when the JVM dies and you use top to see free memory, does it say that 2.4GB of memory is in use by a particular process, or does it just appear that the memory is not available? If it's by a particular process, which one? The JVM process (/usr/bin/java or whatever) either does or does not exist, and if it does not exist, is it retaining memory? If the Tomcat connectors have shut down (thereby releasing the TCP/IP ports), but not the java process, then there should be some indication in catalina.out that the connectors have been shut down explicitly. The whole thing sounds weird. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktN+ecACgkQ9CaO5/Lv0PAZzgCgsZaU16RGcs5pgsgzgLVX7q0W 8xcAnRUb1Zl+0PY6+Umk8nQAEagfl/Su =RA9e -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
2010/1/13 Christopher Schultz ch...@christopherschultz.net: On 1/13/2010 8:49 AM, Peter Crowther wrote: Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. Usually, I would agree that physical memory problems are likely to be a problem, but every time I've had a physical memory problem (much more common than I'd like to admit!), the JVM has crashed in a more classic way: that is, with an hs_log file and almost always with a SIGSEGV, rather than this phantom thing described by Carl. The Linux OOM killer might be a suspect, except that the process is apparently not dying, which is very strange. [...] The whole thing sounds weird. :( Oh, I agree entirely - usually something will turn a reference bad and you'll get a memory access somewhere off in hyperspace during a GC. But it's an easy thing to check, and there is an (admittedly small) possibility of seeing these symptoms. Heck, with hardware errors there's a small probability of seeing pretty much *any* symptoms. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Chris, Carl: when the JVM dies and you use top to see free memory, does it say that 2.4GB of memory is in use by a particular process, It shows the 2.4GB as 'Used' but does not show it attached to any process (remember that the Tomcat process has disappeared... ps aux | grep tomcat yields nothing.) My observation is that the server has 500MB 'used' when it starts and moves to 2.4GB after Tomcat is started. However, the server does not appear to reclaim the memory after the process dies as the 'used' stays right at 2.4GB. Visual LVM continues to report that the now dead Tomcat instance is still holding onto the memory but I am not certain whether this reflects some variable(s) set in Visual JVM or the actual memory something is still holding onto. or does it just appear that the memory is not available? The 2.4GB is just shown as 'Used' by top. If it's by a particular process, which one? No process but I expected that as the Tomcat process (ps aux | grep tomcat) no longer exists (after the 'crash'.) The JVM process (/usr/bin/java or whatever) either does or does not exist, and if it does not exist, is it retaining memory? I don't know how I could tell if the Tomcat java process/JVM was holding onto the memory if the process no longer exists. If the Tomcat connectors have shut down (thereby releasing the TCP/IP ports), but not the java process, then there should be some indication in catalina.out No indication at all... just comes to a stop. (I had a problem a while ago with not properly releasing database connections an I still have a good deal of stuff going to catalina.out (because I have been too busy to comment out the debugging messages.) that the connectors have been shut down explicitly. The whole thing sounds weird. :( That has been a good deal of my frustration... I thought it would leave some tracks somewhere. All thoughts and ideas are appreciated. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 11:50 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peter, On 1/13/2010 8:49 AM, Peter Crowther wrote: Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. Usually, I would agree that physical memory problems are likely to be a problem, but every time I've had a physical memory problem (much more common than I'd like to admit!), the JVM has crashed in a more classic way: that is, with an hs_log file and almost always with a SIGSEGV, rather than this phantom thing described by Carl. The Linux OOM killer might be a suspect, except that the process is apparently not dying, which is very strange. Carl: when the JVM dies and you use top to see free memory, does it say that 2.4GB of memory is in use by a particular process, or does it just appear that the memory is not available? If it's by a particular process, which one? The JVM process (/usr/bin/java or whatever) either does or does not exist, and if it does not exist, is it retaining memory? If the Tomcat connectors have shut down (thereby releasing the TCP/IP ports), but not the java process, then there should be some indication in catalina.out that the connectors have been shut down explicitly. The whole thing sounds weird. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktN+ecACgkQ9CaO5/Lv0PAZzgCgsZaU16RGcs5pgsgzgLVX7q0W 8xcAnRUb1Zl+0PY6+Umk8nQAEagfl/Su =RA9e -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Peter, The memTest is still running but clean so far. Thanks, Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 12:00 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) 2010/1/13 Christopher Schultz ch...@christopherschultz.net: On 1/13/2010 8:49 AM, Peter Crowther wrote: Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the bad server. Usually, I would agree that physical memory problems are likely to be a problem, but every time I've had a physical memory problem (much more common than I'd like to admit!), the JVM has crashed in a more classic way: that is, with an hs_log file and almost always with a SIGSEGV, rather than this phantom thing described by Carl. The Linux OOM killer might be a suspect, except that the process is apparently not dying, which is very strange. [...] The whole thing sounds weird. :( Oh, I agree entirely - usually something will turn a reference bad and you'll get a memory access somewhere off in hyperspace during a GC. But it's an easy thing to check, and there is an (admittedly small) possibility of seeing these symptoms. Heck, with hardware errors there's a small probability of seeing pretty much *any* symptoms. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Carl wrote: Peter, The memTest is still running but clean so far. Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
David, Will do... thanks for the heads up. Carl - Original Message - From: David kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, January 13, 2010 1:17 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Carl wrote: Peter, The memTest is still running but clean so far. Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Peter Crowther wrote: 2010/1/13 David kerber dcker...@verizon.net: Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
