Tomcat 4.1.30 crashing on production site - need help
Hi All, For the past couple of months we have been facing a peculiar problem on our production environment. And we have been unable to resolve this issue so far. We have an application hosted on Tomcat 4.1.30 and using JDK1.4.2_06. We also have an Apache WS - (think it is v2.0.2). Everyday approx after every 24 hours of uptime the website slows down and hangs... On checking the tomcat logs, we see an OutOfMemoryException. Restarting Tomcat was one way to momentarily fix the issue and have the site up - but due to Apache forked processes we are not able to restart Tomcat too. Even if could our Tomcat takes about 30-40 mins to load/startup due to some user data loading on startup. So that leaves the only option of killing certain threads/processes and then starting Tomcat. which works fine - only we have been doing this every night for the past more than a month. We also got a thread dump just to check what is going on.. and I have pasted that info herewith. Sun has suggested that we install all patches to the JDK. I just wanted to find out if anyone had faced a similar issue and if there was something they did that fixed the issue. Was Tomcat the culprit?? We will go ahead and install the patches very soon - but wanted to be ready in case that didnt fix the issue - did anyone experience an improvement once the patches were installed?? * Here is the Exception Trace: An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0xFF380CBC Function=memcpy+0x7E0 Library=/usr/platform/sun4u/lib/libc_psr.so.1 Current Java thread: at COM.ibm.db2.jdbc.app.DB2PreparedStatement.SQLBindChar(Native Method) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.execute2(DB2PreparedStatement.java:1020) - locked 0x9ec56ef8 (a COM.ibm.db2.jdbc.app.DB2Connection) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.executeUpdate(DB2PreparedStatement.java:807) at com.waveset.repository.RelationalDataStore$Item.setPersistentObject(RelationalDataStore.java:999) at com.waveset.repository.AbstractDataStore.setItems(AbstractDataStore.java:4926) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1401) at com.waveset.repository.ServerRepository.checkin(ServerRepository.java:2276) at com.waveset.session.InternalSession.checkinObject(InternalSession.java:581) at com.waveset.task.Scheduler.storeExecutingTask(Scheduler.java:1515) at com.waveset.task.Scheduler.executeReadyTask(Scheduler.java:2918) at com.waveset.task.Scheduler.processReadyTasks(Scheduler.java:2837) at com.waveset.task.Scheduler.processTasks(Scheduler.java:1031) at com.waveset.task.Scheduler.run(Scheduler.java:866) Dynamic libraries: 0x1 /www4/j2sdk1_4_2_06/java/usr/j2se/bin/java 0xff36 /usr/lib/libthread.so.1 0xff39a000 /usr/lib/libdl.so.1 0xff28 /usr/lib/libc.so.1 0xff38 /usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1 0xfec0 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/client/libjvm.so 0xff24 /usr/lib/libCrun.so.1 0xff21 /usr/lib/libsocket.so.1 0xff10 /usr/lib/libnsl.so.1 0xff1e /usr/lib/libm.so.1 0xff1c /usr/lib/libsched.so.1 0xff27 /usr/lib/libw.so.1 0xff0d /usr/lib/libmp.so.2 0xff0b /usr/lib/librt.so.1 0xff09 /usr/lib/libaio.so.1 0xff06 /usr/lib/libmd5.so.1 0xff04 /usr/platform/SUNW,Ultra-80/lib/libmd5_psr.so.1 0xfebd /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/native_threads/libhpi.so 0xfeb8 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libverify.so 0xfeb4 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libjava.so 0xfeb2 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libzip.so 0x737d /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libnet.so 0x7491 /opt/IBMdb2/V7.1/java12/libdb2jdbc.so 0x71c0 /opt/IBMdb2/V7.1/lib/libdb2.so.1 0x73ea /usr/lib/libresolv.so.2 Heap at VM Abort: Heap def new generation total 226112K, used 67456K [0x7580, 0x83b8, 0x83b8) eden space 219264K, 27% used [0x7580, 0x79331618, 0x82e2) from space 6848K, 100% used [0x82e2, 0x834d, 0x834d) to space 6848K, 0% used [0x834d, 0x834d, 0x83b8) tenured generation total 1864192K, used 1152200K [0x83b8, 0xf580, 0xf580) the space 1864192K, 61% used [0x83b8, 0xca0b2380, 0xca0b2400, 0xf580) compacting
Re: Tomcat 4.1.30 crashing on production site - need help
I've seen similar problems in a non-Tomcat Java application. The root cause was a memory leak. Although Java has good garbage collection, it relies on object references being released. This is generally fine, but some buggy applications can add references to large object to the application and forget to release them. This causes memory to be consumed, the app so slow and eventually the app or server to lock. With Tomcat, I can imagine this happening with objects continually being added to, say, an ArrayList in the Application context and not being released. I believe there are various tools around to check object allocation and release. So if you see objects of a certain type have a gradually increasing number, you may well have a memory leak. Hope this helps. Regards, James. = Hi All, For the past couple of months we have been facing a peculiar problem on our production environment. And we have been unable to resolve this issue so far. We have an application hosted on Tomcat 4.1.30 and using JDK1.4.2_06. We also have an Apache WS - (think it is v2.0.2). Everyday approx after every 24 hours of uptime the website slows down and hangs... On checking the tomcat logs, we see an OutOfMemoryException. Restarting Tomcat was one way to momentarily fix the issue and have the site up - but due to Apache forked processes we are not able to restart Tomcat too. Even if could our Tomcat takes about 30-40 mins to load/startup due to some user data loading on startup. So that leaves the only option of killing certain threads/processes and then starting Tomcat. which works fine - only we have been doing this every night for the past more than a month. We also got a thread dump just to check what is going on.. and I have pasted that info herewith. Sun has suggested that we install all patches to the JDK. I just wanted to find out if anyone had faced a similar issue and if there was something they did that fixed the issue. Was Tomcat the culprit?? We will go ahead and install the patches very soon - but wanted to be ready in case that didnt fix the issue - did anyone experience an improvement once the patches were installed?? * Here is the Exception Trace: An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0xFF380CBC Function=memcpy+0x7E0 Library=/usr/platform/sun4u/lib/libc_psr.so.1 Current Java thread: at COM.ibm.db2.jdbc.app.DB2PreparedStatement.SQLBindChar(Native Method) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.execute2(DB2PreparedStatement.java :1020) - locked 0x9ec56ef8 (a COM.ibm.db2.jdbc.app.DB2Connection) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.executeUpdate(DB2PreparedStatement .java:807) at com.waveset.repository.RelationalDataStore$Item.setPersistentObject(Relation alDataStore.java:999) at com.waveset.repository.AbstractDataStore.setItems(AbstractDataStore.java:492 6) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1401 ) at com.waveset.repository.ServerRepository.checkin(ServerRepository.java:2276) at com.waveset.session.InternalSession.checkinObject(InternalSession.java:581) at com.waveset.task.Scheduler.storeExecutingTask(Scheduler.java:1515) at com.waveset.task.Scheduler.executeReadyTask(Scheduler.java:2918) at com.waveset.task.Scheduler.processReadyTasks(Scheduler.java:2837) at com.waveset.task.Scheduler.processTasks(Scheduler.java:1031) at com.waveset.task.Scheduler.run(Scheduler.java:866) Dynamic libraries: 0x1 /www4/j2sdk1_4_2_06/java/usr/j2se/bin/java 0xff36 /usr/lib/libthread.so.1 0xff39a000 /usr/lib/libdl.so.1 0xff28 /usr/lib/libc.so.1 0xff38 /usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1 0xfec0 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/client/libjvm.so 0xff24 /usr/lib/libCrun.so.1 0xff21 /usr/lib/libsocket.so.1 0xff10 /usr/lib/libnsl.so.1 0xff1e /usr/lib/libm.so.1 0xff1c /usr/lib/libsched.so.1 0xff27 /usr/lib/libw.so.1 0xff0d /usr/lib/libmp.so.2 0xff0b /usr/lib/librt.so.1 0xff09 /usr/lib/libaio.so.1 0xff06 /usr/lib/libmd5.so.1 0xff04 /usr/platform/SUNW,Ultra-80/lib/libmd5_psr.so.1 0xfebd /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/native_threads/libhpi.so 0xfeb8 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libverify.so 0xfeb4 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libjava.so 0xfeb2 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libzip.so 0x737d /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/libnet.so
Re: Tomcat 4.1.30 crashing on production site - need help
It might also be that Tomcat needs more memory than the JVM will allocate. We had that problem. With the Sun JVM, the options -Xss128k -Xmx128m increased the stack and max heap size and there no longer was a problem. Try to set CATALINA_OPTS in bin/startup.sh: CATALINA_OPTS=$CATALINA_OPTS -server -Xss128k -Xmx128m Otherwise, try to get MRTG to graph your Tomcat's memory usage, like it's done in the attached PNG. Best regards, -- DAvid --On Monday, March 07, 2005 4:49 PM + James Sys [EMAIL PROTECTED] wrote: I've seen similar problems in a non-Tomcat Java application. The root cause was a memory leak. Although Java has good garbage collection, it relies on object references being released. This is generally fine, but some buggy applications can add references to large object to the application and forget to release them. This causes memory to be consumed, the app so slow and eventually the app or server to lock. With Tomcat, I can imagine this happening with objects continually being added to, say, an ArrayList in the Application context and not being released. I believe there are various tools around to check object allocation and release. So if you see objects of a certain type have a gradually increasing number, you may well have a memory leak. Hope this helps. Regards, James. = Hi All, For the past couple of months we have been facing a peculiar problem on our production environment. And we have been unable to resolve this issue so far. We have an application hosted on Tomcat 4.1.30 and using JDK1.4.2_06. We also have an Apache WS - (think it is v2.0.2). Everyday approx after every 24 hours of uptime the website slows down and hangs... On checking the tomcat logs, we see an OutOfMemoryException. Restarting Tomcat was one way to momentarily fix the issue and have the site up - but due to Apache forked processes we are not able to restart Tomcat too. Even if could our Tomcat takes about 30-40 mins to load/startup due to some user data loading on startup. So that leaves the only option of killing certain threads/processes and then starting Tomcat. which works fine - only we have been doing this every night for the past more than a month. We also got a thread dump just to check what is going on.. and I have pasted that info herewith. Sun has suggested that we install all patches to the JDK. I just wanted to find out if anyone had faced a similar issue and if there was something they did that fixed the issue. Was Tomcat the culprit?? We will go ahead and install the patches very soon - but wanted to be ready in case that didnt fix the issue - did anyone experience an improvement once the patches were installed?? * Here is the Exception Trace: An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0xFF380CBC Function=memcpy+0x7E0 Library=/usr/platform/sun4u/lib/libc_psr.so.1 Current Java thread: at COM.ibm.db2.jdbc.app.DB2PreparedStatement.SQLBindChar(Native Method) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.execute2(DB2PreparedStatement.java :1020) - locked 0x9ec56ef8 (a COM.ibm.db2.jdbc.app.DB2Connection) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.executeUpdate(DB2PreparedStatement .java:807) at com.waveset.repository.RelationalDataStore$Item.setPersistentObject(Relation alDataStore.java:999) at com.waveset.repository.AbstractDataStore.setItems(AbstractDataStore.java:492 6) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1401 ) at com.waveset.repository.ServerRepository.checkin(ServerRepository.java:2276) at com.waveset.session.InternalSession.checkinObject(InternalSession.java:581) at com.waveset.task.Scheduler.storeExecutingTask(Scheduler.java:1515) at com.waveset.task.Scheduler.executeReadyTask(Scheduler.java:2918) at com.waveset.task.Scheduler.processReadyTasks(Scheduler.java:2837) at com.waveset.task.Scheduler.processTasks(Scheduler.java:1031) at com.waveset.task.Scheduler.run(Scheduler.java:866) Dynamic libraries: 0x1 /www4/j2sdk1_4_2_06/java/usr/j2se/bin/java 0xff36 /usr/lib/libthread.so.1 0xff39a000 /usr/lib/libdl.so.1 0xff28 /usr/lib/libc.so.1 0xff38 /usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1 0xfec0 /www4/j2sdk1_4_2_06/java/usr/j2se/jre/lib/sparc/client/libjvm.so 0xff24 /usr/lib/libCrun.so.1 0xff21 /usr/lib/libsocket.so.1 0xff10 /usr/lib/libnsl.so.1 0xff1e /usr/lib/libm.so.1 0xff1c /usr/lib/libsched.so.1 0xff27 /usr/lib/libw.so.1 0xff0d /usr/lib/libmp.so.2 0xff0b /usr/lib/librt.so.1 0xff09
Re: Tomcat 4.1.30 crashing on production site - need help
So - JDK with or without the patches may not be the issue here?? Did anyone install patches for JDK 1.4.2_06 and see any improvement or anything?? -Anoop On Mon, 07 Mar 2005 18:50:49 +0100, David Tonhofer, m-plify S.A. [EMAIL PROTECTED] wrote: It might also be that Tomcat needs more memory than the JVM will allocate. We had that problem. With the Sun JVM, the options -Xss128k -Xmx128m increased the stack and max heap size and there no longer was a problem. Try to set CATALINA_OPTS in bin/startup.sh: CATALINA_OPTS=$CATALINA_OPTS -server -Xss128k -Xmx128m Otherwise, try to get MRTG to graph your Tomcat's memory usage, like it's done in the attached PNG. Best regards, -- DAvid --On Monday, March 07, 2005 4:49 PM + James Sys [EMAIL PROTECTED] wrote: I've seen similar problems in a non-Tomcat Java application. The root cause was a memory leak. Although Java has good garbage collection, it relies on object references being released. This is generally fine, but some buggy applications can add references to large object to the application and forget to release them. This causes memory to be consumed, the app so slow and eventually the app or server to lock. With Tomcat, I can imagine this happening with objects continually being added to, say, an ArrayList in the Application context and not being released. I believe there are various tools around to check object allocation and release. So if you see objects of a certain type have a gradually increasing number, you may well have a memory leak. Hope this helps. Regards, James. = Hi All, For the past couple of months we have been facing a peculiar problem on our production environment. And we have been unable to resolve this issue so far. We have an application hosted on Tomcat 4.1.30 and using JDK1.4.2_06. We also have an Apache WS - (think it is v2.0.2). Everyday approx after every 24 hours of uptime the website slows down and hangs... On checking the tomcat logs, we see an OutOfMemoryException. Restarting Tomcat was one way to momentarily fix the issue and have the site up - but due to Apache forked processes we are not able to restart Tomcat too. Even if could our Tomcat takes about 30-40 mins to load/startup due to some user data loading on startup. So that leaves the only option of killing certain threads/processes and then starting Tomcat. which works fine - only we have been doing this every night for the past more than a month. We also got a thread dump just to check what is going on.. and I have pasted that info herewith. Sun has suggested that we install all patches to the JDK. I just wanted to find out if anyone had faced a similar issue and if there was something they did that fixed the issue. Was Tomcat the culprit?? We will go ahead and install the patches very soon - but wanted to be ready in case that didnt fix the issue - did anyone experience an improvement once the patches were installed?? * Here is the Exception Trace: An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0xFF380CBC Function=memcpy+0x7E0 Library=/usr/platform/sun4u/lib/libc_psr.so.1 Current Java thread: at COM.ibm.db2.jdbc.app.DB2PreparedStatement.SQLBindChar(Native Method) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.execute2(DB2PreparedStatement.java :1020) - locked 0x9ec56ef8 (a COM.ibm.db2.jdbc.app.DB2Connection) at COM.ibm.db2.jdbc.app.DB2PreparedStatement.executeUpdate(DB2PreparedStatement .java:807) at com.waveset.repository.RelationalDataStore$Item.setPersistentObject(Relation alDataStore.java:999) at com.waveset.repository.AbstractDataStore.setItems(AbstractDataStore.java:492 6) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.set(AbstractDataStore.java:1804) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1758 ) at com.waveset.repository.AbstractDataStore.checkin(AbstractDataStore.java:1401 ) at com.waveset.repository.ServerRepository.checkin(ServerRepository.java:2276) at com.waveset.session.InternalSession.checkinObject(InternalSession.java:581) at com.waveset.task.Scheduler.storeExecutingTask(Scheduler.java:1515) at com.waveset.task.Scheduler.executeReadyTask(Scheduler.java:2918) at com.waveset.task.Scheduler.processReadyTasks(Scheduler.java:2837) at com.waveset.task.Scheduler.processTasks(Scheduler.java:1031) at com.waveset.task.Scheduler.run(Scheduler.java:866) Dynamic libraries: 0x1 /www4/j2sdk1_4_2_06/java/usr/j2se/bin/java 0xff36 /usr/lib/libthread.so.1 0xff39a000
