Re: Analyzing Tomcat related VM crash?
On 12.05.2009 01:13, Andre-John Mas wrote: I have added the details to the end of this e-mail. # SIGSEGV (0xb) at pc=0x06546578, pid=7456, tid=2349013936 Current thread (0x08060800): GCTaskThread [stack: 0x8bfb1000,0x8c032000] [id=7459] Other Threads: 0x8bd97800 VMThread [stack: 0x8baba000,0x8bb3b000] [id=7462] 0x8bdb3800 WatcherThread [stack: 0x8b5f3000,0x8b674000] [id=7469] =0x08060800 (exited) GCTaskThread [stack: 0x8bfb1000,0x8c032000] [id=7459] Heap PSYoungGen total 186112K, used 176507K [0xa7ad, 0xb42d, 0xb42d) eden space 167424K, 100% used [0xa7ad,0xb1e5,0xb1e5) from space 18688K, 48% used [0xb1e5,0xb272ec48,0xb309) to space 18496K, 39% used [0xb30c,0xb37d07d8,0xb42d) PSOldGentotal 319488K, used 266765K [0x942d, 0xa7ad, 0xa7ad) object space 319488K, 83% used [0x942d,0xa47535f8,0xa7ad) PSPermGen total 131072K, used 35161K [0x8c2d, 0x942d, 0x942d) object space 131072K, 26% used [0x8c2d,0x8e5265e8,0x942d) The crash happened in the GCTaskThread, so inside the code which is responsible for doing Garbage Collection. As noted by others, the exception you saw in the logo file is extremely unlikely to cause a JVM crash. Look at the release notes for 1.6.0_13 whether there's a fix for C crashes included and try Chucks suggestion about using the client VM. What's a bit strange in the above output concerning your memory use, is that from and to are both non-empty. Usually after a GC, one of the two is always empty. So it seems the crash happened while the JVM was doing a garbage collection (the full young generation is another indicator for that). You old generation is also reletively full, but we can't judge from the numbers whether that would hevae been simply cleaned up by a GC of tenured or not. Look for a GC fix (1.6.0_13), or try the child vm, or try switching to another GC algorithm, like CMS. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Analyzing Tomcat related VM crash?
From: Rainer Jung [mailto:rainer.j...@kippdata.de] Look for a GC fix (1.6.0_13), or try the child vm, or try switching to another GC algorithm, like CMS. (Coming to the thread late, sorry if this has all already been said). Also consider bad RAM, or memory corruption caused by some native code, if you have any native code in your app - JNI, ODBC drivers, other database drivers that are not pure Java, etc. From my days dealing with Smalltalk GC, I found the commonest cause of a crash during GC was a bad object header somewhere in the heap - usually because something's overwritten it or a bit's flipped due to bad RAM. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Analyzing Tomcat related VM crash?
Hi, We have recently put into production a Tomcat 5.5.27 based web site, using JDK 1.6 (JDK 1.6.0_12-b04 on RedHat Linux). We have spent months developing and testing the site, both by us and the customer, and experienced no VM crashes. Now that we are in production we find the VM is crashing from time to time. Trying to see if anyone else has experienced the same thing I simply find plenty of requests for help, but no answers. Looking at my catalina log I see: [2009-05-08 01:22:33,174] ERROR (StandardWrapperValve.java:260) - Servlet.service() for servlet action threw exception java.lang.IllegalStateException at org .apache .catalina.connector.ResponseFacade.sendError(ResponseFacade.java:405) at org .apache .struts.action.RequestProcessor.processMapping(RequestProcessor.java: 658) at org .apache.struts.action.RequestProcessor.process(RequestProcessor.java: 193) at org.apache.struts.action.ActionServlet.process(ActionServlet.java: 1164) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org .apache .catalina .core .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java: 269) at org .apache .catalina .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.myco.web.filter.CharsetFilter.doFilter(CharsetFilter.java:35) I doubt this would be enough to destabilize a VM? We have a second filter that might be picky about the order that it is called, and we will look into this, but not being able to reproduce this issue in a non-producion environment is making it challenging. The 'uname -a' command gives: Linux myco-prod01 2.4.21-37.ELsmp #1 SMP Wed Sep 7 13:28:55 EDT 2005 i686 i686 i386 GNU/Linux Can anyone suggest an approach to establish the route cause? What stuff should I be checking? André-John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Analyzing Tomcat related VM crash?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André-John, On 5/11/2009 6:48 PM, Andre-John Mas wrote: We have recently put into production a Tomcat 5.5.27 based web site, using JDK 1.6 (JDK 1.6.0_12-b04 on RedHat Linux). We have spent months developing and testing the site, both by us and the customer, and experienced no VM crashes. Now that we are in production we find the VM is crashing from time to time. JVM crashes are no good :( Looking at my catalina log I see: [2009-05-08 01:22:33,174] ERROR (StandardWrapperValve.java:260) - Servlet.service() for servlet action threw exception java.lang.IllegalStateException This is unlikely to be causing any JVM crash. I doubt this would be enough to destabilize a VM? We have a second filter that might be picky about the order that it is called, and we will look into this, but not being able to reproduce this issue in a non-production environment is making it challenging. It's pretty much impossible to crash the JVM intentionally just using Java code. You have to go JNI to do that. I would (as always) be suspicious of the physical hardware. Try taking one of the servers out of service and run a memory/cpu test on it. I'll bet it's got bad memory. You didn't give any details of the crash. I'm guessing SIG11? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoIrc4ACgkQ9CaO5/Lv0PACWwCgvXa0DmHKjZMmIQyuWB93hyNk CnIAn2l1cvk0QihaxuguOmC9+LV5b5Wy =pKjQ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Analyzing Tomcat related VM crash?
On 11-May-2009, at 18:59, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André-John, On 5/11/2009 6:48 PM, Andre-John Mas wrote: We have recently put into production a Tomcat 5.5.27 based web site, using JDK 1.6 (JDK 1.6.0_12-b04 on RedHat Linux). We have spent months developing and testing the site, both by us and the customer, and experienced no VM crashes. Now that we are in production we find the VM is crashing from time to time. JVM crashes are no good :( Looking at my catalina log I see: [2009-05-08 01:22:33,174] ERROR (StandardWrapperValve.java:260) - Servlet.service() for servlet action threw exception java.lang.IllegalStateException This is unlikely to be causing any JVM crash. I doubt this would be enough to destabilize a VM? We have a second filter that might be picky about the order that it is called, and we will look into this, but not being able to reproduce this issue in a non-production environment is making it challenging. It's pretty much impossible to crash the JVM intentionally just using Java code. You have to go JNI to do that. I would (as always) be suspicious of the physical hardware. Try taking one of the servers out of service and run a memory/cpu test on it. I'll bet it's got bad memory. You didn't give any details of the crash. I'm guessing SIG11? I have added the details to the end of this e-mail. André-John catalina.out: # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x06546578, pid=7456, tid=2349013936 # # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode linux-x86) # Problematic frame: # V [libjvm.so+0x546578] # # An error report file with more information is saved as: # /home/apache/hs_err_pid7456.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # /home/apache/hs_err_pid7456.log (certain names changed, for anonymity): # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x06546578, pid=7456, tid=2349013936 # # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode linux-x86) # Problematic frame: # V [libjvm.so+0x546578] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x08060800): GCTaskThread [stack: 0x8bfb1000,0x8c032000] [id=7459] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x Registers: EAX=0x0001, EBX=0xa46f8520, ECX=0x, EDX=0x ESP=0x8c0318d0, EBP=0x8c0318f8, ESI=0xb42d, EDI=0x8bd56680 EIP=0x06546578, CR2=0x, EFLAGS=0x00010246 Top of Stack: (sp=0x8c0318d0) 0x8c0318d0: 0805ee88 8bd566d8 8c0318f8 0001 0x8c0318e0: 0805eec8 0100 a46f8520 8bd56680 0x8c0318f0: 08060800 0001 8c031938 0654922b 0x8c031900: 8bd56680 0001 8bd56680 8bd56680 0x8c031910: 0805eec8 8bd56680 063488bd 0x8c031920: 0805eec8 8bd56680 8a8e6cb8 0x8c031930: 08060800 0666128c 8c0319d8 0632848b 0x8c031940: 8a8e6cb8 0805ee88 0001 Instructions: (pc=0x06546578) 0x06546568: f0 89 d9 83 e1 01 84 c9 0f 85 9f 00 00 00 8b 13 0x06546578: 8b 02 83 e0 03 83 f8 03 75 7e 8b 12 80 3d cd 87 Stack: [0x8bfb1000,0x8c032000], sp=0x8c0318d0, free space=514k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x546578] V [libjvm.so+0x54922b] V [libjvm.so+0x32848b] V [libjvm.so+0x506929] C [libpthread.so.0+0x4dec] --- P R O C E S S --- Java Threads: ( = current thread ) 0x8b1d1800 JavaThread http-8085-Monitor [_thread_blocked, id=7498, stack(0x89de3000,0x89e34000)] 0x8b1d JavaThread http-8085-Processor25 daemon [_thread_blocked, id=7497, stack(0x89e34000,0x89e85000)] 0x8b1ae800 JavaThread http-8085-Processor24 daemon [_thread_blocked, id=7496, stack(0x89e85000,0x89ed6000)] 0x8b1ad400 JavaThread http-8085-Processor23 daemon [_thread_blocked, id=7495, stack(0x89ed6000,0x89f27000)] 0x8a79fc00 JavaThread http-8085-Processor22 daemon [_thread_in_native, id=7494, stack(0x89f27000,0x89f78000)] 0x8a79e800 JavaThread http-8085-Processor21 daemon [_thread_blocked, id=7493, stack(0x89f78000,0x89fc9000)] 0x8a6b4c00 JavaThread http-8085-Processor20 daemon [_thread_blocked, id=7492, stack(0x89fc9000,0x8a01a000)] 0x8a6b3800 JavaThread http-8085-Processor19 daemon [_thread_blocked, id=7491, stack(0x8a01a000,0x8a06b000)] 0x8a842400 JavaThread http-8085-Processor18 daemon [_thread_blocked, id=7490, stack(0x8a06b000,0x8a0bc000)] 0x8a841000 JavaThread http-8085-Processor17 daemon [_thread_blocked, id=7489, stack(0x8a0bc000,0x8a10d000)] 0x8a8cd000 JavaThread http-8085-Processor16 daemon [_thread_blocked, id=7488, stack(0x8a10d000,0x8a15e000)] 0x8a8cbc00 JavaThread http-8085-Processor15 daemon [_thread_blocked, id=7487,
RE: Analyzing Tomcat related VM crash?
From: Andre-John Mas [mailto:aj...@sympatico.ca] Subject: Re: Analyzing Tomcat related VM crash? I have added the details to the end of this e-mail. If the hardware tests don't find anything, try running the JVM in client mode rather than server. Historically, the server JIT has been slightly less stable than the client JIT - although I'm not aware of any specific codegen problems in 6u12, client or server. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
