On May 18, 2013, at 11:37 AM, James Snider wrote: > -----Original Message----- > From: Daniel Mikusa [mailto:dmik...@gopivotal.com] > Sent: Saturday, May 18, 2013 10:24 AM > To: Tomcat Users List > Subject: Re: Apache Tomcat service has been shutting down/stopping randomly. > > On May 17, 2013, at 7:52 PM, James Snider wrote: > >> Hi Dan, > > Please don't top-post. The convention followed on this mailing list is to > either post inline (like my response to you) or to reply below any previous > message(s). This allows someone to read from top to bottom in order. > > My apologies, so only post within the quoted texts from past e0mails like so? > > >> To answer your questions below: >> >> 1. What's the exact version that you have installed? >> 7.0.40 >> >> 2. Again, signs that you are running Tomcat 5.5 and not Tomcat 7. Can you >> explain further? >> >> The change in the path from the default " C:\Program Files\Apache >> Software Foundation\Tomcat 7\" that I'm sure you're more accustomed to >> seeing to the our current path " C:\Program Files\Apache Software >> Foundation\Tomcat 5.5\" is due to some hardcoded paths we still have in a >> few of our web applications. We do have plans for our developers to change >> these applications in the future to being dynamic but for now we have been >> changing the default directory to " C:\Program Files\Apache Software >> Foundation\Tomcat 5.5\" when asked during installation as a workaround. > > This is certainly unconventional, but Tomcat doesn't really care about the > path where you install it. My concern with this approach is what happened to > the Tomcat 5.5 install prior to you installing Tomcat 7? Did you completely > remove it or is it possible that there could be old files still hanging > around in that directory? > > > Yes we made a backup then completely removed the past working directory with > the uninstallation. This effectively removed the past Tomcat 5.5 folder and > all of its contents. There were backups/other miscellaneous folders within > the "C:\Program Files\Apache Software Foundation\" folder that would prevent > the "./Apache Software Foundation" folder from being removed completely so we > decided to rename the folder to "C:\Program Files\Apache Software > Foundation.old" prior to starting installation of Tomcat 7. > > Once we finished installing we moved back the root directory (containing our > web applications).
OK > We also copied over our logs directories from the backup Tomcat 5.5 > directory. OK > We then moved back the "Commons" directory and "Shared" directory. This isn't a good idea. The directory structure changed between 5.5 and 6.0 [1]. https://tomcat.apache.org/migration-6.html#Modified_directory_structure While it's possible to use the old directory structure, I wouldn't recommend it. Delete these directories or better yet, start over with a fresh install so that you are sure you aren't running with any old JAR files. > My first suspicion was that maybe the common directory we moved back could be > causing the random shutdown as it contains a few .jar files that were already > contained in the new Tomcat 7 release in the /lib directory. The "Shared" > directory we copied over does not have any files within. > > Here is a list of the .jar files contained in the common directory we copied > back: > > ./Tomcat 5.5/common/classes > >empty > > ./Tomcat 5.5/common/endorsed > >empty > > ./Tomcat 5.5/common/i18n > >tomcat-i18n-en.jar > >tomcat-i18n-es.jar > >tomcat-i18n-fr.jar > >tomcat-i18n-ja.jar > > ./Tomcat 5.5/common/lib > >commons-el.jar > >ecj-3.7.jar > >jasper-compiler.jar > >jasper-runtime.jar > >jsp-api.jar > >naming-factory.jar > >naming-factory-dbcp.jar > >naming-resource.jar > >servlet-api.jar > > I thought that the common directory could be causing a issue so I renamed it > to /common.DNU to see if the random shutdown issue would cease but it still > seems to continue. We also > *Also, here is a list of the files contained in the Tomcat 5.5/lib directory > upon installation. > > ./Tomcat 5.5/lib > >annotations-api.jar > >catalina.jar > >catalina-ant.jar > >catalina-ha.jar > >catalina-tribes.jar > >ecj-4.2.2.jar > >el-api.jar > >jasper.jar > >jasper-el.jar > >jsp-api.jar > >servlet-api.jar > >tomcat-api.jar > >tomcat-coyote.jar > >tomcat-dbcp.jar > >tomcat-i18n-es.jar > >tomcat-i18n-fr.jar > >tomcat-i18n-ja.jar > >tomcat-jdbc.jar > >tomcat-util.jar > > > > > > >> 3. Please also indicate how you installed your Tomcat 7 version. >> >> We upgraded our servers from running Tomcat 5.5 to Tomcat 7.0.40 a >> week ago via the http://tomcat.apache.org/download-70.cgi page using the >> 32-bit/64-bit windows service installer. We installed Tomcat 7.0.40 using >> all defaults except we selected to install the tc-native-1.dll and did not >> select to install the documentation. Also during installation when prompted >> to select a Java Virtual Machine path we selected the path to a Java 7 JRE. >> (This was changed a few minutes AFTER installation once we found that the >> web applications were running on a different version of Java that had also >> been installed on the server (Java 1.6). Finally, during installation we >> changed the install destination folder from the default presented " >> C:\Program Files\Apache Software Foundation\Tomcat 7\" to " C:\Program >> Files\Apache Software Foundation\Tomcat 5.5\". > > Did you also follow the migration guide? > > http://tomcat.apache.org/migration.html > > There are significant changes from 5.5 to 7.0. You'll want to make sure you > migrated correctly or you could certainly see problems. > > > This was my first time seeing the migration guide. After reading I will make > sure to use this in the future. > > >> >> >> As an Heads up as I am not sure if this is related to the random shutting >> down/stopping; but we also are receiving this error in the >> "localhost.(date).log". Don't know if this is related but we are receiving >> this error randomly as well. >> >> SEVERE: Servlet.service() for servlet [jsp] in context with path [] >> threw exception [java.lang.IllegalStateException: getOutputStream() >> has already been called for this response] with root cause >> java.lang.IllegalStateException: getOutputStream() has already been called >> for this response >> at >> org.apache.catalina.connector.Response.getWriter(Response.java:639) >> at >> org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214) >> at >> org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:125) >> at >> org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:118) >> at >> org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:190) >> at >> org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:126) >> at >> org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:80) >> at >> org.apache.jsp.captcha_jsp._jspService(captcha_jsp.java:227) > > This looks like a problem in your JSP file. It's trying to call > getOutputStream() multiple times. See the line number in your JSP file for > more details. > > Dan > > >> at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) >> at >> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) >> at >> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) >> at >> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >> at >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >> at >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >> at >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) >> at >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >> at >> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1852) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Unknown Source) >> >> >> -----Original Message----- >> From: Daniel Mikusa [mailto:dmik...@gopivotal.com] >> Sent: Friday, May 17, 2013 4:38 PM >> To: Tomcat Users List >> Subject: Re: Apache Tomcat service has been shutting down/stopping randomly. >> >> On May 17, 2013, at 4:07 PM, James Snider wrote: >> >>> From: James Snider >>> Sent: Friday, May 17, 2013 10:21 AM >>> To: 'users@tomcat.apache.org' >>> Subject: Apache Tomcat service has been shutting down/stopping randomly. >>> >>> Hi, >>> >>> I have installed Tomcat 7 >> >> What's the exact version that you have installed? >> >>> on a 64 bit server running Windows 2003 R2 a few weeks ago. Since then, >>> the Apache Tomcat service has been shutting down/stopping randomly. When >>> the service stops a hr_err_(pid#).log file is generated in the home >>> directory. The contents of this I have pasted below: >>> (PLEASE NOTE: More details into our issue at bottom of e-mail after >>> log.) >>> >>> >>> >>> # >>> # A fatal error has been detected by the Java Runtime Environment: >>> # >>> # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000001800057b2, >>> pid=2740, tid=264 # # JRE version: 6.0_21-b07 # Java VM: Java >>> HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode windows-amd64 ) # >>> Problematic frame: >>> # C [tcnative-1.dll+0x57b2] >> >> Seems that there was a problem in the Tomcat native code. >> >>> # >>> # If you would like to submit a bug report, please visit: >>> # http://java.sun.com/webapps/bugreport/crash.jsp >>> # The crash happened outside the Java Virtual Machine in native code. >>> # See problematic frame for where to report the bug. >>> # >>> >>> --------------- T H R E A D --------------- >>> >>> Current thread (0x0000000004f38000): JavaThread "Finalizer" daemon >>> [_thread_in_native, id=264, >>> stack(0x0000000005b50000,0x0000000005c50000)] >>> >>> siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000040 >>> >>> Registers: >>> EAX=0x0000000000000000, EBX=0x0000000000000000, >>> ECX=0x0000000006a39160, EDX=0x0000000006f39000 >>> ESP=0x0000000005c4ec50, EBP=0x0000000000000000, >>> ESI=0x00000000068b61e0, EDI=0x000000000000006f >>> EIP=0x00000001800057b2, EFLAGS=0x0000000000010206 >>> >>> Top of Stack: (sp=0x0000000005c4ec50) >>> 0x0000000005c4ec50: 0000000005c4ecb0 0000000001395ae2 >>> 0x0000000005c4ec60: 0000000006f3906f 000000000155f3ae >>> 0x0000000005c4ec70: 00000000013959ce 00000000013a1260 >>> 0x0000000005c4ec80: 0000000005c4ed00 000000000000006f >>> 0x0000000005c4ec90: 0000000080c03178 000000000000006f >>> 0x0000000005c4eca0: 000000000000006f 0000000001395ae2 >>> 0x0000000005c4ecb0: 0000000001395ae2 000000000000006f >>> 0x0000000005c4ecc0: 0000000005c4ecc0 0000000000000000 >>> 0x0000000005c4ecd0: 0000000005c4ed38 0000000080c04790 >>> 0x0000000005c4ece0: 0000000000000000 0000000080c03178 >>> 0x0000000005c4ecf0: 0000000000000000 0000000005c4ed20 >>> 0x0000000005c4ed00: 0000000005c4ed80 00000000013959ce >>> 0x0000000005c4ed10: 0000000080c046d8 000000000139e316 >>> 0x0000000005c4ed20: 000000000000006f 0000000000000000 >>> 0x0000000005c4ed30: 00000000068b61e0 0000000085a31fa0 >>> 0x0000000005c4ed40: 0000000005c4ed40 0000000080f8778e >>> >>> Instructions: (pc=0x00000001800057b2) >>> 0x00000001800057a2: 8d 44 24 38 48 89 44 24 38 48 8b 46 30 48 03 d3 >>> 0x00000001800057b2: ff 50 40 85 c0 75 29 48 8b 44 24 38 48 85 c0 74 >>> >>> >>> Stack: [0x0000000005b50000,0x0000000005c50000], >>> sp=0x0000000005c4ec50, free space=3fb0000000000000000k Native frames: >>> (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C >>> [tcnative-1.dll+0x57b2] >>> >>> [error occurred during error reporting (printing native stack), id >>> 0xc0000005] >>> >>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j >>> org.apache.tomcat.jni.Socket.sendbb(JII)I+0 >>> j org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V+22 >>> j org.apache.coyote.http11.InternalAprOutputBuffer.flush()V+5 >>> J >>> org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/c >>> o >>> yote/ActionCode;Ljava/lang/Object;)V >>> j >>> org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava >>> / >>> lang/Object;)V+31 j >>> org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+97 >>> j org.apache.catalina.connector.OutputBuffer.flush()V+2 >>> j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4 >>> j javax.imageio.stream.FileCacheImageOutputStream.close()V+60 >>> j javax.imageio.stream.ImageInputStreamImpl.finalize()V+8 >>> v ~StubRoutines::call_stub >>> j >>> java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0 >>> j java.lang.ref.Finalizer.runFinalizer()V+45 >>> j java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1 >>> j java.lang.ref.Finalizer$FinalizerThread.run()V+11 >>> v ~StubRoutines::call_stub The stack traces reported by the crash dump file indicates that the error originated in the Tomcat native connector code. If you're OK with just working around this issue, switching to either the NIO or BIO connectors would likely cause this issue to go away. Dan >>> >>> Other Threads: >>> 0x0000000004f2d000 VMThread [stack: >>> 0x0000000005950000,0x0000000005a50000] [id=1644] >>> 0x0000000004fbf800 WatcherThread [stack: >>> 0x0000000006150000,0x0000000006250000] [id=608] >>> >>> VM state:not at safepoint (normal execution) >>> >>> VM Mutex/Monitor currently owned by a thread: None >>> >>> Heap >>> PSYoungGen total 5824K, used 589K [0x00000000da950000, >>> 0x00000000db7a0000, 0x00000001053f0000) >>> eden space 5696K, 10% used >>> [0x00000000da950000,0x00000000da9e3470,0x00000000daee0000) >>> from space 128K, 0% used >>> [0x00000000daee0000,0x00000000daee0000,0x00000000daf00000) >>> to space 4480K, 0% used >>> [0x00000000db340000,0x00000000db340000,0x00000000db7a0000) >>> PSOldGen total 53824K, used 20474K [0x00000000853f0000, >>> 0x0000000088880000, 0x00000000da950000) >>> object space 53824K, 38% used >>> [0x00000000853f0000,0x00000000867eebc8,0x0000000088880000) >>> PSPermGen total 55680K, used 25605K [0x000000007fff0000, >>> 0x0000000083650000, 0x00000000853f0000) >>> object space 55680K, 45% used >>> [0x000000007fff0000,0x00000000818f1700,0x0000000083650000) >>> >>> Dynamic libraries: >>> 0x0000000140000000 - 0x000000014001e000 C:\Program Files\Apache >>> Software Foundation\Tomcat 5.5\bin\Tomcat7.exe >> >> Can you explain this odd looking path? >> >>> 0x0000000077ec0000 - 0x0000000077ffb000 >>> C:\WINDOWS\system32\ntdll.dll >>> 0x0000000077d40000 - 0x0000000077eb3000 >>> C:\WINDOWS\system32\kernel32.dll >>> 0x000007ff7fee0000 - 0x000007ff7ffe6000 >>> C:\WINDOWS\system32\ADVAPI32.dll >>> 0x000007ff7fd30000 - 0x000007ff7fec9000 >>> C:\WINDOWS\system32\RPCRT4.dll >>> 0x000007ff7e9c0000 - 0x000007ff7e9e2000 >>> C:\WINDOWS\system32\Secur32.dll >>> 0x000007ff7f190000 - 0x000007ff7fb9b000 >>> C:\WINDOWS\system32\SHELL32.dll >>> 0x000007ff7fc00000 - 0x000007ff7fc86000 >>> C:\WINDOWS\system32\msvcrt.dll >>> 0x000007ff7fc90000 - 0x000007ff7fd2c000 >>> C:\WINDOWS\system32\GDI32.dll >>> 0x0000000077c20000 - 0x0000000077d2c000 >>> C:\WINDOWS\system32\USER32.dll >>> 0x000007ff7ef60000 - 0x000007ff7effb000 >>> C:\WINDOWS\system32\SHLWAPI.dll >>> 0x000007ff7d500000 - 0x000007ff7d539000 >>> C:\WINDOWS\system32\IMM32.DLL >>> 0x000000006fa00000 - 0x000000006fa3f000 >>> C:\PROGRA~2\Sophos\SOPHOS~1\SOPHOS~2.DLL >>> 0x000007ff7e380000 - 0x000007ff7e390000 >>> C:\WINDOWS\system32\PSAPI.DLL >>> 0x000007ff7f000000 - 0x000007ff7f187000 >>> C:\WINDOWS\WinSxS\amd64_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_0A7B2435\comctl32.dll >>> 0x000000006d7f0000 - 0x000000006dec1000 C:\Program >>> Files\Java\jre6\bin\server\jvm.dll >>> 0x000007ff7e240000 - 0x000007ff7e290000 >>> C:\WINDOWS\system32\WINMM.dll >>> 0x000000006d760000 - 0x000000006d76e000 C:\Program >>> Files\Java\jre6\bin\verify.dll >>> 0x000000006d3b0000 - 0x000000006d3d7000 C:\Program >>> Files\Java\jre6\bin\java.dll >>> 0x000000006d310000 - 0x000000006d31a000 C:\Program >>> Files\Java\jre6\bin\hpi.dll >>> 0x000000006d7b0000 - 0x000000006d7c2000 C:\Program >>> Files\Java\jre6\bin\zip.dll >>> 0x000000006d540000 - 0x000000006d54a000 C:\Program >>> Files\Java\jre6\bin\management.dll >>> 0x0000000180000000 - 0x000000018018c000 C:\Program Files\Apache >>> Software Foundation\Tomcat 5.5\bin\tcnative-1.dll >> >> Again, this looks odd. It looks like it's loading the tcnative library from >> Tomcat 5.5, but you said you're running Tomcat 7. >> >>> 0x000007ff77310000 - 0x000007ff77340000 >>> C:\WINDOWS\system32\WS2_32.dll >>> 0x000007ff77140000 - 0x000007ff7714c000 >>> C:\WINDOWS\system32\WS2HELP.dll >>> 0x000007ff771b0000 - 0x000007ff7722d000 >>> C:\WINDOWS\system32\MSWSOCK.dll >>> 0x0000000068000000 - 0x000000006804c000 >>> C:\WINDOWS\system32\rsaenh.dll >>> 0x000007ff6d200000 - 0x000007ff6d293000 >>> C:\WINDOWS\system32\hnetcfg.dll >>> 0x000007ff77170000 - 0x000007ff7717b000 >>> C:\WINDOWS\System32\wshtcpip.dll >>> 0x000000006d600000 - 0x000000006d617000 C:\Program >>> Files\Java\jre6\bin\net.dll >>> 0x000007ff7e8f0000 - 0x000007ff7e93e000 >>> C:\WINDOWS\system32\DNSAPI.dll >>> 0x000007ff7e9f0000 - 0x000007ff7e9fb000 >>> C:\WINDOWS\System32\winrnr.dll >>> 0x000007ff7e950000 - 0x000007ff7e9b6000 >>> C:\WINDOWS\system32\WLDAP32.dll >>> 0x000007ff7ea00000 - 0x000007ff7ea07000 >>> C:\WINDOWS\system32\rasadhlp.dll >>> 0x000000006d000000 - 0x000000006d1c3000 C:\Program >>> Files\Java\jre6\bin\awt.dll >>> 0x000007ff78f10000 - 0x000007ff78f51000 >>> C:\WINDOWS\system32\WINSPOOL.DRV >>> 0x000007ff57140000 - 0x000007ff573c5000 >>> C:\WINDOWS\system32\ole32.dll >>> 0x000000006d2a0000 - 0x000000006d30b000 C:\Program >>> Files\Java\jre6\bin\fontmanager.dll >>> 0x000000006d620000 - 0x000000006d62b000 C:\Program >>> Files\Java\jre6\bin\nio.dll >>> 0x000000006d4d0000 - 0x000000006d505000 C:\Program >>> Files\Java\jre6\bin\jpeg.dll >>> >>> VM Arguments: >>> jvm_args: -Dcatalina.home=C:\Program Files\Apache Software >>> Foundation\Tomcat 5.5 -Dcatalina.base=C:\Program Files\Apache >>> Software Foundation\Tomcat 5.5 -Djava.endorsed.dirs=C:\Program >>> Files\Apache Software Foundation\Tomcat 5.5\endorsed >>> -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat >>> 5.5\temp >>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >>> -Djava.util.logging.config.file=C:\Program Files\Apache Software >>> Foundation\Tomcat 5.5\conf\logging.properties exit >> >> Again, signs that you are running Tomcat 5.5 and not Tomcat 7. Can you >> explain further? >> >> Please also indicate how you installed your Tomcat 7 version. >> >> Dan >> >> >>> java_command: <unknown> >>> Launcher Type: generic >>> >>> Environment Variables: >>> PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Progr >>> a m Files (x86)\Dell\SysMgt\RAC5;C:\Program Files >>> (x86)\Dell\SysMgt\oma\bin;C:\WINDOWS\system32\WindowsPowerShell\v1.0; >>> C :\Program Files (x86)\Common Files\Acronis\SnapAPI\ OS=Windows_NT >>> PROCESSOR_IDENTIFIER=EM64T Family 6 Model 23 Stepping 10, >>> GenuineIntel >>> >>> >>> >>> --------------- S Y S T E M --------------- >>> >>> OS: Windows Server 2003 family Build 3790 Service Pack 2 >>> >>> CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 23 >>> stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1 >>> >>> Memory: 4k page, physical 8386584k(6556180k free), swap >>> 12121980k(10978712k free) >>> >>> vm_info: Java HotSpot(TM) 64-Bit Server VM (17.0-b17) for >>> windows-amd64 JRE (1.6.0_21-b07), built on Jul 17 2010 01:05:36 by >>> "java_re" with MS VC++ 8.0 (VS2005) >>> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
