RE: Apache Tomcat service has been shutting down/stopping randomly.
Hi Dan, To answer your questions below: 1. What's the exact version that you have installed? 7.0.40 2. Again, signs that you are running Tomcat 5.5 and not Tomcat 7. Can you explain further? The change in the path from the default " C:\Program Files\Apache Software Foundation\Tomcat 7\" that I'm sure you're more accustomed to seeing to the our current path " C:\Program Files\Apache Software Foundation\Tomcat 5.5\" is due to some hardcoded paths we still have in a few of our web applications. We do have plans for our developers to change these applications in the future to being dynamic but for now we have been changing the default directory to " C:\Program Files\Apache Software Foundation\Tomcat 5.5\" when asked during installation as a workaround. 3. Please also indicate how you installed your Tomcat 7 version. We upgraded our servers from running Tomcat 5.5 to Tomcat 7.0.40 a week ago via the http://tomcat.apache.org/download-70.cgi page using the 32-bit/64-bit windows service installer. We installed Tomcat 7.0.40 using all defaults except we selected to install the tc-native-1.dll and did not select to install the documentation. Also during installation when prompted to select a Java Virtual Machine path we selected the path to a Java 7 JRE. (This was changed a few minutes AFTER installation once we found that the web applications were running on a different version of Java that had also been installed on the server (Java 1.6). Finally, during installation we changed the install destination folder from the default presented " C:\Program Files\Apache Software Foundation\Tomcat 7\" to " C:\Program Files\Apache Software Foundation\Tomcat 5.5\". As an Heads up as I am not sure if this is related to the random shutting down/stopping; but we also are receiving this error in the "localhost.(date).log". Don't know if this is related but we are receiving this error randomly as well. SEVERE: Servlet.service() for servlet [jsp] in context with path [] threw exception [java.lang.IllegalStateException: getOutputStream() has already been called for this response] with root cause java.lang.IllegalStateException: getOutputStream() has already been called for this response at org.apache.catalina.connector.Response.getWriter(Response.java:639) at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214) at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:125) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:118) at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:190) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:126) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:80) at org.apache.jsp.captcha_jsp._jspService(captcha_jsp.java:227) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1852) at java.util.concurrent.ThreadPoolExecutor$Worker.
Re: Apache Tomcat service has been shutting down/stopping randomly.
On May 17, 2013, at 4:07 PM, James Snider wrote: > From: James Snider > Sent: Friday, May 17, 2013 10:21 AM > To: 'users@tomcat.apache.org' > Subject: Apache Tomcat service has been shutting down/stopping randomly. > > Hi, > > I have installed Tomcat 7 What's the exact version that you have installed? > on a 64 bit server running Windows 2003 R2 a few weeks ago. Since then, the > Apache Tomcat service has been shutting down/stopping randomly. When the > service stops a hr_err_(pid#).log file is generated in the home directory. > The contents of this I have pasted below: > (PLEASE NOTE: More details into our issue at bottom of e-mail after log.) > > > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x0001800057b2, > pid=2740, tid=264 > # > # JRE version: 6.0_21-b07 > # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode > windows-amd64 ) > # Problematic frame: > # C [tcnative-1.dll+0x57b2] Seems that there was a problem in the Tomcat native code. > # > # If you would like to submit a bug report, please visit: > # http://java.sun.com/webapps/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > > --- T H R E A D --- > > Current thread (0x04f38000): JavaThread "Finalizer" daemon > [_thread_in_native, id=264, stack(0x05b5,0x05c5)] > > siginfo: ExceptionCode=0xc005, reading address 0x0040 > > Registers: > EAX=0x, EBX=0x, ECX=0x06a39160, > EDX=0x06f39000 > ESP=0x05c4ec50, EBP=0x, ESI=0x068b61e0, > EDI=0x006f > EIP=0x0001800057b2, EFLAGS=0x00010206 > > Top of Stack: (sp=0x05c4ec50) > 0x05c4ec50: 05c4ecb0 01395ae2 > 0x05c4ec60: 06f3906f 0155f3ae > 0x05c4ec70: 013959ce 013a1260 > 0x05c4ec80: 05c4ed00 006f > 0x05c4ec90: 80c03178 006f > 0x05c4eca0: 006f 01395ae2 > 0x05c4ecb0: 01395ae2 006f > 0x05c4ecc0: 05c4ecc0 > 0x05c4ecd0: 05c4ed38 80c04790 > 0x05c4ece0: 80c03178 > 0x05c4ecf0: 05c4ed20 > 0x05c4ed00: 05c4ed80 013959ce > 0x05c4ed10: 80c046d8 0139e316 > 0x05c4ed20: 006f > 0x05c4ed30: 068b61e0 85a31fa0 > 0x05c4ed40: 05c4ed40 80f8778e > > Instructions: (pc=0x0001800057b2) > 0x0001800057a2: 8d 44 24 38 48 89 44 24 38 48 8b 46 30 48 03 d3 > 0x0001800057b2: ff 50 40 85 c0 75 29 48 8b 44 24 38 48 85 c0 74 > > > Stack: [0x05b5,0x05c5], sp=0x05c4ec50, free > space=3fbk > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [tcnative-1.dll+0x57b2] > > [error occurred during error reporting (printing native stack), id 0xc005] > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j org.apache.tomcat.jni.Socket.sendbb(JII)I+0 > j org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V+22 > j org.apache.coyote.http11.InternalAprOutputBuffer.flush()V+5 > J > org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V > j > org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31 > j org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+97 > j org.apache.catalina.connector.OutputBuffer.flush()V+2 > j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4 > j javax.imageio.stream.FileCacheImageOutputStream.close()V+60 > j javax.imageio.stream.ImageInputStreamImpl.finalize()V+8 > v ~StubRoutines::call_stub > j java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0 > j java.lang.ref.Finalizer.runFinalizer()V+45 > j java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1 > j java.lang.ref.Finalizer$FinalizerThread.run()V+11 > v ~StubRoutines::call_stub > > --- P R O C E S S --- > > Java Threads: ( => current thread ) > 0x071c0800 JavaThread "http-apr-80-exec-10" daemon > [_thread_blocked, id=4240, stack(0x0df5,0x0e05)] > 0x071c JavaThread "http-apr-80-exec-9" daemon [_thread_blocked, > id=4236, stack(0x0de5,0x0df5)] > 0x07164000 JavaThread "http-apr-80-exec-8" daemon [_thread_blocked, > id=4232, stack(0x0dd5,0x0de5)] > 0x07163000 JavaThread "http-apr-80-e
Re: ErrorReportValve's char encoding ignored
Am 2013-05-15 16:35, schrieb Christopher Schultz:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Micheal,
On 5/15/13 3:46 AM, Michael-O wrote:
I have modified the ErrorReportValve's messages outside of the
one-byte range and the output got mangled. The response does not
contain a char encoding anymore.
Can you be more specific? What exactly did you modify, and how did you
modify it? What are the exact HTTP response headers that you get?
I know that the error valve does this:
response.setContentType("text/html");
response.setCharacterEncoding("utf-8");
but the setCharEnc method is completely ignored. Because the var
usingWriter is already set to true. This happens in the
ApplicationDispatcher#doForward line 393/394 when a servlet
performs a forward.
Looking at Tomcat 6.0.x/trunk line 393 is part of the cleanup code,
and the response PrintWriter is immediately closed. That does not
appear to have changed since 6.0.35. At this point, the request should
be /over/.
What else is going on? Is any content being generated before the
ErrorReportValve gets invoked? If the response has already been
committed, then Tomcat can't re-write the response headers.
Though a response.getReporter() is used but this one is completely
unaware of the char encoding.
I am on Tomcat 6.0.35.
Where is response.getReporter being called?
I presume this to be a bug. Should I file a BZ issue?
Hi Christopher,
did you get any chance to check my reply from 2013-05-15 17:13+02:00?
Hopefully, that mail made the issue clearer.
Thanks,
Michael
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
On 17/05/2013 15:34, Jeffrey Janner wrote: > Michael and Mark - I happened to be reviewing how Oracle handles > QueryTimeout yesterday on an unrelated issue and came across a > passage in the JDBC Developers Guide (11g) that covered this > Monitoring Thread (page E-3). My reading of it was that Oracle > creates a single monitoring thread per JVM, so if there are multiple > apps utilizing the Oracle driver, you certainly don't want one app > shutting down and taking the monitor thread with it. I'm sure this > one of the reasons that Mark says that the monitor thread should use > the parent class loader. Oracle should probably consider this. My > question though is how is all this affected when the jdbc library is > loaded on a per-app basis, i.e., it's not shared but loaded from each > app's WEB-INF/lib folder? Should the app then unload the driver when > it shuts down to avoid memory leaks? Or is there a special process > that needs to be followed? Jeff If the driver is in WEB-INF/lib things get really messy, very quickly. You can make it work if only that application is using the driver (you have to make sure you manually deregister the driver in a ServletContextListener). If multiple apps try and use it, or if multiple apps ship the same Driver, bad things happen. There are multiple problems: 1. Using a driver automatically adds it to the DriverManager. DriverManager is a JVM singleton. It maintains a list of loaded drivers but does not key this list by class loader so there can only ever be one instance of a Driver loaded even if different web applications ship different versions of the same Driver. 2. If web app A loads the Driver and web app B tries to use it the chances of a CNFE or similar are fairly high. You might get away with it. You probably won't. 3. If webapp A loads the driver, webapp B manages to use it and then webapp A is undeployed, assuming webapp A does the right thing and deregisaters it, webapp A will pull the driver out from underneath webapp B. Generally, putting it in CATALINA_HOME/lib is better. That also means Tomcat can provide database connection pooling. DriverManager is just one of the many JRE classes that make no allowances for running in the class loader environments you get in J2EE containers. LogManager is another. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Aw: RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
> > -Original Message- > > From: Mark Thomas [mailto:ma...@apache.org] > > Sent: Friday, May 17, 2013 7:26 AM > > To: Tomcat Users List > > Subject: Re: Follow-up: Possible false-postive with > > JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > > OracleTimeoutPollingThread > > > > On 17/05/2013 12:31, Michael-O wrote: > > > Hi Mark, > > > > > > thanks again for the detailed answer, details inline. > > > > > >> Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr > > >> Von: "Mark Thomas" > > >> An: "Tomcat Users List" > > >> Betreff: Re: Follow-up: Possible false-postive with > > >> JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > > >> OracleTimeoutPollingThread > > >> > > >> On 17/05/2013 09:28, Michael-O wrote: > > >>> Hi folks, > > >>> > > >>> there's now a follow-up on the issue [1]. > > >>> Recap: JreMemoryLeakPreventionListener reported that my webapp has > > spawned OracleTimeoutPollingThread because I have set a QueryTimeout in > > the JDBC Pool. > > >>> Mark Thomas noted that this is likely a bug in the driver. I have > > taken action and created a service request with Oracle. > > >>> > > >>> My personal analysis with VisualVM: > > >>> The thread is spawned when the first query is run. The thread keeps > > running when the webapp is shut down or undeployed. Classes remain in > > memory. > > >>> The JDBC Pool does a simple Class.forName to load the driver, it > > neither uses the DriverManager loading nor does it actively unload the > > driver. > > >>> > > >>> Oracle answer: > > >>> They were able to reproduce the issue. The technical analysis says: > > >>> > > Hi Michael, > > > > I confirmned internally that this message from Tomcat can be > > ignored as there is no risk of any real leak from" > > OracleTimeoutPollingThread" thread. > > This thread is related to the JDBC driver which may be used by > > many apps simultaneously. Unloading the app does not unload the driver > > and should not and does not stop the thread. > > > > It seems to be the design behaviour. > > >>> > > >>> So my questions would be: > > >>> 1. Is that still not a false positive? > > >> > > >> No, that is not a false positive. The response from Oracle is wrong. > > >> > > >> There is nothing wrong with the driver creating the thread or the > > >> thread continuing after the web application has stopped. The problem > > is as follows: > > >> > > >> 1. When the JDBC driver creates the Thread, the Thread's context > > >> class loader (as returned by Thread.getContextClassLoader()) is set > > >> to the web application's class loader. > > >> > > >> 2. The correct behaviour at this point would be for the Driver to > > set > > >> the Thread's context class loader to the class loader that loaded > > the > > >> Driver class when the Thread is created. > > >> > > >> 3. The memory leak occurs as follows: > > >> - the web application is stopped > > >> - Tomcat clears all references to the web application and its > > classes > > >> - The web application should be eligible for garbage collection > > >> - The JDBC driver is still loaded (as it should be) > > >> - The JDBC driver retains a reference to the Thread (as it should) > > >> - The thread retains a reference to the web application class loader > > >> (this is the memory leak). > > >> > > >> The reference chain is: > > >> a) JDBC driver > > >> b) Thread > > >> c) Web application class loader > > >> d) Every class loaded by the web app > > >> > > >> Everything from c) onwards should be eligible for garbage collection > > >> but isn't because of the leak in the Thread that is retaining a > > >> reference to the web application class loader. > > > > > > This is what I would assume as correct behavior. If the thread is > > still attached ot the WebAppClassLoader, does that mean that the > > WebAppClassLoader cannot be garbage collected? > > > > The correct behaviour is that there is no reference to c) held by b) > > and therefore c) onwards would be eligible for GC (assuming the web app > > has been stopped). The problem here is that because b) has a reference > > to c), c) can't be GC'd and that is a memory leak. > > > > > > >>> 2. Why does the JDBC Pool not unload the driver? That my cause the > > thread to stop after the last app has been stopped/undeployed. > > >> > > >> 1. Because the driver is in CATALINA_HOME/lib it is available to > > >> other applications. > > > > > > So first app, loads driver and keeps it in memory. Even if all apps > > are undeployed, driver remains cached and GCed when Tomcat is shut > > down? > > > > The JDBC driver will be loaded though the services API as soon as any > > code references the DriverManager. > > > > >> 2. JDBC drivers are never automatically unloaded. You have to do so > > >> explicitly (not doing so is an other source of memory leaks when the > > >> driver is packaged in WEB-INF/lib). > > > > > > OK, this won't be the case. Driver is always shared here. > > > > > >> Yo
Re: Running Thrift Server inside Tomcat.
Hi, Thanks. It worked when followed those steps. -Denuwanthi On Fri, May 17, 2013 at 9:17 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Isuru, > > On 5/16/13 10:36 AM, Isuru Ranawaka wrote: > > hi, Did you try to start thriftserver in a separate thread rather > > than start it directly in the init method. because i think > > execution on current thread is blocked due to start listening on > > perticular port .try it by creating a Runnable class and start > > Serversocket in run method and start thread by giving that runnable > > class as target in the init method. > > ... and then move everything into a ServletContextListener where it > belongs. > > Don't forget to shut-down the TServetSocket and the associated thread > in the destroy() method, too, or you will have a pretty serious > resource leak. > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRlahpAAoJEBzwKT+lPKRYbaIP/3prOBuf7tERGzSMVth38Cyl > Y0Z8cs1xCTVcOVxNJsEDIQN1imv3fy2S4nOZmRC5hKfqh3rr/vYlL60e9TKktPnL > QfYJS2wAGgUN++Y1yAZ6fvo/LQ12x0TRDbf4EuwBJ5NyAUFy9pP1La3MpmiWMcvP > LTgRA4VEaW6b3GugqlgiqIFPF6yV5JfzO28cQmAbHY2DVFHQ2toT7BjAI5hCfAY6 > tqdEO1xNbJud8jGYTy2NHEs5Jgu/z1gRmNna7RZ4loFe0tj/inNhqOWWerlOWCp4 > XkHyGcfqnb3Wn/+Oq+IiLR1lCPImwVOHX5kJrHwEdIP4IhnoDNdXQihfNt5vg1Qv > EDd0ax6PGisJ7a5bK7YVAnpM0/CJ9hQ61/tu72nKEoF+oGHuZfdFdvmFWgEnKaqC > xqT89dWfsY7Gw4tK1pBhI74wl+Ci9iWw8LlbqWtvYhqAihOv7okYmBvH5LjFfvaZ > dRRHAq7aTHnPa+GvqxSja/B37O42pfqdYPROT8YqQ1nYw21IJ+iNoECuTwS/Czdo > WNZtFB/qUhrCNa7nmqClqLOpRAnerP9i+Xe7vrCNcCN6jj3Q/BL2lTLLPdUUwccH > QakmVk1NwLPfqe+SE98Za2DowM3dwIP8fKT+ZxPQUUU0fuPBP0MXnlrAjxYRdv90 > a4qcsLybuN8DgtQrrJwA > =7NEA > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
> -Original Message- > From: Mark Thomas [mailto:ma...@apache.org] > Sent: Friday, May 17, 2013 7:26 AM > To: Tomcat Users List > Subject: Re: Follow-up: Possible false-postive with > JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > OracleTimeoutPollingThread > > On 17/05/2013 12:31, Michael-O wrote: > > Hi Mark, > > > > thanks again for the detailed answer, details inline. > > > >> Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr > >> Von: "Mark Thomas" > >> An: "Tomcat Users List" > >> Betreff: Re: Follow-up: Possible false-postive with > >> JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > >> OracleTimeoutPollingThread > >> > >> On 17/05/2013 09:28, Michael-O wrote: > >>> Hi folks, > >>> > >>> there's now a follow-up on the issue [1]. > >>> Recap: JreMemoryLeakPreventionListener reported that my webapp has > spawned OracleTimeoutPollingThread because I have set a QueryTimeout in > the JDBC Pool. > >>> Mark Thomas noted that this is likely a bug in the driver. I have > taken action and created a service request with Oracle. > >>> > >>> My personal analysis with VisualVM: > >>> The thread is spawned when the first query is run. The thread keeps > running when the webapp is shut down or undeployed. Classes remain in > memory. > >>> The JDBC Pool does a simple Class.forName to load the driver, it > neither uses the DriverManager loading nor does it actively unload the > driver. > >>> > >>> Oracle answer: > >>> They were able to reproduce the issue. The technical analysis says: > >>> > Hi Michael, > > I confirmned internally that this message from Tomcat can be > ignored as there is no risk of any real leak from" > OracleTimeoutPollingThread" thread. > This thread is related to the JDBC driver which may be used by > many apps simultaneously. Unloading the app does not unload the driver > and should not and does not stop the thread. > > It seems to be the design behaviour. > >>> > >>> So my questions would be: > >>> 1. Is that still not a false positive? > >> > >> No, that is not a false positive. The response from Oracle is wrong. > >> > >> There is nothing wrong with the driver creating the thread or the > >> thread continuing after the web application has stopped. The problem > is as follows: > >> > >> 1. When the JDBC driver creates the Thread, the Thread's context > >> class loader (as returned by Thread.getContextClassLoader()) is set > >> to the web application's class loader. > >> > >> 2. The correct behaviour at this point would be for the Driver to > set > >> the Thread's context class loader to the class loader that loaded > the > >> Driver class when the Thread is created. > >> > >> 3. The memory leak occurs as follows: > >> - the web application is stopped > >> - Tomcat clears all references to the web application and its > classes > >> - The web application should be eligible for garbage collection > >> - The JDBC driver is still loaded (as it should be) > >> - The JDBC driver retains a reference to the Thread (as it should) > >> - The thread retains a reference to the web application class loader > >> (this is the memory leak). > >> > >> The reference chain is: > >> a) JDBC driver > >> b) Thread > >> c) Web application class loader > >> d) Every class loaded by the web app > >> > >> Everything from c) onwards should be eligible for garbage collection > >> but isn't because of the leak in the Thread that is retaining a > >> reference to the web application class loader. > > > > This is what I would assume as correct behavior. If the thread is > still attached ot the WebAppClassLoader, does that mean that the > WebAppClassLoader cannot be garbage collected? > > The correct behaviour is that there is no reference to c) held by b) > and therefore c) onwards would be eligible for GC (assuming the web app > has been stopped). The problem here is that because b) has a reference > to c), c) can't be GC'd and that is a memory leak. > > > >>> 2. Why does the JDBC Pool not unload the driver? That my cause the > thread to stop after the last app has been stopped/undeployed. > >> > >> 1. Because the driver is in CATALINA_HOME/lib it is available to > >> other applications. > > > > So first app, loads driver and keeps it in memory. Even if all apps > are undeployed, driver remains cached and GCed when Tomcat is shut > down? > > The JDBC driver will be loaded though the services API as soon as any > code references the DriverManager. > > >> 2. JDBC drivers are never automatically unloaded. You have to do so > >> explicitly (not doing so is an other source of memory leaks when the > >> driver is packaged in WEB-INF/lib). > > > > OK, this won't be the case. Driver is always shared here. > > > >> You need to go back to Oracle. > > > > Yes, I will. We're paying probably a lot of money for he company-wide > license. > > I'd expect so. > > From my own experience with Oracle commercial support for things Java > related, you'll have to go
Apache Tomcat service has been shutting down/stopping randomly.
Hi, I have installed Tomcat 7 on a 64 bit server running Windows 2003 R2 a few weeks ago. Since then, the Apache Tomcat service has been shutting down/stopping randomly. When the service stops a hr_err_(pid#).log file is generated in the home directory. The contents of this I have pasted below: (PLEASE NOTE: More details into our issue at bottom of e-mail after log.) # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x0001800057b2, pid=2740, tid=264 # # JRE version: 6.0_21-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode windows-amd64 ) # Problematic frame: # C [tcnative-1.dll+0x57b2] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --- T H R E A D --- Current thread (0x04f38000): JavaThread "Finalizer" daemon [_thread_in_native, id=264, stack(0x05b5,0x05c5)] siginfo: ExceptionCode=0xc005, reading address 0x0040 Registers: EAX=0x, EBX=0x, ECX=0x06a39160, EDX=0x06f39000 ESP=0x05c4ec50, EBP=0x, ESI=0x068b61e0, EDI=0x006f EIP=0x0001800057b2, EFLAGS=0x00010206 Top of Stack: (sp=0x05c4ec50) 0x05c4ec50: 05c4ecb0 01395ae2 0x05c4ec60: 06f3906f 0155f3ae 0x05c4ec70: 013959ce 013a1260 0x05c4ec80: 05c4ed00 006f 0x05c4ec90: 80c03178 006f 0x05c4eca0: 006f 01395ae2 0x05c4ecb0: 01395ae2 006f 0x05c4ecc0: 05c4ecc0 0x05c4ecd0: 05c4ed38 80c04790 0x05c4ece0: 80c03178 0x05c4ecf0: 05c4ed20 0x05c4ed00: 05c4ed80 013959ce 0x05c4ed10: 80c046d8 0139e316 0x05c4ed20: 006f 0x05c4ed30: 068b61e0 85a31fa0 0x05c4ed40: 05c4ed40 80f8778e Instructions: (pc=0x0001800057b2) 0x0001800057a2: 8d 44 24 38 48 89 44 24 38 48 8b 46 30 48 03 d3 0x0001800057b2: ff 50 40 85 c0 75 29 48 8b 44 24 38 48 85 c0 74 Stack: [0x05b5,0x05c5], sp=0x05c4ec50, free space=3fbk Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [tcnative-1.dll+0x57b2] [error occurred during error reporting (printing native stack), id 0xc005] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.Socket.sendbb(JII)I+0 j org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V+22 j org.apache.coyote.http11.InternalAprOutputBuffer.flush()V+5 J org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V j org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31 j org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+97 j org.apache.catalina.connector.OutputBuffer.flush()V+2 j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4 j javax.imageio.stream.FileCacheImageOutputStream.close()V+60 j javax.imageio.stream.ImageInputStreamImpl.finalize()V+8 v ~StubRoutines::call_stub j java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0 j java.lang.ref.Finalizer.runFinalizer()V+45 j java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1 j java.lang.ref.Finalizer$FinalizerThread.run()V+11 v ~StubRoutines::call_stub --- P R O C E S S --- Java Threads: ( => current thread ) 0x071c0800 JavaThread "http-apr-80-exec-10" daemon [_thread_blocked, id=4240, stack(0x0df5,0x0e05)] 0x071c JavaThread "http-apr-80-exec-9" daemon [_thread_blocked, id=4236, stack(0x0de5,0x0df5)] 0x07164000 JavaThread "http-apr-80-exec-8" daemon [_thread_blocked, id=4232, stack(0x0dd5,0x0de5)] 0x07163000 JavaThread "http-apr-80-exec-7" daemon [_thread_blocked, id=4228, stack(0x0dc5,0x0dd5)] 0x07162800 JavaThread "http-apr-80-exec-6" daemon [_thread_blocked, id=4224, stack(0x0db5,0x0dc5)] 0x07162000 JavaThread "http-apr-80-exec-5" daemon [_thread_blocked, id=4196, stack(0x0da5,0x0db5)] 0x07161000 JavaThread "http-apr-80-exec-4" daemon [_thread_blocked, id=4192, stack(0x0d95,0x0da5)] 0x07160800 JavaThread "http-apr-80-exec-3" daemon [_thread_
Aw: Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
> Gesendet: Freitag, 17. Mai 2013 um 14:26 Uhr > Von: "Mark Thomas" > > On 17/05/2013 12:31, Michael-O wrote: > > Hi Mark, > > > > thanks again for the detailed answer, details inline. > > > >> Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr > >> Von: "Mark Thomas" > >> An: "Tomcat Users List" > >> Betreff: Re: Follow-up: Possible false-postive with > >> JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > >> OracleTimeoutPollingThread > >> > >> On 17/05/2013 09:28, Michael-O wrote: > >>> Hi folks, > >>> > >>> there's now a follow-up on the issue [1]. > >>> Recap: JreMemoryLeakPreventionListener reported that my webapp has > >>> spawned OracleTimeoutPollingThread because I have set a QueryTimeout in > >>> the JDBC Pool. > >>> Mark Thomas noted that this is likely a bug in the driver. I have taken > >>> action and created a service request with Oracle. > >>> > >>> My personal analysis with VisualVM: > >>> The thread is spawned when the first query is run. The thread keeps > >>> running when the webapp is shut down or undeployed. Classes remain in > >>> memory. > >>> The JDBC Pool does a simple Class.forName to load the driver, it neither > >>> uses the DriverManager loading nor does it actively unload the driver. > >>> > >>> Oracle answer: > >>> They were able to reproduce the issue. The technical analysis says: > >>> > Hi Michael, > > I confirmned internally that this message from Tomcat can be ignored as > there is no risk of any real leak from" OracleTimeoutPollingThread" > thread. > This thread is related to the JDBC driver which may be used by many apps > simultaneously. Unloading the app does not unload the driver and should > not and does not stop the thread. > > It seems to be the design behaviour. > >>> > >>> So my questions would be: > >>> 1. Is that still not a false positive? > >> > >> No, that is not a false positive. The response from Oracle is wrong. > >> > >> There is nothing wrong with the driver creating the thread or the thread > >> continuing after the web application has stopped. The problem is as > >> follows: > >> > >> 1. When the JDBC driver creates the Thread, the Thread's context class > >> loader (as returned by Thread.getContextClassLoader()) is set to the web > >> application's class loader. > >> > >> 2. The correct behaviour at this point would be for the Driver to set > >> the Thread's context class loader to the class loader that loaded the > >> Driver class when the Thread is created. > >> > >> 3. The memory leak occurs as follows: > >> - the web application is stopped > >> - Tomcat clears all references to the web application and its classes > >> - The web application should be eligible for garbage collection > >> - The JDBC driver is still loaded (as it should be) > >> - The JDBC driver retains a reference to the Thread (as it should) > >> - The thread retains a reference to the web application class loader > >> (this is the memory leak). > >> > >> The reference chain is: > >> a) JDBC driver > >> b) Thread > >> c) Web application class loader > >> d) Every class loaded by the web app > >> > >> Everything from c) onwards should be eligible for garbage collection but > >> isn't because of the leak in the Thread that is retaining a reference to > >> the web application class loader. > > > > This is what I would assume as correct behavior. If the thread is still > > attached ot the WebAppClassLoader, does that mean that the > > WebAppClassLoader cannot be garbage collected? > > The correct behaviour is that there is no reference to c) held by b) and > therefore c) onwards would be eligible for GC (assuming the web app has > been stopped). The problem here is that because b) has a reference to > c), c) can't be GC'd and that is a memory leak. Exactly, WebAppClassLoader keeps in a stale status. > >>> 2. Why does the JDBC Pool not unload the driver? That my cause the thread > >>> to stop after the last app has been stopped/undeployed. > >> > >> 1. Because the driver is in CATALINA_HOME/lib it is available to other > >> applications. > > > > So first app, loads driver and keeps it in memory. Even if all apps are > > undeployed, driver remains cached and GCed when Tomcat is shut down? > > The JDBC driver will be loaded though the services API as soon as any > code references the DriverManager. > > >> 2. JDBC drivers are never automatically unloaded. You have to do so > >> explicitly (not doing so is an other source of memory leaks when the > >> driver is packaged in WEB-INF/lib). > > > > OK, this won't be the case. Driver is always shared here. > > > >> You need to go back to Oracle. > > > > Yes, I will. We're paying probably a lot of money for he company-wide > > license. > > I'd expect so. > > From my own experience with Oracle commercial support for things Java > related, you'll have to go through the "This is a bug. No it isn't. Yes > it is..." cycle several times before you
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
I am running it on Solaris x86 server. I haven't even changed my code much. Let me add a profiler,will post the results. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 17/05/2013 13:29, Chirag Dewan wrote: > Another important factor is the CPU utilization. Earlier for same trans/sec > it was 40% now its close to 80%. I'd upgrade to the latest 7.0.39 but otherwise, definitely time for a profiler. If you want raw performance on kept-alive connections then BIO is the best choice. Just for reference and so I can compare my own tests what hardware and OS are you running this on? > And yes,the OS,JVM and memory setting are unchanged. OK. That bit is good. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat hangs every day
I don't use HttpClient, directly, but if I'm not mistaking, javax.mail.jar (JavaMail) and Google Calendar API uses HttpClient, and I use JavaMail and Google Calendar API, directly, in my app. Google Calendar API seemed to always close their connections and I have even experienced downtime (host unknown and/or server error responses from Google Calendar API service), and I have experienced downtime (unknown host exceptions) when using JavaMail to connect to GMAIL (via IMAP connection) to check-for-and-download emails. Some months ago, I recognized that I had many unclosed connections on my Windows Server 2008 production server, where Tomcat was running my app. Instead of blame Tomcat, I listened in on Tomcat user list and I searched the internet, and learned that the unclosed connections were due to my inexperience (using JavaMail), so I refactored my code and closed my JavaMail IMAP connection when checking-and-downloading emails from GMAIL (google) servers. I just searched google, stackoverflow tomcat httpclient timeout[1] and found the following: Best Practice to Use HttpClient in Multithreaded Environment[2] Using Apache HttpClient how to set the TIMEOUT on a request and response[3] and many more search results (and related links) that you can read. [1] http://lmgtfy.com/?q=stackoverflow+tomcat+httpclient+timeout [2] http://stackoverflow.com/questions/1281219/best-practice-to-use-httpclient-in-multithreaded-environment [3] http://stackoverflow.com/questions/9873810/using-apache-httpclient-how-to-set-the-timeout-on-a-request-and-response On Fri, May 17, 2013 at 8:02 AM, Paolo Botta wrote: > Sorry for the question, but if I set the timeout on the HttpClient I force > the client to close the connection after the timeout, but have I informed > the server that there is a timeout, so the server close the connection > after > the timeout? I mean HttpClient tells to the server about the timeout? > > Thx > Paolo > > -Messaggio originale- > Da: chk...@gmail.com [mailto:chk...@gmail.com] Per conto di Christian > Kaltepoth > Inviato: venerdì 17 maggio 2013 13:30 > A: Tomcat Users List > Oggetto: Re: Tomcat hangs every day > > Hey, > > I'm also not an expert for HttpClient, but when creating connections to > remote services it is usually a good idea to set connection timeouts and > socket timeouts so that the client doesn't block forever if there are > problems with the connection. Exactly this seems to be the problem in your > case. > > Best regards > > Christian Kaltepoth > > > > > 2013/5/17 Sascha Troll > > > Christian, > > > > thanks for this. > > > > Can you give me a hint. I am just the server guy, so I can tell the > > developer. > > > > Thanks a lot ! > > > > Sascha > > > > > > > > From: Christian Kaltepoth > > To: Tomcat Users List > > Date: 17.05.2013 11:24 > > Subject:Re: Tomcat hangs every day > > Sent by:chk...@gmail.com > > > > > > > > Seems like you have a class called SearchClientRemoteClient which uses > > HTTPClient. Many threads seems to wait for remote responses. I guess > > you don't set any timeouts for the HTTPClient and therefore many > > threads hang in HttpClient.executeMethod() forever. You should ALWAYS > > set timeouts when using HTTPClient. :) > > > > I hope this helps :) > > > > Christian > > > > > > 2013/5/17 Sascha Troll > > > > > Hi ! > > > > > > I have problem with our Tomcat 7.0.40 (upgrade already done from > > > 7.0.39 and 7.0.37 and still the same issue). > > > > > > Its still running, but not longer accepting any connection on port > 8080. > > > > > > Attached are the thread dumps which were created this morning when > > > the server was not longer available. > > > I cannot find any deadlocks and need some help to find the cause. > > > > > > > > > > > > Thanks > > > Sascha > > > > > > > > > > > > > > > > -- > > > > > Disclaimer: > > > The content of this e-mail (including attachments) is confidential > > > and intended for the use of the addressee only. If you are not the > > > intended recipient please delete the e-mail; dissemination or > > > disclosure of its content to anyone is strictly prohibited! > > > Before opening an attachment please check it for viruses. We accept > > > no liability for any damage caused by viruses. > > > > > > > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > > > > -- > > Christian Kaltepoth > > Blog: http://blog.kaltepoth.de/ > > Twitter: http://twitter.com/chkal > > GitHub: https://github.com/chkal > > > > > > > > > > > > > > -- > > > > Disclaimer: > > The content of this e-mail (including attachments) is confidential and > > intende
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Another important factor is the CPU utilization. Earlier for same trans/sec it was 40% now its close to 80%. And yes,the OS,JVM and memory setting are unchanged. Thanks. Sent from Yahoo! Mail on Android
Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
On 17/05/2013 12:31, Michael-O wrote: > Hi Mark, > > thanks again for the detailed answer, details inline. > >> Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr >> Von: "Mark Thomas" >> An: "Tomcat Users List" >> Betreff: Re: Follow-up: Possible false-postive with >> JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and >> OracleTimeoutPollingThread >> >> On 17/05/2013 09:28, Michael-O wrote: >>> Hi folks, >>> >>> there's now a follow-up on the issue [1]. >>> Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned >>> OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC >>> Pool. >>> Mark Thomas noted that this is likely a bug in the driver. I have taken >>> action and created a service request with Oracle. >>> >>> My personal analysis with VisualVM: >>> The thread is spawned when the first query is run. The thread keeps running >>> when the webapp is shut down or undeployed. Classes remain in memory. >>> The JDBC Pool does a simple Class.forName to load the driver, it neither >>> uses the DriverManager loading nor does it actively unload the driver. >>> >>> Oracle answer: >>> They were able to reproduce the issue. The technical analysis says: >>> Hi Michael, I confirmned internally that this message from Tomcat can be ignored as there is no risk of any real leak from" OracleTimeoutPollingThread" thread. This thread is related to the JDBC driver which may be used by many apps simultaneously. Unloading the app does not unload the driver and should not and does not stop the thread. It seems to be the design behaviour. >>> >>> So my questions would be: >>> 1. Is that still not a false positive? >> >> No, that is not a false positive. The response from Oracle is wrong. >> >> There is nothing wrong with the driver creating the thread or the thread >> continuing after the web application has stopped. The problem is as follows: >> >> 1. When the JDBC driver creates the Thread, the Thread's context class >> loader (as returned by Thread.getContextClassLoader()) is set to the web >> application's class loader. >> >> 2. The correct behaviour at this point would be for the Driver to set >> the Thread's context class loader to the class loader that loaded the >> Driver class when the Thread is created. >> >> 3. The memory leak occurs as follows: >> - the web application is stopped >> - Tomcat clears all references to the web application and its classes >> - The web application should be eligible for garbage collection >> - The JDBC driver is still loaded (as it should be) >> - The JDBC driver retains a reference to the Thread (as it should) >> - The thread retains a reference to the web application class loader >> (this is the memory leak). >> >> The reference chain is: >> a) JDBC driver >> b) Thread >> c) Web application class loader >> d) Every class loaded by the web app >> >> Everything from c) onwards should be eligible for garbage collection but >> isn't because of the leak in the Thread that is retaining a reference to >> the web application class loader. > > This is what I would assume as correct behavior. If the thread is still > attached ot the WebAppClassLoader, does that mean that the WebAppClassLoader > cannot be garbage collected? The correct behaviour is that there is no reference to c) held by b) and therefore c) onwards would be eligible for GC (assuming the web app has been stopped). The problem here is that because b) has a reference to c), c) can't be GC'd and that is a memory leak. >>> 2. Why does the JDBC Pool not unload the driver? That my cause the thread >>> to stop after the last app has been stopped/undeployed. >> >> 1. Because the driver is in CATALINA_HOME/lib it is available to other >> applications. > > So first app, loads driver and keeps it in memory. Even if all apps are > undeployed, driver remains cached and GCed when Tomcat is shut down? The JDBC driver will be loaded though the services API as soon as any code references the DriverManager. >> 2. JDBC drivers are never automatically unloaded. You have to do so >> explicitly (not doing so is an other source of memory leaks when the >> driver is packaged in WEB-INF/lib). > > OK, this won't be the case. Driver is always shared here. > >> You need to go back to Oracle. > > Yes, I will. We're paying probably a lot of money for he company-wide license. I'd expect so. >From my own experience with Oracle commercial support for things Java related, you'll have to go through the "This is a bug. No it isn't. Yes it is..." cycle several times before you manage to get the issue in front of someone with the necessary knowledge and skills to correctly identify this is a bug in Oracle's JDBC driver. I'd recommend escalating it through your account manager sooner rather than later. Feel free to point Oracle support to this thread and/or my presentation on memory leaks. If Oracle support require further help understanding
R: Tomcat hangs every day
Sorry for the question, but if I set the timeout on the HttpClient I force the client to close the connection after the timeout, but have I informed the server that there is a timeout, so the server close the connection after the timeout? I mean HttpClient tells to the server about the timeout? Thx Paolo -Messaggio originale- Da: chk...@gmail.com [mailto:chk...@gmail.com] Per conto di Christian Kaltepoth Inviato: venerdì 17 maggio 2013 13:30 A: Tomcat Users List Oggetto: Re: Tomcat hangs every day Hey, I'm also not an expert for HttpClient, but when creating connections to remote services it is usually a good idea to set connection timeouts and socket timeouts so that the client doesn't block forever if there are problems with the connection. Exactly this seems to be the problem in your case. Best regards Christian Kaltepoth 2013/5/17 Sascha Troll > Christian, > > thanks for this. > > Can you give me a hint. I am just the server guy, so I can tell the > developer. > > Thanks a lot ! > > Sascha > > > > From: Christian Kaltepoth > To: Tomcat Users List > Date: 17.05.2013 11:24 > Subject:Re: Tomcat hangs every day > Sent by:chk...@gmail.com > > > > Seems like you have a class called SearchClientRemoteClient which uses > HTTPClient. Many threads seems to wait for remote responses. I guess > you don't set any timeouts for the HTTPClient and therefore many > threads hang in HttpClient.executeMethod() forever. You should ALWAYS > set timeouts when using HTTPClient. :) > > I hope this helps :) > > Christian > > > 2013/5/17 Sascha Troll > > > Hi ! > > > > I have problem with our Tomcat 7.0.40 (upgrade already done from > > 7.0.39 and 7.0.37 and still the same issue). > > > > Its still running, but not longer accepting any connection on port 8080. > > > > Attached are the thread dumps which were created this morning when > > the server was not longer available. > > I cannot find any deadlocks and need some help to find the cause. > > > > > > > > Thanks > > Sascha > > > > > > > > > > -- > > > Disclaimer: > > The content of this e-mail (including attachments) is confidential > > and intended for the use of the addressee only. If you are not the > > intended recipient please delete the e-mail; dissemination or > > disclosure of its content to anyone is strictly prohibited! > > Before opening an attachment please check it for viruses. We accept > > no liability for any damage caused by viruses. > > > > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > -- > Christian Kaltepoth > Blog: http://blog.kaltepoth.de/ > Twitter: http://twitter.com/chkal > GitHub: https://github.com/chkal > > > > > > > -- > > Disclaimer: > The content of this e-mail (including attachments) is confidential and > intended for the use of the addressee only. If you are not the > intended recipient please delete the e-mail; dissemination or > disclosure of its content to anyone is strictly prohibited! > Before opening an attachment please check it for viruses. We accept no > liability for any damage caused by viruses. > -- Christian Kaltepoth Blog: http://blog.kaltepoth.de/ Twitter: http://twitter.com/chkal GitHub: https://github.com/chkal - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
I am comparing tomcat 6.0.18 with tomcat 7.0.30. Would adding NIO or APR connector help? Currently I am using default(BIO) connector. Thanks. Sent from Yahoo! Mail on Android
Re: Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
Hi Mark, thanks again for the detailed answer, details inline. > Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr > Von: "Mark Thomas" > An: "Tomcat Users List" > Betreff: Re: Follow-up: Possible false-postive with > JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and > OracleTimeoutPollingThread > > On 17/05/2013 09:28, Michael-O wrote: > > Hi folks, > > > > there's now a follow-up on the issue [1]. > > Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned > > OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC > > Pool. > > Mark Thomas noted that this is likely a bug in the driver. I have taken > > action and created a service request with Oracle. > > > > My personal analysis with VisualVM: > > The thread is spawned when the first query is run. The thread keeps running > > when the webapp is shut down or undeployed. Classes remain in memory. > > The JDBC Pool does a simple Class.forName to load the driver, it neither > > uses the DriverManager loading nor does it actively unload the driver. > > > > Oracle answer: > > They were able to reproduce the issue. The technical analysis says: > > > >> Hi Michael, > >> > >> I confirmned internally that this message from Tomcat can be ignored as > >> there is no risk of any real leak from" OracleTimeoutPollingThread" thread. > >> This thread is related to the JDBC driver which may be used by many apps > >> simultaneously. Unloading the app does not unload the driver and should > >> not and does not stop the thread. > >> > >> It seems to be the design behaviour. > > > > So my questions would be: > > 1. Is that still not a false positive? > > No, that is not a false positive. The response from Oracle is wrong. > > There is nothing wrong with the driver creating the thread or the thread > continuing after the web application has stopped. The problem is as follows: > > 1. When the JDBC driver creates the Thread, the Thread's context class > loader (as returned by Thread.getContextClassLoader()) is set to the web > application's class loader. > > 2. The correct behaviour at this point would be for the Driver to set > the Thread's context class loader to the class loader that loaded the > Driver class when the Thread is created. > > 3. The memory leak occurs as follows: > - the web application is stopped > - Tomcat clears all references to the web application and its classes > - The web application should be eligible for garbage collection > - The JDBC driver is still loaded (as it should be) > - The JDBC driver retains a reference to the Thread (as it should) > - The thread retains a reference to the web application class loader > (this is the memory leak). > > The reference chain is: > a) JDBC driver > b) Thread > c) Web application class loader > d) Every class loaded by the web app > > Everything from c) onwards should be eligible for garbage collection but > isn't because of the leak in the Thread that is retaining a reference to > the web application class loader. This is what I would assume as correct behavior. If the thread is still attached ot the WebAppClassLoader, does that mean that the WebAppClassLoader cannot be garbage collected? > > 2. Why does the JDBC Pool not unload the driver? That my cause the thread > > to stop after the last app has been stopped/undeployed. > > 1. Because the driver is in CATALINA_HOME/lib it is available to other > applications. So first app, loads driver and keeps it in memory. Even if all apps are undeployed, driver remains cached and GCed when Tomcat is shut down? > 2. JDBC drivers are never automatically unloaded. You have to do so > explicitly (not doing so is an other source of memory leaks when the > driver is packaged in WEB-INF/lib). OK, this won't be the case. Driver is always shared here. > You need to go back to Oracle. Yes, I will. We're paying probably a lot of money for he company-wide license. Thanks again, Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 17/05/2013 11:51, Chirag Dewan wrote: > > > Hi All, > > I have upgraded my application from Embedded Tomcat 6 to Embedded > Tomcat 7. With Tomcat 6, I was getting around 7 Transactions per > sec with a simple HTTP service,which sets the HTTPResponse and > returns it. > > Now with Tomcat 7,I am getting 54000 transactions per sec. Going > through the web I figured out that it may be due to Servlet 3.0 API. > I have updated my web.xml so that Servlet API doesn't scan the > classes for annotations. This is how my web.xml looks like: Annotation scanning affects start time, not runtime performance. > Can anyone provide some other measures I can implement? Which Tomcat versions are you comparing? Is everything else the same (OS, JVM, etc.)? With trunk I get ~37k req/s with the default configuration and using ab as the client. Just changing maxKeepAliveRequests to -1 increases that to ~43k reqs/sec and all of that without shutting down the stuff I normally have running. There have been changes to the connector code that could add a few microseconds here and there to a request. That could explain the numbers you are seeing. Using a profiler would be the next step to see where the time is being spent. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat hangs every day
Hey, I'm also not an expert for HttpClient, but when creating connections to remote services it is usually a good idea to set connection timeouts and socket timeouts so that the client doesn't block forever if there are problems with the connection. Exactly this seems to be the problem in your case. Best regards Christian Kaltepoth 2013/5/17 Sascha Troll > Christian, > > thanks for this. > > Can you give me a hint. I am just the server guy, so I can tell the > developer. > > Thanks a lot ! > > Sascha > > > > From: Christian Kaltepoth > To: Tomcat Users List > Date: 17.05.2013 11:24 > Subject:Re: Tomcat hangs every day > Sent by:chk...@gmail.com > > > > Seems like you have a class called SearchClientRemoteClient which uses > HTTPClient. Many threads seems to wait for remote responses. I guess you > don't set any timeouts for the HTTPClient and therefore many threads hang > in HttpClient.executeMethod() forever. You should ALWAYS set timeouts when > using HTTPClient. :) > > I hope this helps :) > > Christian > > > 2013/5/17 Sascha Troll > > > Hi ! > > > > I have problem with our Tomcat 7.0.40 (upgrade already done from 7.0.39 > > and 7.0.37 and still the same issue). > > > > Its still running, but not longer accepting any connection on port 8080. > > > > Attached are the thread dumps which were created this morning when the > > server was not longer available. > > I cannot find any deadlocks and need some help to find the cause. > > > > > > > > Thanks > > Sascha > > > > > > > > > > -- > > Disclaimer: > > The content of this e-mail (including attachments) is confidential and > > intended for the use of the addressee only. If you are not the intended > > recipient please delete the e-mail; dissemination or disclosure of its > > content to anyone is strictly prohibited! > > Before opening an attachment please check it for viruses. We accept no > > liability for any damage caused by viruses. > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > -- > Christian Kaltepoth > Blog: http://blog.kaltepoth.de/ > Twitter: http://twitter.com/chkal > GitHub: https://github.com/chkal > > > > > > > -- > Disclaimer: > The content of this e-mail (including attachments) is confidential and > intended for the use of the addressee only. If you are not the intended > recipient please delete the e-mail; dissemination or disclosure of its > content to anyone is strictly prohibited! > Before opening an attachment please check it for viruses. We accept no > liability for any damage caused by viruses. > -- Christian Kaltepoth Blog: http://blog.kaltepoth.de/ Twitter: http://twitter.com/chkal GitHub: https://github.com/chkal
Re: Tomcat hangs every day
Christian, thanks for this. Can you give me a hint. I am just the server guy, so I can tell the developer. Thanks a lot ! Sascha From: Christian Kaltepoth To: Tomcat Users List Date: 17.05.2013 11:24 Subject:Re: Tomcat hangs every day Sent by:chk...@gmail.com Seems like you have a class called SearchClientRemoteClient which uses HTTPClient. Many threads seems to wait for remote responses. I guess you don't set any timeouts for the HTTPClient and therefore many threads hang in HttpClient.executeMethod() forever. You should ALWAYS set timeouts when using HTTPClient. :) I hope this helps :) Christian 2013/5/17 Sascha Troll > Hi ! > > I have problem with our Tomcat 7.0.40 (upgrade already done from 7.0.39 > and 7.0.37 and still the same issue). > > Its still running, but not longer accepting any connection on port 8080. > > Attached are the thread dumps which were created this morning when the > server was not longer available. > I cannot find any deadlocks and need some help to find the cause. > > > > Thanks > Sascha > > > > -- > Disclaimer: > The content of this e-mail (including attachments) is confidential and > intended for the use of the addressee only. If you are not the intended > recipient please delete the e-mail; dissemination or disclosure of its > content to anyone is strictly prohibited! > Before opening an attachment please check it for viruses. We accept no > liability for any damage caused by viruses. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Christian Kaltepoth Blog: http://blog.kaltepoth.de/ Twitter: http://twitter.com/chkal GitHub: https://github.com/chkal -- Disclaimer: The content of this e-mail (including attachments) is confidential and intended for the use of the addressee only. If you are not the intended recipient please delete the e-mail; dissemination or disclosure of its content to anyone is strictly prohibited! Before opening an attachment please check it for viruses. We accept no liability for any damage caused by viruses.
Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Hi All, I have upgraded my application from Embedded Tomcat 6 to Embedded Tomcat 7. With Tomcat 6, I was getting around 7 Transactions per sec with a simple HTTP service,which sets the HTTPResponse and returns it. Now with Tomcat 7,I am getting 54000 transactions per sec. Going through the web I figured out that it may be due to Servlet 3.0 API. I have updated my web.xml so that Servlet API doesn't scan the classes for annotations. This is how my web.xml looks like: http://java.sun.com/xml/ns/javaee"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"; version="3.0"> Can anyone provide some other measures I can implement? I am using an HTTP Client to test my performnace. Thanks
Re: IOException from the response during an async request
On 15/05/2013 20:35, Rossen Stoyanchev wrote: > Mark, > > While there is no response on the Servlet spec list, I have some follow-up > questions: > >> If you call complete() you'll end up with an IllegalStateException when >> the error handler kicks in. You have to use dispatch(). The error handle >> kicks in as soon as the dispatch() tries to write to the response. > > Do you mean dispatch() + write to the response from within the dispatched > thread thereby causing an error? I actually have no reason to write to the > response at that point. I'm just trying to complete the async request and > release any associated Servlet container resources. Yes, the target of the dispatch() has to try and write something so the error handler kicks in. > I did try dispatch() and then complete() from within the dispatched thread. > It leads to a similar ISE from AsyncStateMachine. You shouldn't need to call complete(). Just try and write something and let the container handle the resulting error. > Lastly if I do call complete() without dispatch, thereby causing the ISE, > does that actually complete the async request and release any container > resources? I'm just trying to decide if this is something I can live with > for now while the EG decides. I haven't tested it but I don't believe it will. Something needs to happen to get the request back on a container thread. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Maximum instances in Tomcat cluster
On 17/05/2013 10:08, Soumya Chatterjee wrote: > Hi, > > I am using Apache Tomcat (version: 6.0.35) with horizontal clustering > (static membership) for hosting my web application. > > I want to know if there is any upper limit of instances that Tomcat > supports in order to maintain its optimum performance. (I searched the > Tomcat clustering documentation but could not find any information > regarding this). The upper limit to the number of cluster members is Integer.MAX_VALUE I suspect you'll hit other limits before then. Which limits, and when you hit them, is highly dependent on your application and cluster configuration. Optimum performance depends on how you define it. If you mean highest throughput and fastest response time then the optimum number of cluster nodes is 1. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
On 17/05/2013 09:28, Michael-O wrote: > Hi folks, > > there's now a follow-up on the issue [1]. > Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned > OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC Pool. > Mark Thomas noted that this is likely a bug in the driver. I have taken > action and created a service request with Oracle. > > My personal analysis with VisualVM: > The thread is spawned when the first query is run. The thread keeps running > when the webapp is shut down or undeployed. Classes remain in memory. > The JDBC Pool does a simple Class.forName to load the driver, it neither uses > the DriverManager loading nor does it actively unload the driver. > > Oracle answer: > They were able to reproduce the issue. The technical analysis says: > >> Hi Michael, >> >> I confirmned internally that this message from Tomcat can be ignored as >> there is no risk of any real leak from" OracleTimeoutPollingThread" thread. >> This thread is related to the JDBC driver which may be used by many apps >> simultaneously. Unloading the app does not unload the driver and should not >> and does not stop the thread. >> >> It seems to be the design behaviour. > > So my questions would be: > 1. Is that still not a false positive? No, that is not a false positive. The response from Oracle is wrong. There is nothing wrong with the driver creating the thread or the thread continuing after the web application has stopped. The problem is as follows: 1. When the JDBC driver creates the Thread, the Thread's context class loader (as returned by Thread.getContextClassLoader()) is set to the web application's class loader. 2. The correct behaviour at this point would be for the Driver to set the Thread's context class loader to the class loader that loaded the Driver class when the Thread is created. 3. The memory leak occurs as follows: - the web application is stopped - Tomcat clears all references to the web application and its classes - The web application should be eligible for garbage collection - The JDBC driver is still loaded (as it should be) - The JDBC driver retains a reference to the Thread (as it should) - The thread retains a reference to the web application class loader (this is the memory leak). The reference chain is: a) JDBC driver b) Thread c) Web application class loader d) Every class loaded by the web app Everything from c) onwards should be eligible for garbage collection but isn't because of the leak in the Thread that is retaining a reference to the web application class loader. > 2. Why does the JDBC Pool not unload the driver? That my cause the thread to > stop after the last app has been stopped/undeployed. 1. Because the driver is in CATALINA_HOME/lib it is available to other applications. 2. JDBC drivers are never automatically unloaded. You have to do so explicitly (not doing so is an other source of memory leaks when the driver is packaged in WEB-INF/lib). You need to go back to Oracle. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat hangs every day
Seems like you have a class called SearchClientRemoteClient which uses HTTPClient. Many threads seems to wait for remote responses. I guess you don't set any timeouts for the HTTPClient and therefore many threads hang in HttpClient.executeMethod() forever. You should ALWAYS set timeouts when using HTTPClient. :) I hope this helps :) Christian 2013/5/17 Sascha Troll > Hi ! > > I have problem with our Tomcat 7.0.40 (upgrade already done from 7.0.39 > and 7.0.37 and still the same issue). > > Its still running, but not longer accepting any connection on port 8080. > > Attached are the thread dumps which were created this morning when the > server was not longer available. > I cannot find any deadlocks and need some help to find the cause. > > > > Thanks > Sascha > > > > -- > Disclaimer: > The content of this e-mail (including attachments) is confidential and > intended for the use of the addressee only. If you are not the intended > recipient please delete the e-mail; dissemination or disclosure of its > content to anyone is strictly prohibited! > Before opening an attachment please check it for viruses. We accept no > liability for any damage caused by viruses. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Christian Kaltepoth Blog: http://blog.kaltepoth.de/ Twitter: http://twitter.com/chkal GitHub: https://github.com/chkal
Maximum instances in Tomcat cluster
Hi, I am using Apache Tomcat (version: 6.0.35) with horizontal clustering (static membership) for hosting my web application. I want to know if there is any upper limit of instances that Tomcat supports in order to maintain its optimum performance. (I searched the Tomcat clustering documentation but could not find any information regarding this). Thanks, Soumya
Re: How to set the timeout for context initializing/startup
Sven Schönfeldt wrote: Hi Tomcat-Users I have found no information in the documentation, to configure the timeout for initializing/startup a context. there is the possibility? and if, how to configure? If I remember correctly, this question came up already in the last couple of days, and was answered. Check the list archives. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
How to set the timeout for context initializing/startup
Hi Tomcat-Users I have found no information in the documentation, to configure the timeout for initializing/startup a context. there is the possibility? and if, how to configure? Regards, Sven - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread
Hi folks, there's now a follow-up on the issue [1]. Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC Pool. Mark Thomas noted that this is likely a bug in the driver. I have taken action and created a service request with Oracle. My personal analysis with VisualVM: The thread is spawned when the first query is run. The thread keeps running when the webapp is shut down or undeployed. Classes remain in memory. The JDBC Pool does a simple Class.forName to load the driver, it neither uses the DriverManager loading nor does it actively unload the driver. Oracle answer: They were able to reproduce the issue. The technical analysis says: > Hi Michael, > > I confirmned internally that this message from Tomcat can be ignored as there > is no risk of any real leak from" OracleTimeoutPollingThread" thread. > This thread is related to the JDBC driver which may be used by many apps > simultaneously. Unloading the app does not unload the driver and should not > and does not stop the thread. > > It seems to be the design behaviour. So my questions would be: 1. Is that still not a false positive? 2. Why does the JDBC Pool not unload the driver? That my cause the thread to stop after the last app has been stopped/undeployed. Thanks, Michael [1] http://www.mail-archive.com/users@tomcat.apache.org/msg106186.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
