Hi Mark, ________________________________________ From: Mark Thomas <ma...@apache.org> Sent: 17 March 2016 17:36 To: Tomcat Users List Subject: Re: Tomcat 8 Hung State
On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny <theo.swe...@avios.com> wrote: >Hello - yesterday Tomcat 8 went into a hung state, because an external >API went offline. > > >Eventually the external service was restored but Tomcat was unable to >recovery from it. > > >When a new connection is sent to Tomcat the connection hangs as no >additional thread can be allocated to the new request. > > >Here is an example of the log - > > >17-Mar-2016 15:53:36.671 WARNING >[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]] >org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected >Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717 >milliseconds (since 3/17/16 3:53 PM) to serve the same request for >http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck >(configured threshold for this StuckThreadDetectionValve is 30 >seconds). There is/are 1 thread(s) in total that are monitored by this >Valve and may be stuck. > java.lang.Throwable > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425) > at java.util.concurrent.FutureTask.get(FutureTask.java:187) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104) >at >com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86) >at >com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke(<generated>) >at >org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) >at >org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) >at >org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) >at >org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) >at >com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices(<generated>) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48) >at >com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74) >at >com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70) > at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source) >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) >at >org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) >at >org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) >at >org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706) >at >org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) >at >org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943) >at >org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877) >at >org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) >at >org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) >at >org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) >at >org.springframework.boot.context.web.ErrorPageFilter.access$000(ErrorPageFilter.java:59) >at >org.springframework.boot.context.web.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:88) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) >at >org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) >at >org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) >at >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) >at >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) >at >org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) >at >org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208) >at >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) >at >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518) >at >org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) >at >org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668) >at >org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223) >at >org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1517) >at >org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1474) >at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >at >org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > > >Are there any ideas as to why Tomcat is unable to free up its threads? That is an application problem. Nothing to do with Tomcat code. Once a thread enters application code Tomcat has no control over it. Mark Thanks for the reply - the developers are pushing back as they believe Tomcat should kill off any rogue webapps after a certain timeframe as this is how they have it done in Weblogic.When I do an Strace on a broken webapp, the following can be seen - 11297 11:39:23.023838 write(105, "2016-03-18 11:39:23,021 138066033 [http-nio-10080-exec-27] ERROR com.spring.business.service.security.ldap.filter.CustomAuthFilter - Exception thrown in doFilterInternal: {}\norg.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393) ~[catalina.jar:8.0.21]\n\tat org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426) ~[tomcat-util.jar:8.0.21]\n\tat org.apache.catalina.connector."..., 17236 <unfinished ...> Is this due to Tomcat's inability to recover from a lost datasource? Regards, Theo Avios Group (AGL) Ltd is a limited company registered in England (registered number 2260073 and VAT number 512566754) whose registered address is Astral Towers, Betts Way, London Road, Crawley, West Sussex RH10 9XY . Avios Group (AGL) Limited is part of the IAG group of companies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
