On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny <theo.swe...@avios.com> wrote: >Hello - yesterday Tomcat 8 went into a hung state, because an external >API went offline. > > >Eventually the external service was restored but Tomcat was unable to >recovery from it. > > >When a new connection is sent to Tomcat the connection hangs as no >additional thread can be allocated to the new request. > > >Here is an example of the log - > > >17-Mar-2016 15:53:36.671 WARNING >[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]] >org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected >Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717 >milliseconds (since 3/17/16 3:53 PM) to serve the same request for >http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck >(configured threshold for this StuckThreadDetectionValve is 30 >seconds). There is/are 1 thread(s) in total that are monitored by this >Valve and may be stuck. > java.lang.Throwable > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425) > at java.util.concurrent.FutureTask.get(FutureTask.java:187) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104) >at >com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86) >at >com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke(<generated>) >at >org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) >at >org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) >at >org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) >at >org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) >at >com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices(<generated>) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48) >at >com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74) >at >com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70) > at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source) >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) >at >org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) >at >org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) >at >org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706) >at >org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) >at >org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943) >at >org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877) >at >org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) >at >org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) >at >org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) >at >org.springframework.boot.context.web.ErrorPageFilter.access$000(ErrorPageFilter.java:59) >at >org.springframework.boot.context.web.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:88) >at >org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >at >org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101) >at >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) >at >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >at >org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) >at >org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) >at >org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) >at >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) >at >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) >at >org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) >at >org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208) >at >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) >at >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518) >at >org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) >at >org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668) >at >org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223) >at >org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1517) >at >org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1474) >at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >at >org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > > >Are there any ideas as to why Tomcat is unable to free up its threads?
That is an application problem. Nothing to do with Tomcat code. Once a thread enters application code Tomcat has no control over it. Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
