> -----Original Message----- > From: Bob DeRemer [mailto:[email protected]] > Sent: Friday, September 06, 2013 9:45 AM > To: Tomcat Users List > Subject: RE: (7.0.40) Seeing deadlock followed by maxThread usage using a > Async Servlet AsyncContext.dispatch with BIO > > > > > -----Original Message----- > > From: Mark Thomas [mailto:[email protected]] > > Sent: Friday, September 06, 2013 9:25 AM > > To: [email protected] > > Subject: Re: (7.0.40) Seeing deadlock followed by maxThread usage > > using a Async Servlet AsyncContext.dispatch with BIO > > > > On 06/09/2013 13:28, Daniel Mikusa wrote: > > > On Sep 5, 2013, at 3:30 PM, Bob DeRemer <[email protected]> > > wrote: > > > > > >> I'm trying to determine if there are any known issues using an > > >> Async servlet > > with the default BIO connector in Tomcat 7.0.40 on windows server 2008. > > >> > > >> PROBLEM: > > >> We have a situation where we're occasionally getting a deadlock > > >> (according > > to Java Melody). When this happens, it's just a matter of time until > > all the threads in the connector's internal pool are exhausted because > > browser-based clients continue to make calls on this async servlet - > > effectively polling for work. > > >> > > >> QUESTION: > > >> I'm in the process of going through the stacks and code, but I'm > > >> not well > > versed in the async servlet code (which is mostly from the Vysper > > project), so I'm trying to figure out if the underlying > > AsyncContext.dispatch usage is a problem [at least when used with BIO]. > > >> > > >> Given the 3 stacks below, I'd greatly appreciate it if someone who > > understands the Async servlet plumbing might be able to take a quick > > look and see if anything stands out that could be the cause. If you > > need any additional information, please let me know. > > >> > > >> Thanks, > > >> Bob > > > > > > Is there any chance you could get a full thread dump while the > > > problem is > > occurring? The thread dump below doesn't have any information about > > what locks are being held by the blocked threads. I think that might > > help to see why each thread is blocking. > > > > +1. > > > > What I would say is that there is no async code involved in the > > http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect > > the vysper code. > > > > The only locks I'd expect Tomcat to be holding are on the socket to > > ensure that only one thread processes a connection at a time and I'm > > pretty such there is no opportunity for deadlock there. > > > > A closer look at BoshBackedSessionContext is probably in order. > > > > Mark > > > > Thanks for the suggestions! I have already found the problem, which (as you > suspected), is in the BoshBackedSessionContext. There is a potential deadlock > between the BoshBackedSessionContext and the underlying JIoEndpoint. It's > happening because the Vypser code has 2 methods using "synchronized": > > - 1 thd is doing an AsyncContext.dispatch from a synchronized method of > BoshBackedSessionContext which hangs waiting for the JIoEndpoint > synchronized socket wrapper in processSocketAsync > - the other thd is the async timeout in JIoEndpoint firing back into the > BoshBackedSessionContext; he acquired the socket wrapper sync and is > attempting to a different synchronized method on the same > BoshBackedSessionContext > > So, I'm going to remove the synchronized usage from the > BoshBackedSessionContext and replace with a Lock/Condition using a tryLock > with timeout and 1 retry. I've used this approach before and it's typically > a lot > more performant and provides better control over situations like this. > > Thx, bob
CORRECTION: only a Lock is needed for this fix, no Condition; so accustomed to using them together > > > > > > > Dan > > > > > > > > >> > > >> DEADLOCKED STACK INFORMATION > > >> > > >> Warning, the following threads are deadlocked : > > >> http-bio-192.168.230.133-443-exec-1329, > > >> http-bio-192.168.230.133-443-exec-685, > > >> http-bio-192.168.230.133-443-exec-687 > > >> > > >> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED > > >> > > org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint. > > java:55 > > 6) > > >> > > > org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java: > > 365) > > >> > > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 > > Proce > > ssor.java:848) > > >> org.apache.coyote.Request.action(Request.java:344) > > >> > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja > > va:22 > > 2) > > >> > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja > > va:18 > > 1) > > >> > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja > > va:17 > > 5) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.writ > > e0( > > BoshBackedSessionContext.java:251) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession( > > Bos > > hHandler.java:237) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(Bo > > shH > > andler.java:228) > > >> > > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler. > > java:192) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitSta > > nz > > a(BoshSaxContentHandler.java:90) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElem > > e > > nt(BoshSaxContentHandler.java:78) > > >> > > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480 > > ) > > >> > > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > > >> > > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154 > > ) > > >> > > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:10 > > 8) > > >> > > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > > >> > > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(Defau > > ltN > > onBlockingXMLReader.java:185) > > >> > > > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecode > > r > > .java:81) > > >> > > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPo > > st( > > ThingworxBoshServlet.java:176) > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:305) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > > >> > > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java > > >> :179) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:243) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Au > > thentica > > tionFilter.java:279) > > >> > > com.thingworx.security.filter.AuthenticationFilter.doFilter(Authentica > > tionFilter.j > > ava:150) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:243) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > > lv > > e.java:222) > > >> > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > > lve.j > > ava:123) > > >> > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat > > orBase > > .java:581) > > >> > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > > va:17 > > 1) > > >> > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > > va:99) > > >> > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:9 > > 53) > > >> > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv > > e.jav > > a:118) > > >> > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > > :408) > > >> > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp1 > > 1Proc > > essor.java:1008) > > >> > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A > > bstra > > ctProtocol.java:589) > > >> > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint > > .java:3 > > 12) > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > >> java.lang.Thread.run(Unknown Source) > > >> > > >> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requ > > est > > Expired(BoshBackedSessionContext.java:479) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.acce > > ss$ > > 000(BoshBackedSessionContext.java:49) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.on > > Ti > > meout(BoshBackedSessionContext.java:669) > > >> > > org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListe > > ner > > Wrapper.java:45) > > >> > > org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.jav > > a:151 > > ) > > >> > > org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapte > > r.ja > > va:293) > > >> > > org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(Abstrac > > tHttp > > 11Processor.java:1568) > > >> > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A > > bstra > > ctProtocol.java:583) > > >> > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint > > .java:3 > > 12) > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > >> java.lang.Thread.run(Unknown Source) > > >> > > >> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED > > >> > > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler. > > java:160) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitSta > > nz > > a(BoshSaxContentHandler.java:90) > > >> > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElem > > e > > nt(BoshSaxContentHandler.java:78) > > >> > > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480 > > ) > > >> > > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > > >> > > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154 > > ) > > >> > > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:10 > > 8) > > >> > > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > > >> > > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(Defau > > ltN > > onBlockingXMLReader.java:185) > > >> > > > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecode > > r > > .java:81) > > >> > > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPo > > st( > > ThingworxBoshServlet.java:176) > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:305) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > > >> > > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java > > >> :179) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:243) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Au > > thentica > > tionFilter.java:279) > > >> > > com.thingworx.security.filter.AuthenticationFilter.doFilter(Authentica > > tionFilter.j > > ava:150) > > >> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > cationFilt > > erChain.java:243) > > >> > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > > java:210) > > >> > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > > lv > > e.java:222) > > >> > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > > lve.j > > ava:123) > > >> > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat > > orBase > > .java:581) > > >> > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > > va:17 > > 1) > > >> > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > > va:99) > > >> > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:9 > > 53) > > >> > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv > > e.jav > > a:118) > > >> > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > > :408) > > >> > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp1 > > 1Proc > > essor.java:1008) > > >> > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A > > bstra > > ctProtocol.java:589) > > >> > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint > > .java:3 > > 10) > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > >> java.lang.Thread.run(Unknown Source) > > >> > > >> > > >> Bob DeRemer > > >> Senior Director, Architecture and Development > > >> > > >> > > >> http://www.thingworx.com > > >> Skype: bob.deremer.thingworx > > >> O: 610.594.6200 x812 > > >> M: 717.881.3986 > > >> > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
