I'm trying to determine if there are any known issues using an Async servlet
with the default BIO connector in Tomcat 7.0.40 on windows server 2008.
PROBLEM:
We have a situation where we're occasionally getting a deadlock (according to
Java Melody). When this happens, it's just a matter of time until all the
threads in the connector's internal pool are exhausted because browser-based
clients continue to make calls on this async servlet - effectively polling for
work.
QUESTION:
I'm in the process of going through the stacks and code, but I'm not well
versed in the async servlet code (which is mostly from the Vysper project), so
I'm trying to figure out if the underlying AsyncContext.dispatch usage is a
problem [at least when used with BIO].
Given the 3 stacks below, I'd greatly appreciate it if someone who understands
the Async servlet plumbing might be able to take a quick look and see if
anything stands out that could be the cause. If you need any additional
information, please let me know.
Thanks,
Bob
DEADLOCKED STACK INFORMATION
Warning, the following threads are deadlocked :
http-bio-192.168.230.133-443-exec-1329, http-bio-192.168.230.133-443-exec-685,
http-bio-192.168.230.133-443-exec-687
"http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556)
org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365)
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848)
org.apache.coyote.Request.action(Request.java:344)
org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222)
org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181)
org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175)
org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251)
org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237)
org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228)
org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192)
org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
"http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479)
org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49)
org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669)
org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45)
org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151)
org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293)
org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
"http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160)
org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
Bob DeRemer
Senior Director, Architecture and Development
[Description: Description: Description: Description:
cid:image001.png@01CBE3DE.51A12030]
http://www.thingworx.com<http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986
