Hello there. We have been running a reverse proxy based on Jetty's AsyncMiddleManServlet for years without any major issue. However, we started noticing that once in a great while a thread gets stuck in a hot loop. The only way to recover it is to bounce the application.
"Hot" part of the code seems to be around these methods calls (this was verified via JMX CPU sampling) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117) I've taken heap dumps and thread-dumps but don't seem to be able to figure out what is going on. As of now, this is not a concern for us, given the frequency is extremely low (a stuck thread a week). Just want to reach out to the group. Any suggestions would be appreciated. Thanks, -Lu Details: Jetty version: 9.4.14.v20181114 JDK version: 10.0.2 Comlete stack traces: Incident #1 2019-04-30 17:18:55 "default-pool-36369" - Thread t@36369 java.lang.Thread.State: RUNNABLE at app//org.eclipse.jetty.server.HttpConnection$SendCallback.access$300(HttpConnection.java:678) at app//org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:536) at app//org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:841) at app//org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:891) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:398) at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:244) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:105) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) at app//org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:298) at app//org.eclipse.jetty.server.Response.closeOutput(Response.java:1044) at app//org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:507) at app//org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305) at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:109) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at [email protected]/java.lang.Thread.run(Thread.java:844) Locked ownable synchronizers: - None Incident #2 2019-04-22 13:41:59 "default-pool-76" - Thread t@76 java.lang.Thread.State: RUNNABLE at app//org.eclipse.jetty.util.thread.Locker.lock(Locker.java:46) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:195) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) at app//org.eclipse.jetty.server.HttpOutput$AsyncWrite.process(HttpOutput.java:1243) at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:471) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.write(AsyncMiddleManServlet.java:219) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.writeProxyResponseContent(AsyncMiddleManServlet.java:201) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyWriter.onWritePossible(AsyncMiddleManServlet.java:651) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyResponseListener.onContent(AsyncMiddleManServlet.java:499) at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:133) at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:125) at app//org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:338) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:283) at app//org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1787) at app//org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) at app//org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) at app//org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) at app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at app//org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:110) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at [email protected]/java.lang.Thread.run(Thread.java:844) Locked ownable synchronizers: - locked <7ee79395> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) Incident #3 2019-04-17 10:03:03 "default-pool-74" - Thread t@74 java.lang.Thread.State: RUNNABLE at [email protected] /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1237) at [email protected] /java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267) at app//org.eclipse.jetty.util.thread.Locker.lock(Locker.java:46) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:195) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) at app//org.eclipse.jetty.server.HttpOutput$AsyncWrite.process(HttpOutput.java:1243) at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:471) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.write(AsyncMiddleManServlet.java:219) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.writeProxyResponseContent(AsyncMiddleManServlet.java:201) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyWriter.onWritePossible(AsyncMiddleManServlet.java:651) at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyResponseListener.onContent(AsyncMiddleManServlet.java:499) at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:133) at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:125) at app//org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:338) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:283) at app//org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1787) at app//org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) at app//org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) at app//org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) at app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at app//org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:110) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at [email protected]/java.lang.Thread.run(Thread.java:844) Locked ownable synchronizers: - locked <76e220c> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) Incident #4 2019-04-30 17:16:43 "default-pool-109866" - Thread t@109866 java.lang.Thread.State: BLOCKED at [email protected]/java.util.zip.Deflater.needsInput(Deflater.java:340) - waiting to lock <876a793> (a java.util.zip.Deflater$DeflaterZStreamRef) owned by "default-pool-36369" t@36369 at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:327) at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:244) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:105) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) at app//org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:298) at app//org.eclipse.jetty.server.Response.closeOutput(Response.java:1044) at app//org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:507) at app//org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305) at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:109) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at [email protected]/java.lang.Thread.run(Thread.java:844) Locked ownable synchronizers: - None 2019-04-30 17:18:19 "default-pool-109866" - Thread t@109866 java.lang.Thread.State: RUNNABLE at [email protected]/java.util.zip.Deflater.deflateBytes(Native Method) at [email protected]/java.util.zip.Deflater.deflate(Deflater.java:466) - locked <876a793> (a java.util.zip.Deflater$DeflaterZStreamRef) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:390) at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:244) at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:105) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) at app//org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:298) at app//org.eclipse.jetty.server.Response.closeOutput(Response.java:1044) at app//org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:507) at app//org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305) at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:109) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at [email protected]/java.lang.Thread.run(Thread.java:844) Locked ownable synchronizers: - None
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
