Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 12:47 PM Mark Thomas wrote: > On 06/10/2020 10:35, Martin Grigorov wrote: > > On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov > > wrote: > > > >> > >> > >> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas wrote: > >> > >>> On 06/10/2020 07:30, Martin Grigorov wrote: > Hi, > > I face an issue with the NIO2 protocol. > When I increase the request rate to more than 500 requests per second > it > starts failing with: > >>> > >>> I'm unable to reproduce this with a 10.0.x build requesting a simple > >>> text file from the ROOT web app using NIO2 and Java 1.8.0_265. > >>> > >>> I see around 15k req/s and no failures. > >>> > >> > >> I use latest 9.0.x. > >> Let me try with non-embedded Tomcat! > >> > > > > Same build of Tomcat started with ./bin/startup.sh has no such problems! > > It seems > > > https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java > > either misses something or does something more than what conf/server.xml > > does. > > I've tried to minify server.xml by removing all listeners, valves and > JNDI > > related stuff but it still works fine. > > maxThreads="8" setting? That looks very low to me. > Yep! This was the reason! Thanks! The EOFs happen with standalone Tomcat too when maxThreads is 8. I've reduced it because otherwise the throughput of NIO is lower. Async Profiler (perf) says that there are many context switches. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: A lot of EOFs when using NIO2 with HTTP2
On 06/10/2020 10:35, Martin Grigorov wrote: > On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov > wrote: > >> >> >> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas wrote: >> >>> On 06/10/2020 07:30, Martin Grigorov wrote: Hi, I face an issue with the NIO2 protocol. When I increase the request rate to more than 500 requests per second it starts failing with: >>> >>> I'm unable to reproduce this with a 10.0.x build requesting a simple >>> text file from the ROOT web app using NIO2 and Java 1.8.0_265. >>> >>> I see around 15k req/s and no failures. >>> >> >> I use latest 9.0.x. >> Let me try with non-embedded Tomcat! >> > > Same build of Tomcat started with ./bin/startup.sh has no such problems! > It seems > https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java > either misses something or does something more than what conf/server.xml > does. > I've tried to minify server.xml by removing all listeners, valves and JNDI > related stuff but it still works fine. maxThreads="8" setting? That looks very low to me. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov wrote: > > > On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas wrote: > >> On 06/10/2020 07:30, Martin Grigorov wrote: >> > Hi, >> > >> > I face an issue with the NIO2 protocol. >> > When I increase the request rate to more than 500 requests per second it >> > starts failing with: >> >> I'm unable to reproduce this with a 10.0.x build requesting a simple >> text file from the ROOT web app using NIO2 and Java 1.8.0_265. >> >> I see around 15k req/s and no failures. >> > > I use latest 9.0.x. > Let me try with non-embedded Tomcat! > Same build of Tomcat started with ./bin/startup.sh has no such problems! It seems https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java either misses something or does something more than what conf/server.xml does. I've tried to minify server.xml by removing all listeners, valves and JNDI related stuff but it still works fine. > > >> >> Mark >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 12:13 PM Rémy Maucherat wrote: > On Tue, Oct 6, 2020 at 8:31 AM Martin Grigorov > wrote: > > > Hi, > > > > I face an issue with the NIO2 protocol. > > When I increase the request rate to more than 500 requests per second it > > starts failing with: > > > > 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6] > > org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State > > [NEW] > > 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6] > > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed > > Connection [1], Stream [0], Frame type [null], Error > > java.io.EOFException > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1005) > > > > Also, it is a real EOF as the error says, caused by the client > disconnecting for whatever reason (good or bad). > The client is exactly the same for Tomcat NIO/NIO2/APR, Golang, Node.js, Rust and Netty. It works for all but NIO2. > > Rémy >
Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas wrote: > On 06/10/2020 07:30, Martin Grigorov wrote: > > Hi, > > > > I face an issue with the NIO2 protocol. > > When I increase the request rate to more than 500 requests per second it > > starts failing with: > > I'm unable to reproduce this with a 10.0.x build requesting a simple > text file from the ROOT web app using NIO2 and Java 1.8.0_265. > > I see around 15k req/s and no failures. > I use latest 9.0.x. Let me try with non-embedded Tomcat! > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 8:31 AM Martin Grigorov wrote: > Hi, > > I face an issue with the NIO2 protocol. > When I increase the request rate to more than 500 requests per second it > starts failing with: > > 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6] > org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State > [NEW] > 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6] > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed > Connection [1], Stream [0], Frame type [null], Error > java.io.EOFException > at > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1005) > Also, it is a real EOF as the error says, caused by the client disconnecting for whatever reason (good or bad). Rémy
Re: A lot of EOFs when using NIO2 with HTTP2
On 06/10/2020 07:30, Martin Grigorov wrote: > Hi, > > I face an issue with the NIO2 protocol. > When I increase the request rate to more than 500 requests per second it > starts failing with: I'm unable to reproduce this with a 10.0.x build requesting a simple text file from the ROOT web app using NIO2 and Java 1.8.0_265. I see around 15k req/s and no failures. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: A lot of EOFs when using NIO2 with HTTP2
On Tue, Oct 6, 2020 at 9:30 AM Martin Grigorov wrote: > Hi, > > I face an issue with the NIO2 protocol. > When I increase the request rate to more than 500 requests per second it > starts failing with: > > 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6] > org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW] > 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6] > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed > Connection [1], Stream [0], Frame type [null], Error > java.io.EOFException > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > at > java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > at > org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126) > at > org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > at > java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) > at > org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124) > at > org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025) > at > org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004) > at > org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450) > at > org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293) > at > org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265) > at > org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55) > at > org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241) > at > org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40) > at > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316) > at > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40) > at > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) > at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) > at > org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679) > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > at > org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104) > at > org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99) > at > org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219) > at > java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.base/java.lang.Thread.run(Thread.java:832) > 06-Oct-2020 09:18:19.980 FINE [https-jsse-nio2-8080-exec-6] > org.apache.coyote.http2.Http2UpgradeHandl
A lot of EOFs when using NIO2 with HTTP2
Hi, I face an issue with the NIO2 protocol. When I increase the request rate to more than 500 requests per second it starts failing with: 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6] org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW] 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6] org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed Connection [1], Stream [0], Frame type [null], Error java.io.EOFException at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005) at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) at org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126) at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054) at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112) at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038) at org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001) at org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124) at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025) at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004) at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450) at org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293) at org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265) at org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55) at org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241) at org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40) at org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316) at org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104) at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99) at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92) at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219) at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:832) 06-Oct-2020 09:18:19.980 FINE [https-jsse-nio2-8080-exec-6] org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Entry, Connection [1], SocketStatus [ERROR] If I just change the protocol to org.apache.coyote.http11.Http11NioProtocol then everything is OK. To reproduce: 1) git clone https://github.com/martin-g/http2-server-perf-tests.git 2) cd
