http://issues.apache.org/bugzilla/show_bug.cgi?id=44502
Summary: NIO connector under heavy load drops incoming requests
Product: Tomcat 6
Version: 6.0.16
Platform: Other
OS/Version: Linux
Status: NEW
Severity: major
Priority: P2
Component: Connectors
AssignedTo: [EMAIL PROTECTED]
ReportedBy: [EMAIL PROTECTED]
We have a high load environment where we are running tomcat 5.5.15
successfully. We are interesting in reducing the system CPU load and
switching to Tomcat 6 with NIO, but so far have run into a few issues.
After trying out the patch Filip recommended
(http://svn.apache.org/viewvc?view=rev&revision=618420) we did get
further (our CPUs were not pegged any more and exceptions went away),
but it appears that about 2/3rds of the requests are being dropped. I
was going to file a bug, but I thought I'd post it first to see if
anyone has any ideas.
We have many tomcat servers running 5.5.15. I took one of the machines,
and put 6.0.16 on it, patched it. Each server gets around 3K requests a
minute during the time window when I saw doing my experiments. Using
6.0.16 regular HTTP connections, I do get around 3K requests per minute
that we process. With NIO, we only see around 1.2K per minute. Here are
the two connector configurations.
Regular config
<Connector port="8000"
maxThreads="500" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="443" acceptCount="100"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml"
connectionTimeout="0" disableUploadTimeout="true"
maxHttpHeaderSize="8192" />
NIO config
<Connector port="8000"
protocol="org.apache.coyote.http11.Http11NioProtocol" acceptorThreadCount="4"
maxThreads="500" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="443" acceptCount="100"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml"
connectionTimeout="0" disableUploadTimeout="true"
maxHttpHeaderSize="8192" />
I tried to tune various NIO parameters and had no luck. I can only
assume that acceptors turn away some requests for some reason without
passing them down to the worker threads. Or that I have mis-configured
something.
Here are some interesting threads
A worker thread
http-8000-exec-20
52 TIMED_WAITING 20000000
.....omited....
javax.servlet.http.HttpServlet.service(690)
javax.servlet.http.HttpServlet.service(803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(206)
org.apache.catalina.core.StandardWrapperValve.invoke(233)
org.apache.catalina.core.StandardContextValve.invoke(175)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(433)
org.apache.catalina.core.StandardHostValve.invoke(128)
org.apache.catalina.valves.ErrorReportValve.invoke(102)
org.apache.catalina.valves.AccessLogValve.invoke(568)
org.apache.catalina.core.StandardEngineValve.invoke(109)
org.apache.catalina.connector.CoyoteAdapter.service(286)
org.apache.coyote.http11.Http11NioProcessor.process(879)
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(719)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(2080)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(907)
java.lang.Thread.run(619)
Idle worker thread
http-8000-exec-1
21 TIMED_WAITING 50000000
[EMAIL PROTECTED]
-1 41 -1 17 -1
sun.misc.Unsafe.park(-2)
java.util.concurrent.locks.LockSupport.parkNanos(198)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(1963)
java.util.concurrent.LinkedBlockingQueue.poll(395)
java.util.concurrent.ThreadPoolExecutor.getTask(944)
java.util.concurrent.ThreadPoolExecutor$Worker.run(906)
java.lang.Thread.run(619)
NIO endpoint
http-8000-ClientPoller
20 RUNNABLE 370000000
-1 285 -1
8 -1
sun.nio.ch.EPollArrayWrapper.epollWait(-2)
sun.nio.ch.EPollArrayWrapper.poll(215)
sun.nio.ch.EPollSelectorImpl.doSelect(65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(69)
sun.nio.ch.SelectorImpl.select(80)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(1473)
java.lang.Thread.run(619)
My 4 acceptors
http-8000-Acceptor-3
19 BLOCKED 50000000 [EMAIL PROTECTED]
16 71 -1
0 -1
sun.nio.ch.ServerSocketChannelImpl.accept(129)
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)
java.lang.Thread.run(619)
http-8000-Acceptor-2
18 BLOCKED 30000000 [EMAIL PROTECTED]
16 48 -1
0 -1
sun.nio.ch.ServerSocketChannelImpl.accept(129)
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)
java.lang.Thread.run(619)
http-8000-Acceptor-1
17 BLOCKED 40000000 [EMAIL PROTECTED]
16 51 -1
0 -1
sun.nio.ch.ServerSocketChannelImpl.accept(129)
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)
java.lang.Thread.run(619)
http-8000-Acceptor-0
16 RUNNABLE 80000000
-1 59 -1
0 -1
sun.nio.ch.ServerSocketChannelImpl.accept0(-2)
sun.nio.ch.ServerSocketChannelImpl.accept(145)
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)
java.lang.Thread.run(619)
My workers go up all the way to http-8000-exec-272. It looked like a handful of
them were doing work, but must were idle. Our servlet requests are very short
lived - 30ms tops.
Our application works with normal HTTP polling socket connector, but we would
like to use NIO because it would reduce the load on each tomcat node, and let
us scale better. I am very interested in trying any suggestions or patches you
might have. We have a unique scalability environment for Tomcat - many unique
requests from the internet at a pretty fast pace. If NIO can be made work in
our environment, it would be a major milestone of stability and performance for
it. It is also possibly a configuration issue. We would be happy to try any
patches/config changes.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
