I have a simple servlet which I'm running on Tomcat 8 trunk with Java 7. It's
using a non-blocking IO WriteListener to write 8k blocks of static data to the
response. The number of blocks it will write is given as a request parameter.
When I start out making individual requests to the server (1-1000 blocks)
things are fine and those are handled without a problem. However once I start
to send multiple requests using JMeter, requests begin to fail. It starts
slowly with only a few requests failing and then proceeds to the point where
every request fails. At that point, Tomcat will serve JSP pages and Servlets
that use blocking IO, but it will not serve any requests that use the
non-blocking IO apis (ReadListener / WriteListener). If I try to hit the byte
counter servlet or number writer servlet included in the examples then my
browser will timeout or display the error "java.lang.IllegalStateException:
getOutputStream() has already been called for this response".
In my tests most of the time the requests fail because the async context times
out, but occasionally (one or two times out of 100) I'll see these exceptions.
1.) org.apache.catalina.connector.ClientAbortException: java.io.IOException:
Key must be cancelled
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:396)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:315)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:421)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:409)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:90)
at
com.pivotal.demos.nbio.DataStreamerNbioServlet$2.onWritePossible(DataStreamerNbioServlet.java:64)
at org.apache.coyote.Response.onWritePossible(Response.java:655)
at
org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:369)
at
org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1618)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:631)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1576)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Key must be cancelled
at
org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:131)
at
org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:198)
at
org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:42)
at
org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:321)
at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:116)
at
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:257)
at org.apache.coyote.Response.doWrite(Response.java:520)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
... 16 more
2.) 19-Dec-2013 11:14:34.470 SEVERE [http-nio-8080-ClientPoller-0]
org.apache.tomcat.util.net.NioEndpoint$PollerEvent.run
java.nio.channels.ClosedChannelException
at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:194)
at
org.apache.tomcat.util.net.NioEndpoint$PollerEvent.run(NioEndpoint.java:797)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:925)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1045)
at java.lang.Thread.run(Thread.java:744)
I put together a unit test that replicates the async context timeout issue.
Code is listed below. When run, you'll see that it processes some of the
requests but fails due to a timeout. I've not been able to replicate the other
exceptions with the unit test though, so those may be unrelated.
public class TestWriteListener extends TomcatBaseTest {
private static final int BLOCK_SIZE = 8192;
private static final byte[] DATA = new byte[BLOCK_SIZE];
@Test
public void testEchoListener1() throws Exception {
Tomcat tomcat = getTomcatInstance();
// Must have a real docBase - just use temp
StandardContext ctx = (StandardContext) tomcat.addContext("",
System.getProperty("java.io.tmpdir"));
DataStreamingServlet servlet = new DataStreamingServlet();
String servletName = DataStreamingServlet.class.getName();
Tomcat.addServlet(ctx, servletName, servlet);
ctx.addServletMapping("/", servletName);
tomcat.start();
Arrays.fill(DATA, Byte.parseByte("1"));
for (int i = 0; i < 200; i++) {
int blocks = 200;
Map<String, List<String>> resHeaders = new HashMap<>();
ByteChunk out = new ByteChunk();
int rc = getUrl("http://localhost:"; + getPort() + "/?blocks=" +
blocks, out, resHeaders);
Assert.assertEquals(HttpServletResponse.SC_OK, rc);
Assert.assertEquals(BLOCK_SIZE * blocks, out.getLength());
}
}
@WebServlet(asyncSupported = true)
public class DataStreamingServlet extends HttpServlet {
private static final long serialVersionUID = -3038617032944575769L;
public static final int BUFFER_SIZE = 8 * 1024;
@Override
protected void service(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException
{
int blocks = Integer.parseInt(request.getParameter("blocks"));
AsyncContext asyncContext = request.startAsync(request,
response);
asyncContext.setTimeout(10);
asyncContext.addListener(new AsyncListener() {
@Override
public void onTimeout(AsyncEvent event) throws
IOException {
Assert.fail("Request has timed out");
}
@Override
public void onStartAsync(AsyncEvent event)
throws IOException {
}
@Override
public void onError(AsyncEvent event) throws
IOException {
}
@Override
public void onComplete(AsyncEvent event) throws
IOException {
}
});
DataStreamWriteListener dswl = new
DataStreamWriteListener(asyncContext, blocks);
response.getOutputStream().setWriteListener(dswl);
}
private class DataStreamWriteListener implements WriteListener {
private AsyncContext context;
private int blocks = 1;
private int block_count = 0;
public DataStreamWriteListener(AsyncContext context, int
blocks) {
this.context = context;
this.blocks = blocks;
}
@Override
public void onWritePossible() throws IOException {
ServletOutputStream output =
context.getResponse().getOutputStream();
while (output.isReady() && block_count <
blocks) {
output.write(DATA);
block_count += 1;
}
if (block_count >= blocks) {
context.complete();
}
}
@Override
public void onError(Throwable throwable) {
throwable.printStackTrace(System.err);
context.complete();
}
}
}
}
Any thoughts on what's happening here?
Thanks
Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
