Leaked SharedInputBuffer and SharedOutputBuffers slowly consume
HttpServerWorker pool
-------------------------------------------------------------------------------------
Key: SYNAPSE-415
URL: https://issues.apache.org/jira/browse/SYNAPSE-415
Project: Synapse
Issue Type: Bug
Components: Transports
Affects Versions: 1.2
Environment: Synapse 1.2 on Red Hat EL3
Reporter: Jason Walton
Priority: Critical
1) Create a synapse configuration with a single proxy service (Sample 150).
2) Put a breakpoint in
org.apache.synapse.transport.nhttp.ClientHandler.outputReady() at the beginning
of the method.
3) Put another breakpoint in ClientHandler.closed(), again at the beginning of
the method.
4) Send a message through the proxy service with SoapUI.
5) When you hit the breakpoint in outputReady(), wait 15 seconds and then
continue.
6) You should hit the breakpoint in closed() immediately. closed() will remove
the source buffer from the HttpContext without first calling shutdown on it
(you can verify this by finding the SharedOutputBuffer in the
NHttpClientConnection's context, and you will see that the private boolean
"shutdown" is false). The HttpServerWorker which is currently waiting on that
buffer will never get a notify, and will effectively be leaked.
The HttpServerWorker in question will have a stack trace as follows:
Thread [HttpServerWorker-2] (Suspended)
Object.wait(long) line: not available [native method]
Object.wait() line: 485
SharedOutputBuffer.flushContent() line: 161
SharedOutputBuffer.write(byte[], int, int) line: 118
ContentOutputStream.write(byte[], int, int) line: 63
UTF8Writer.write(char[], int, int) line: 139
BufferingXmlWriter.writeRaw(char[], int, int) line: 259
BufferingXmlWriter.writeCharacters(char[], int, int) line: 543
SimpleNsStreamWriter(BaseStreamWriter).writeCharacters(String) line:
509
MTOMXMLStreamWriter.writeCharacters(String) line: 237
StreamingOMSerializer.serializeText(XMLStreamReader, XMLStreamWriter)
line: 369
StreamingOMSerializer.serializeNode(XMLStreamReader, XMLStreamWriter,
boolean) line: 109
StreamingOMSerializer.serialize(XMLStreamReader, XMLStreamWriter,
boolean) line: 68
StreamingOMSerializer.serialize(XMLStreamReader, XMLStreamWriter) line:
57
OMSerializerUtil.serializeByPullStream(OMElement, XMLStreamWriter,
boolean) line: 548
SOAPEnvelopeImpl.internalSerialize(XMLStreamWriter, boolean) line: 232
SOAPEnvelopeImpl(OMElementImpl).internalSerializeAndConsume(XMLStreamWriter)
line: 947
SOAPEnvelopeImpl(OMNodeImpl).serializeAndConsume(OutputStream,
OMOutputFormat) line: 471
SOAPMessageFormatter.writeTo(MessageContext, OMOutputFormat,
OutputStream, boolean) line: 79
Axis2HttpRequest.streamMessageContents() line: 221
HttpCoreNIOSender.sendAsyncRequest(EndpointReference, MessageContext)
line: 346
HttpCoreNIOSender.invoke(MessageContext) line: 256
AxisEngine.send(MessageContext) line: 448
DynamicAxisOperation$DynamicOperationClient.send(MessageContext) line:
190
DynamicAxisOperation$DynamicOperationClient.executeImpl(boolean) line:
174
DynamicAxisOperation$DynamicOperationClient(OperationClient).execute(boolean)
line: 163
Axis2FlexibleMEPClient.send(EndpointDefinition, MessageContext) line:
288
Axis2Sender.sendOn(EndpointDefinition, MessageContext) line: 57
Axis2SynapseEnvironment.send(EndpointDefinition, MessageContext) line:
222
AddressEndpoint.send(MessageContext) line: 195
ProxyServiceMessageReceiver.receive(MessageContext) line: 179
AxisEngine.receive(MessageContext) line: 176
HTTPTransportUtils.processHTTPPostRequest(MessageContext, InputStream,
OutputStream, String, String, String) line: 275
ServerWorker.processPost() line: 253
ServerWorker.run() line: 194
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619
The fix is simply to call shutdown on the SharedInputerBuffer and
SharedOutputBuffer after removing them from the context. This same problem
exists in ServerHandler.closed() as well. Here's my modified versions of these
methods which seems to fix the problem:
In ServerHander.java:
public void closed(final NHttpServerConnection conn) {
HttpContext context = conn.getContext();
SharedInputBuffer sinkBuffer =
(SharedInputBuffer)context.getAttribute(REQUEST_SINK_BUFFER);
context.removeAttribute(REQUEST_SINK_BUFFER);
if(sinkBuffer != null) {
sinkBuffer.shutdown();
}
SharedOutputBuffer sourceBuffer =
(SharedOutputBuffer)context.getAttribute(RESPONSE_SOURCE_BUFFER);
context.removeAttribute(RESPONSE_SOURCE_BUFFER);
if(sourceBuffer != null) {
sourceBuffer.shutdown();
}
if (log.isTraceEnabled()) {
log.trace("Connection closed");
}
}
And, in ClientHandler.java:
public void closed(final NHttpClientConnection conn) {
ConnectionPool.forget(conn);
checkAxisRequestComplete(conn, "Abnormal connection close", null);
HttpContext context = conn.getContext();
SharedInputBuffer sinkBuffer =
(SharedInputBuffer)context.getAttribute(RESPONSE_SINK_BUFFER);
context.removeAttribute(RESPONSE_SINK_BUFFER);
if(sinkBuffer != null) {
sinkBuffer.shutdown();
}
SharedOutputBuffer sourceBuffer =
(SharedOutputBuffer)context.getAttribute(REQUEST_SOURCE_BUFFER);
context.removeAttribute(REQUEST_SOURCE_BUFFER);
if(sourceBuffer != null) {
sourceBuffer.shutdown();
}
if (log.isTraceEnabled()) {
log.trace("Connection closed");
}
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
