Kristof Jozsa created HTTPCORE-629:
--------------------------------------
Summary: IOReactor shutdowns on ConnectionClosedException
Key: HTTPCORE-629
URL: https://issues.apache.org/jira/browse/HTTPCORE-629
Project: HttpComponents HttpCore
Issue Type: Bug
Affects Versions: 4.4.12
Reporter: Kristof Jozsa
We run a Camel/CXFRS project with httpcore-nio 4.4.12 and we experience
IOReactor shutdowns which seem to happen on ConnectionClosedExceptions in our
production environment.
We found this spamming in our logs:
{{Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O
reactor status: STOPPED}}
{{ at org.apache.http.util.Asserts.check(Asserts.java:46)}}
{{ at
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)}}
{{ at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.connect(AsyncHTTPConduit.java:596)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.setupWrappedStream(AsyncHTTPConduit.java:442)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.onFirstWrite(AsyncHTTPConduit.java:435)}}
Hunting for reasons we first found `ConnectionClosedException`s - these are
completely fine:
{{Caused by: org.apache.http.ConnectionClosedException:
ConnectionClosedException invoking
https://target-host-masked.com/api/rest/v1/products/21110363: Connection closed
unexpectedlyCaused by: org.apache.http.ConnectionClosedException:
ConnectionClosedException invoking
https://target-host-masked.com/api/rest/v1/products/21110363: Connection closed
unexpectedly at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384)
at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429)
at
org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)
at
org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228) at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at
org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)}}
but these following traces:
{{Caused by: org.apache.http.ConnectionClosedException: Connection closed
unexpectedlyCaused by: org.apache.http.ConnectionClosedException: Connection
closed unexpectedly at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:590)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)}}
seem to show that the IOReactor went to hardShutdown upon the
ConnectionClosedException?
We tried digging into the source to understand how this can happen and we
believe that if the {{processClosedSessions()}} method in {{AbstractIOReactor}}
class shows any uncaught exceptions (like a ConnectionClosedException?), it
will indeed cause the reactor to stop outside its infinite loop in the finally
block.
Can you please double check and confirm this behaviour? How can we avoid
getting this problem? I would assume if a certain connection gets broken on the
server side, it's handled gracefully. The fact that the whole reactor shuts
down leaves us no other option then to restart our service - clearly we need a
proper way to manage this somehow.
I hope I'm not horribly mistaken with the drawn consequences above - if so,
please point out my errors. Thanks really much in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
