H2StreamResetExceptions and TimeoutExceptions seen when communicating with Apple's APNs service using HTTP/2
Recently we switched from the Apple's legacy Binary Provider API to the
newer HTTP/2 based API in order to send our notifications using APNs. We
decided on using Apache's HttpClient 5 in order to meet the HTTP/2
requirement. However, our current support for this seems unstable as we
are seeing the occasional H2StreamResetExceptions and TimeoutExceptions.
We are currently creating the HttpAsyncClient as follows:
HttpAsyncClients.custom().
setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2).
setConnectionManager(
PoolingAsyncClientConnectionManagerBuilder.create().
setTlsStrategy(
new H2TlsStrategy(
SSLContexts.custom().
loadKeyMaterial(_keyStore, _keyStorePassword).
build(),
new String[] {"TLSv1.2"},
null, null, new DefaultHostnameVerifier()
)
).
build()
).
build()
This HttpAsyncClient is shared among a configurable amount of threads
with each thread having its own HttpClientContext instance. Each thread
is sending our HTTP/2 requests as follows:
Future _future =
getHTTPClient().execute(_request, getHTTPContext(), null);
try {
SimpleHttpResponse _response = _future.get(3,
TimeUnit.MINUTES);
...
} catch (final CancellationException exception) { // If the
computation was cancelled.
...
} catch (final ExecutionException exception) { // If the
computation threw an exception.
...
} catch (final InterruptedException exception) { // If the
current thread was interrupted while waiting.
...
} catch (final TimeoutException exception) { // If the wait
timed out.
...
}
On occasion we see and log the H2StreamResetException, wrapped inside an
ExecutionException, for now. The details of this exception is as follows:
java.util.concurrent.ExecutionException:
org.apache.hc.core5.http2.H2StreamResetException: Connection
terminated by the peer
at
org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.java:71)
at
org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:102)
...
Caused by: org.apache.hc.core5.http2.H2StreamResetException:
Connection terminated by the peer
at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer.consumeFrame(AbstractHttp2StreamMultiplexer.java:974)
at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer.onInput(AbstractHttp2StreamMultiplexer.java:419)
at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2IOEventHandler.inputReady(AbstractHttp2IOEventHandler.java:63)
at
org.apache.hc.core5.http2.impl.nio.ClientHttp2IOEventHandler.inputReady(ClientHttp2IOEventHandler.java:38)
at
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:117)
at
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:50)
at
org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:173)
at
org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:123)
at
org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:80)
at
org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
... 1 more
As we are using a PoolingAsyncClientConnectionManager and this exception
is indicating "Connection terminated by the peer", is there something we
need to do on our end in order to deal with this exception and to
recover from it?
Also on occassion we see and log the TimeoutException for now. The
details of this exception is as follows:
java.util.concurrent.TimeoutException
at
org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:106)
...
Why would this happen on occasion? Is this something caused by the
HttpAsyncClient or possible by the APNs service?
Re: H2StreamResetExceptions and TimeoutExceptions seen when communicating with Apple's APNs service using HTTP/2
On Mon, 2018-09-24 at 16:56 +0200, Jack van Ooststroom wrote:
> Recently we switched from the Apple's legacy Binary Provider API to
> the
> newer HTTP/2 based API in order to send our notifications using APNs.
> We
> decided on using Apache's HttpClient 5 in order to meet the HTTP/2
> requirement. However, our current support for this seems unstable as
> we
> are seeing the occasional H2StreamResetExceptions and
> TimeoutExceptions.
>
> We are currently creating the HttpAsyncClient as follows:
>
> HttpAsyncClients.custom().
> setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2).
> setConnectionManager(
> PoolingAsyncClientConnectionManagerBuilder.create().
> setTlsStrategy(
> new H2TlsStrategy(
> SSLContexts.custom().
> loadKeyMaterial(_keyStore, _keyStorePassword).
> build(),
> new String[]
> {"TLSv1.2"},
> null, null, new DefaultHostnameVerifier()
> )
> ).
> build()
> ).
> build()
>
> This HttpAsyncClient is shared among a configurable amount of
> threads
> with each thread having its own HttpClientContext instance. Each
> thread
> is sending our HTTP/2 requests as follows:
>
> Future _future =
> getHTTPClient().execute(_request, getHTTPContext(), null);
> try {
> SimpleHttpResponse _response = _future.get(3,
> TimeUnit.MINUTES);
> ...
> } catch (final CancellationException exception) { // If
> the
> computation was cancelled.
> ...
> } catch (final ExecutionException exception) { // If the
> computation threw an exception.
> ...
> } catch (final InterruptedException exception) { // If
> the
> current thread was interrupted while waiting.
> ...
> } catch (final TimeoutException exception) { // If the
> wait
> timed out.
> ...
> }
>
> On occasion we see and log the H2StreamResetException, wrapped inside
> an
> ExecutionException, for now. The details of this exception is as
> follows:
>
> java.util.concurrent.ExecutionException:
> org.apache.hc.core5.http2.H2StreamResetException: Connection
> terminated by the peer
> at
> org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.
> java:71)
> at
> org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:1
> 02)
> ...
> Caused by: org.apache.hc.core5.http2.H2StreamResetException:
> Connection terminated by the peer
> at
> org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer
> .consumeFrame(AbstractHttp2StreamMultiplexer.java:974)
> at
> org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer
> .onInput(AbstractHttp2StreamMultiplexer.java:419)
> at
> org.apache.hc.core5.http2.impl.nio.AbstractHttp2IOEventHandler.in
> putReady(AbstractHttp2IOEventHandler.java:63)
> at
> org.apache.hc.core5.http2.impl.nio.ClientHttp2IOEventHandler.inpu
> tReady(ClientHttp2IOEventHandler.java:38)
> at
> org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(Interna
> lDataChannel.java:117)
> at
> org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(Interna
> lChannel.java:50)
> at
> org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(Sin
> gleCoreIOReactor.java:173)
> at
> org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleC
> oreIOReactor.java:123)
> at
> org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(A
> bstractSingleCoreIOReactor.java:80)
> at
> org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.j
> ava:44)
> ... 1 more
Please note that HttpClient signals the fact that the connection has
been terminated by the opposite endpoint. By itself this does not mean
there is anything wrong with the client side. H2StreamResetException
includes an error code that would help understand the reason for
connection termination.
>
> As we are using a PoolingAsyncClientConnectionManager and this
> exception
> is indicating "Connection terminated by the peer", is there something
> we
> need to do on our end in order to deal with this exception and to
> recover from it?
>
Depending on the error code and the request method it might be
sufficient to re-try the request.
> Also on occassion we see and log the TimeoutException for now. The
> details of this exception is as
