Hello,
We recently moved from HttpClient 5.5 to 5.6.1 and started seeing
connection request timeouts on certain bursty traffic patterns. These
bubble up as RequestFailedException. Our connection pool config was left
unchanged during the migration and has a pretty "standard" setup I'd say :
var connectionPool = PoolingHttpClientConnectionManagerBuilder.create()
.setMaxConnTotal(200)
.setMaxConnPerRoute(100)
.build();
var httpClient = HttpClientBuilder.create()
.setConnectionManager(connectionPool)
.setDefaultRequestConfig(RequestConfig.custom()
.setConnectionRequestTimeout(Timeout.ofSeconds(1))
.build())
.evictIdleConnections(Timeout.ofSeconds(10))
.build();
I traced changes down a bit and I believe the cause is the change from
HTTPCLIENT-2388 / PR #714. In 5.5, HttpClientBuilder passed sleepTime =
maxIdleTime, so the evictor swept every 10 seconds. In 5.6 it passes null,
and calculateSleepTime() returns maxIdleTime/10 with a one second floor, so
it now sweeps every second. That is ten times more often, and there is no
way to set sleepTime through the builder.
The higher frequency causes a contention issue because closing idle/expired
connections is, when using the StrictConnPool, not lock free, and the
DefaultDisposalCallback
impose a timeout of 1 second when disposing connections. We believe this is
the cause of the sporadic issues we've been seeing. I've included a
redacted stack trace at the end of the email
This being said, I understand the nature of the changes and the desire to
close idle connections closer to their desired idle time. Therefore, I
think these 2 options could make sense :
- Allow the easy customization of the sleepTime in HttpClientBuilder (and
its async builder buddy)
- Reconsider the divisor used to calculate the sleepTime. Using 10 might
make sense for an idle timeout of 100 seconds, but less for an idle timeout
of 10.
I know about the off lock disposal new feature and I'm gonna try it in
order to fix the issue, but I still think it's worthy of checking it since
that off lock disposal is still experimental and disabled by default.
Thanks to all!
Jacques-Etienne
org.apache.hc.client5.http.impl.classic.RequestFailedException: Request
execution failed
at
org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:133)
at
org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:127)
at
org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
at
org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:138)
at
org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:112)
at
org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
at
org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)
... application frames omitted ...
Caused by: org.apache.hc.core5.util.DeadlineTimeoutException: Deadline:
2026-09-16T21:11:31.165+0000, -116 MILLISECONDS overdue
at
org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeoutException.java:49)
at
org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:216)
at
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:366)
at
org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:105)
