Hi all Please excuse any awkward phrasing in my message, as I am not a native English speaker and am using translation software.
I'm sharing some information and I have some questions on an issue I encountered where the IdleConnectionEvictor thread in HttpClient 5.4.3 freezes while holding a lock on StrictConnPool, making it impossible to acquire a connection. This issue does not occur in HttpClient 5.3.x or 5.4.4 and later. The issue occurs under the following conditions: - Version: 5.4 through 5.4.3 - Connection Manager: Using PoolingHttpClientConnectionManager - Connectivity: Connecting via a proxy - TLS Protocol: Using TLS 1.2 (does not occur with TLS 1.3) - Trigger: When IdleConnectionEvictor attempts to close an expired socket, if it fails to receive a close_notify message, it will wait indefinitely on Socket.read(). The IdleConnectionEvictor periodically attempts to close expired connections. During this process, it holds a lock using the StrictConnPool's enumAvailable method. If it fails to receive a close_notify message, it waits forever while still holding the lock. The stack trace of the IdleConnectionEvictor thread when it freezes while waiting for close_notify is as follows: "idle-connection-evictor-1" #326 daemon prio=5 ... runnable ... java.lang.Thread.State: RUNNABLE at sun.nio.ch.Net.poll([email protected]/Native Method) at sun.nio.ch.NioSocketImpl.park([email protected]/NioSocketImpl.java:186) at sun.nio.ch.NioSocketImpl.park([email protected]/NioSocketImpl.java:195) at sun.nio.ch.NioSocketImpl.implRead([email protected]/NioSocketImpl.java:319) at sun.nio.ch.NioSocketImpl.read([email protected]/NioSocketImpl.java:355) at sun.nio.ch.NioSocketImpl$1.read([email protected]/NioSocketImpl.java:808) at java.net.Socket$SocketInputStream.read([email protected]/Socket.java:966) at sun.security.ssl.SSLSocketInputRecord.read([email protected]/SSLSocketInputRecord.java:484) at sun.security.ssl.SSLSocketInputRecord.readHeader([email protected]/SSLSocketInputRecord.java:478) at sun.security.ssl.SSLSocketInputRecord.decode([email protected]/SSLSocketInputRecord.java:160) at sun.security.ssl.SSLTransport.decode([email protected]/SSLTransport.java:111) at sun.security.ssl.SSLSocketImpl.decode([email protected]/SSLSocketImpl.java:1510) at sun.security.ssl.SSLSocketImpl.waitForClose([email protected]/SSLSocketImpl.java:1847) at sun.security.ssl.SSLSocketImpl.closeSocket([email protected]/SSLSocketImpl.java:1821) at sun.security.ssl.SSLSocketImpl.shutdown([email protected]/SSLSocketImpl.java:1766) at sun.security.ssl.SSLSocketImpl.bruteForceCloseInput([email protected]/SSLSocketImpl.java:799) at sun.security.ssl.SSLSocketImpl.duplexCloseOutput([email protected]/SSLSocketImpl.java:664) at sun.security.ssl.SSLSocketImpl.close([email protected]/SSLSocketImpl.java:584) at org.apache.hc.core5.io.Closer.close(Closer.java:48) at org.apache.hc.core5.io.Closer.closeQuietly(Closer.java:71) at org.apache.hc.core5.http.impl.io.BHttpConnectionBase.close(BHttpConnectionBase.java:268) at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.close(DefaultBHttpClientConnection.java:71) at org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection.close(DefaultManagedHttpClientConnection.java:176) at org.apache.hc.core5.pool.PoolEntry.discardConnection(PoolEntry.java:180) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.closeIfExpired(PoolingHttpClientConnectionManager.java:650) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$1.lambda$closeExpired$0(PoolingHttpClientConnectionManager.java:228) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$1$$Lambda$3021/0x00007f03a26afb60.execute(Unknown Source) at org.apache.hc.core5.pool.StrictConnPool.enumAvailable(StrictConnPool.java:590) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$1.closeExpired(PoolingHttpClientConnectionManager.java:228) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.closeExpired(PoolingHttpClientConnectionManager.java:542) at org.apache.hc.client5.http.impl.IdleConnectionEvictor.lambda$new$0(IdleConnectionEvictor.java:61) at org.apache.hc.client5.http.impl.IdleConnectionEvictor$$Lambda$2977/0x00007f03a2686b48.run(Unknown Source) at java.lang.Thread.run([email protected]/Thread.java:840) In this state, attempts to get a connection from the pool will fail with a DeadlineTimeoutException because the StrictPool lock cannot be acquired. The stack trace for the DeadlineTimeoutException is as follows: Caused by: org.apache.hc.core5.util.DeadlineTimeoutException: Deadline: 2025-05-11T23:30:14.996+0000, 0 MILLISECONDS overdue at org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeoutException.java:49) at org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:221) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:326) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:105) ... 138 common frames omitted Analysis of the Cause and Fix The reason this bug does not occur in version 5.3.x appears to be a change made in the AbstractClientTlsStrategy in a previous commit (HTTPCLIENT-2328), which changed the autoClose setting for the socket from true to false. - https://github.com/apache/httpcomponents-client/commit/ee0a10210#diff-a5e74a0fa48dd91a1e1bac65ff84722564da9e8b73fa835551c811a845aa2f2dL207-R208 - HTTPCLIENT-2328: Blocking i/o connections to check if the opposite TLS endpoint has been closed by the opposite endpoint while writing out request body - https://issues.apache.org/jira/browse/HTTPCLIENT-2328 The issue was resolved in version 5.4.4 by the fix for HTTPCLIENT-2364. This fix ensures that when DefaultHttpClientConnectionOperator.upgrade() upgrades to an SSLSocket, the baseSocket is also passed to ManagedHttpClientConnection.bind(). This allows the close method for expired connections to be called on the baseSocket, which prevents the deadlock. - https://github.com/apache/httpcomponents-client/commit/f3b1536843 - HTTPCLIENT-2364: Fixed incorrect re-binding of the upgraded SSL socket to the HTTP connection by the #upgrade method of the DefaultHttpClientConnectionOperator - https://issues.apache.org/jira/browse/HTTPCLIENT-2364 I have two questions: - Given the bug I encountered, was the change to set autoClose from true to false in the HTTPCLIENT-2328 fix the correct decision? - I tried setting a SocketTimeout to prevent the indefinite wait, but it didn't work. I found that this was because PoolingHttpClientConnectionManager.release() calls DefaultManagedHttpClientConnection.passivate(), which sets the socket timeout to Timeout.ZERO_MILLISECONDS when a connection is returned to the pool. Is it standard practice or correct to reset the timeout to an indefinite value when a connection is returned to the pool? Best regards, Kiyoshi Iwasaki --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
