On Thu, 2025-09-11 at 10:46 +0000, [email protected] wrote: > Hi all > > Please excuse any awkward phrasing in my message, as I am not a > native English > speaker and am using translation software. > > I'm sharing some information and I have some questions on an issue I > encountered > where the IdleConnectionEvictor thread in HttpClient 5.4.3 freezes > while holding > a lock on StrictConnPool, making it impossible to acquire a > connection. > This issue does not occur in HttpClient 5.3.x or 5.4.4 and later. > > The issue occurs under the following conditions: > - Version: 5.4 through 5.4.3 > - Connection Manager: Using PoolingHttpClientConnectionManager > - Connectivity: Connecting via a proxy > - TLS Protocol: Using TLS 1.2 (does not occur with TLS 1.3) > - Trigger: When IdleConnectionEvictor attempts to close an expired > socket, > if it fails to receive a close_notify message, it will wait > indefinitely > on Socket.read(). > > The IdleConnectionEvictor periodically attempts to close expired > connections. > During this process, it holds a lock using the StrictConnPool's > enumAvailable > method. If it fails to receive a close_notify message, it waits > forever > while still holding the lock. > > The stack trace of the IdleConnectionEvictor thread when it freezes > while > waiting for close_notify is as follows: > > "idle-connection-evictor-1" #326 daemon prio=5 ... runnable ... > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.Net.poll([email protected]/Native Method) > at > sun.nio.ch.NioSocketImpl.park([email protected]/NioSocketImpl.java:18 > 6) > at > sun.nio.ch.NioSocketImpl.park([email protected]/NioSocketImpl.java:19 > 5) > at > sun.nio.ch.NioSocketImpl.implRead([email protected]/NioSocketImpl.jav > a:319) > at > sun.nio.ch.NioSocketImpl.read([email protected]/NioSocketImpl.java:35 > 5) > at > sun.nio.ch.NioSocketImpl$1.read([email protected]/NioSocketImpl.java: > 808) > at > java.net.Socket$SocketInputStream.read([email protected]/Socket.java: > 966) > at > sun.security.ssl.SSLSocketInputRecord.read([email protected]/SSLSocke > tInputRecord.java:484) > at > sun.security.ssl.SSLSocketInputRecord.readHeader([email protected]/SS > LSocketInputRecord.java:478) > at > sun.security.ssl.SSLSocketInputRecord.decode([email protected]/SSLSoc > ketInputRecord.java:160) > at > sun.security.ssl.SSLTransport.decode([email protected]/SSLTransport.j > ava:111) > at > sun.security.ssl.SSLSocketImpl.decode([email protected]/SSLSocketImpl > .java:1510) > at > sun.security.ssl.SSLSocketImpl.waitForClose([email protected]/SSLSock > etImpl.java:1847) > at > sun.security.ssl.SSLSocketImpl.closeSocket([email protected]/SSLSocke > tImpl.java:1821) > at > sun.security.ssl.SSLSocketImpl.shutdown([email protected]/SSLSocketIm > pl.java:1766) > at > sun.security.ssl.SSLSocketImpl.bruteForceCloseInput([email protected] > /SSLSocketImpl.java:799) > at > sun.security.ssl.SSLSocketImpl.duplexCloseOutput([email protected]/SS > LSocketImpl.java:664) > at > sun.security.ssl.SSLSocketImpl.close([email protected]/SSLSocketImpl. > java:584) > at org.apache.hc.core5.io.Closer.close(Closer.java:48) > at org.apache.hc.core5.io.Closer.closeQuietly(Closer.java:71) > at > org.apache.hc.core5.http.impl.io.BHttpConnectionBase.close(BHttpConne > ctionBase.java:268) > at > org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.close(D > efaultBHttpClientConnection.java:71) > at > org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection > .close(DefaultManagedHttpClientConnection.java:176) > at > org.apache.hc.core5.pool.PoolEntry.discardConnection(PoolEntry.java:1 > 80) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > .closeIfExpired(PoolingHttpClientConnectionManager.java:650) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > $1.lambda$closeExpired$0(PoolingHttpClientConnectionManager.java:228) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > $1$$Lambda$3021/0x00007f03a26afb60.execute(Unknown Source) > at > org.apache.hc.core5.pool.StrictConnPool.enumAvailable(StrictConnPool. > java:590) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > $1.closeExpired(PoolingHttpClientConnectionManager.java:228) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > .closeExpired(PoolingHttpClientConnectionManager.java:542) > at > org.apache.hc.client5.http.impl.IdleConnectionEvictor.lambda$new$0(Id > leConnectionEvictor.java:61) > at > org.apache.hc.client5.http.impl.IdleConnectionEvictor$$Lambda$2977/0x > 00007f03a2686b48.run(Unknown Source) > at java.lang.Thread.run([email protected]/Thread.java:840) > > In this state, attempts to get a connection from the pool will fail > with a > DeadlineTimeoutException because the StrictPool lock cannot be > acquired. > > The stack trace for the DeadlineTimeoutException is as follows: > > Caused by: org.apache.hc.core5.util.DeadlineTimeoutException: > Deadline: 2025-05-11T23:30:14.996+0000, 0 MILLISECONDS overdue > at > org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeou > tException.java:49) > at > org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:221 > ) > at > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager > .lease(PoolingHttpClientConnectionManager.java:326) > at > org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEn > dpoint(InternalExecRuntime.java:105) > ... 138 common frames omitted > > Analysis of the Cause and Fix > > The reason this bug does not occur in version 5.3.x appears to be a > change > made in the AbstractClientTlsStrategy in a previous commit > (HTTPCLIENT-2328), > which changed the autoClose setting for the socket from true to > false. > > - > https://github.com/apache/httpcomponents-client/commit/ee0a10210#diff-a5e74a0fa48dd91a1e1bac65ff84722564da9e8b73fa835551c811a845aa2f2dL207-R208 > - HTTPCLIENT-2328: Blocking i/o connections to check if the > opposite > TLS endpoint has been closed by the opposite endpoint while > writing > out request body > - https://issues.apache.org/jira/browse/HTTPCLIENT-2328 > > The issue was resolved in version 5.4.4 by the fix for HTTPCLIENT- > 2364. > This fix ensures that when > DefaultHttpClientConnectionOperator.upgrade() > upgrades to an SSLSocket, the baseSocket is also passed to > ManagedHttpClientConnection.bind(). > This allows the close method for expired connections to be called on > the > baseSocket, which prevents the deadlock. > - https://github.com/apache/httpcomponents-client/commit/f3b1536843 > - HTTPCLIENT-2364: Fixed incorrect re-binding of the upgraded SSL > socket > to the HTTP connection by the #upgrade method of the > DefaultHttpClientConnectionOperator > - https://issues.apache.org/jira/browse/HTTPCLIENT-2364 > > > I have two questions: > > - Given the bug I encountered, was the change to set autoClose from > true > to false in the HTTPCLIENT-2328 fix the correct decision?
Yes, it was. It was necessary in order to have a finer control over the state of TLS session. > - I tried setting a SocketTimeout to prevent the indefinite wait, but > it > didn't work. > I found that this was because > PoolingHttpClientConnectionManager.release() > calls DefaultManagedHttpClientConnection.passivate(), which sets > the socket > timeout to Timeout.ZERO_MILLISECONDS when a connection is returned > to the pool. > Is it standard practice or correct to reset the timeout to an > indefinite > value when a connection is returned to the pool? The connection is meant to be inactive while it sits idle in the pool. What is wrong however is the connection pool trying to execute a potentially blocking i/o operation while holding a global lock on the pool. This is wrong and needs to be fixed. Please raise a JIRA for this defect https://issues.apache.org/jira/browse/HTTPCLIENT Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
