[
https://issues.apache.org/jira/browse/HTTPCORE-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Sartissohn updated HTTPCORE-596:
---------------------------------------
Description:
We have some http endpoint not returning the CLOSE_NOTIFY in time. Sometime it
takes about up to 15 minutes till the connection is closed.
During that time we find the PoolingHttpClientConnectionManager in closing the
idle connections. Here the stack:
{{"IdleConnectionMonitor:MyHTTPClient" Id=765 in RUNNABLE}}
\{{ at java.net.SocketInputStream.socketRead0(Native Method)}}
\{{ at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)}}
\{{ at java.net.SocketInputStream.read(SocketInputStream.java:171)}}
\{{ at java.net.SocketInputStream.read(SocketInputStream.java:141)}}
\{{ at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)}}
\{{ at sun.security.ssl.InputRecord.read(InputRecord.java:503)}}
\{{ at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)}}
\{{ - locked java.lang.Object@7b9c4e14}}
\{{ at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)}}
\{{ at sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1571)}}
\{{ at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1705)}}
\{{ at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1594)}}
\{{ at
a.b.security.zone.api.factories.ZoneSSLSocket.close(ZoneSSLSocket.java:572)}}
\{{ - locked a.b.security.zone.api.factories.ZoneSSLSocket@307c017c}}
\{{ at
a.b.http.client.sdk.HTTPDumpingSSLSocket.close(HTTPDumpingSSLSocket.java:77)}}
\{{ - locked a.b.http.client.sdk.HTTPDumpingSSLSocket@658dab05}}
\{{ at
a.b.http.base.connection.MyDefaultClientConnection.close(MyDefaultClientConnection.java:711)}}
\{{ at
org.apache.http.impl.conn.CPoolEntry.closeConnection(CPoolEntry.java:70)}}
\{{ at org.apache.http.impl.conn.CPoolEntry.close(CPoolEntry.java:96)}}
\{{ at
org.apache.http.pool.AbstractConnPool$3.process(AbstractConnPool.java:622)}}
\{{ at
org.apache.http.pool.AbstractConnPool.enumAvailable(AbstractConnPool.java:561)}}
\{{ at org.apache.http.impl.conn.CPool.enumAvailable(CPool.java:76)}}
\{{ at
org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:617)}}
\{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:424)
}}
While the thread is waiting here for the close the whole connection pool is
blocked due to some lock in the enumAvailble method. Any try to lease a new
connection needs to wait till the lock is released here.
The problem is that the whole pool is blocked and not only connections to that
specific route the close is pending. Finally my whole application is waiting
for this single partner to reply on the close.
Is there any chance to optimze the pool to be less strict when iterating the
connection pool?
Here an example for a a call waiting for a connection due to the lock in the
enumAvailable method:
{{"pool-144-thread-13" Id=30179 in WAITING on
lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@7dbb4bbf owned by
IdleConnectionMonitor:MyHTTPClient Id=27798}}
\{{ at sun.misc.Unsafe.park(Native Method)}}
\{{ at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)}}
\{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)}}
\{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)}}
\{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)}}
\{{ at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)}}
\{{ at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)}}
\{{ at
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:305)}}
\{{ at
org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)}}
\{{ at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)}}
\{{ - locked org.apache.http.pool.AbstractConnPool$2@1e25f993}}
\{{ at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)}}
\{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304)}}
\{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)}}
\{{ at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)}}
\{{ at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)}}
\{{ at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)}}
\{{ at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)}}
\{{ at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)}}
\{{ at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)}}
was:
We have some http endpoint not returning the CLOSE_NOTIFY in time. Sometime it
takes about up to 15 minutes till the connection is closed.
During that time we find the PoolingHttpClientConnectionManager in closing the
idle connections. Here the stack:
{{"IdleConnectionMonitor:SeeburgerHTTPClient" Id=765 in RUNNABLE}}
{{ at java.net.SocketInputStream.socketRead0(Native Method)}}
{{ at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)}}
{{ at java.net.SocketInputStream.read(SocketInputStream.java:171)}}
{{ at java.net.SocketInputStream.read(SocketInputStream.java:141)}}
{{ at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)}}
{{ at sun.security.ssl.InputRecord.read(InputRecord.java:503)}}
{{ at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)}}
{{ - locked java.lang.Object@7b9c4e14}}
{{ at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)}}
{{ at sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1571)}}
{{ at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1705)}}
{{ at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1594)}}
{{ at
a.b.security.zone.api.factories.ZoneSSLSocket.close(ZoneSSLSocket.java:572)}}
{{ - locked a.b.security.zone.api.factories.ZoneSSLSocket@307c017c}}
{{ at
a.b.http.client.sdk.HTTPDumpingSSLSocket.close(HTTPDumpingSSLSocket.java:77)}}
{{ - locked a.b.http.client.sdk.HTTPDumpingSSLSocket@658dab05}}
{{ at
a.b.http.base.connection.MyDefaultClientConnection.close(MyDefaultClientConnection.java:711)}}
{{ at org.apache.http.impl.conn.CPoolEntry.closeConnection(CPoolEntry.java:70)}}
{{ at org.apache.http.impl.conn.CPoolEntry.close(CPoolEntry.java:96)}}
{{ at
org.apache.http.pool.AbstractConnPool$3.process(AbstractConnPool.java:622)}}
{{ at
org.apache.http.pool.AbstractConnPool.enumAvailable(AbstractConnPool.java:561)}}
{{ at org.apache.http.impl.conn.CPool.enumAvailable(CPool.java:76)}}
{{ at
org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:617)}}
{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:424)
}}
While the thread is waiting here for the close the whole connection pool is
blocked due to some lock in the enumAvailble method. Any try to lease a new
connection needs to wait till the lock is released here.
The problem is that the whole pool is blocked and not only connections to that
specific route the close is pending. Finally my whole application is waiting
for this single partner to reply on the close.
Is there any chance to optimze the pool to be less strict when iterating the
connection pool?
Here an example for a a call waiting for a connection due to the lock in the
enumAvailable method:
{{"pool-144-thread-13" Id=30179 in WAITING on
lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@7dbb4bbf owned by
IdleConnectionMonitor:MyHTTPClient Id=27798}}
{{ at sun.misc.Unsafe.park(Native Method)}}
{{ at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)}}
{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)}}
{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)}}
{{ at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)}}
{{ at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)}}
{{ at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)}}
{{ at
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:305)}}
{{ at
org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)}}
{{ at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)}}
{{ - locked org.apache.http.pool.AbstractConnPool$2@1e25f993}}
{{ at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)}}
{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304)}}
{{ at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)}}
{{ at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)}}
{{ at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)}}
{{ at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)}}
{{ at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)}}
{{ at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)}}
{{ at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)}}
> Closing idle connections may end up in a blocked connection pool
> ----------------------------------------------------------------
>
> Key: HTTPCORE-596
> URL: https://issues.apache.org/jira/browse/HTTPCORE-596
> Project: HttpComponents HttpCore
> Issue Type: Improvement
> Components: HttpCore
> Affects Versions: 4.4.11
> Reporter: Thomas Sartissohn
> Priority: Major
>
> We have some http endpoint not returning the CLOSE_NOTIFY in time. Sometime
> it takes about up to 15 minutes till the connection is closed.
> During that time we find the PoolingHttpClientConnectionManager in closing
> the idle connections. Here the stack:
> {{"IdleConnectionMonitor:MyHTTPClient" Id=765 in RUNNABLE}}
> \{{ at java.net.SocketInputStream.socketRead0(Native Method)}}
> \{{ at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)}}
> \{{ at java.net.SocketInputStream.read(SocketInputStream.java:171)}}
> \{{ at java.net.SocketInputStream.read(SocketInputStream.java:141)}}
> \{{ at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)}}
> \{{ at sun.security.ssl.InputRecord.read(InputRecord.java:503)}}
> \{{ at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)}}
> \{{ - locked java.lang.Object@7b9c4e14}}
> \{{ at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)}}
> \{{ at sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1571)}}
> \{{ at
> sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1705)}}
> \{{ at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1594)}}
> \{{ at
> a.b.security.zone.api.factories.ZoneSSLSocket.close(ZoneSSLSocket.java:572)}}
> \{{ - locked a.b.security.zone.api.factories.ZoneSSLSocket@307c017c}}
> \{{ at
> a.b.http.client.sdk.HTTPDumpingSSLSocket.close(HTTPDumpingSSLSocket.java:77)}}
> \{{ - locked a.b.http.client.sdk.HTTPDumpingSSLSocket@658dab05}}
> \{{ at
> a.b.http.base.connection.MyDefaultClientConnection.close(MyDefaultClientConnection.java:711)}}
> \{{ at
> org.apache.http.impl.conn.CPoolEntry.closeConnection(CPoolEntry.java:70)}}
> \{{ at org.apache.http.impl.conn.CPoolEntry.close(CPoolEntry.java:96)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool$3.process(AbstractConnPool.java:622)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool.enumAvailable(AbstractConnPool.java:561)}}
> \{{ at org.apache.http.impl.conn.CPool.enumAvailable(CPool.java:76)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:617)}}
> \{{ at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:424)
> }}
>
> While the thread is waiting here for the close the whole connection pool is
> blocked due to some lock in the enumAvailble method. Any try to lease a new
> connection needs to wait till the lock is released here.
> The problem is that the whole pool is blocked and not only connections to
> that specific route the close is pending. Finally my whole application is
> waiting for this single partner to reply on the close.
> Is there any chance to optimze the pool to be less strict when iterating the
> connection pool?
> Here an example for a a call waiting for a connection due to the lock in the
> enumAvailable method:
> {{"pool-144-thread-13" Id=30179 in WAITING on
> lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@7dbb4bbf owned by
> IdleConnectionMonitor:MyHTTPClient Id=27798}}
> \{{ at sun.misc.Unsafe.park(Native Method)}}
> \{{ at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)}}
> \{{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)}}
> \{{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)}}
> \{{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)}}
> \{{ at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)}}
> \{{ at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:305)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)}}
> \{{ at
> org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)}}
> \{{ - locked org.apache.http.pool.AbstractConnPool$2@1e25f993}}
> \{{ at
> org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)}}
> \{{ at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304)}}
> \{{ at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)}}
> \{{ at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)}}
> \{{ at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)}}
> \{{ at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)}}
> \{{ at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)}}
> \{{ at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)}}
> \{{ at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)}}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
