[ https://issues.apache.org/jira/browse/QPID-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767683#comment-16767683 ]
Alex Rudyy commented on QPID-8276: ---------------------------------- The fixes were committed against wrong JIRA: Commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b in qpid-broker-j's branch refs/heads/master from Alex Rudyy [ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=94de25e ] QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream Commit ae6fecffad5edb4abde74f1e4d2e2060702523f2 in qpid-broker-j's branch refs/heads/7.1.x from Alex Rudyy [ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=ae6fecf ] QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream (cherry picked from commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b) Commit 2e72e832d9b8274cc09f7f1cfba313a40f8cc0da in qpid-broker-j's branch refs/heads/7.0.x from Alex Rudyy [ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=2e72e83 ] QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream (cherry picked from commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b) > [Broker-J] Broker can leak closed NonBlockingConnection objects and > eventually run out of heap memory > ----------------------------------------------------------------------------------------------------- > > Key: QPID-8276 > URL: https://issues.apache.org/jira/browse/QPID-8276 > Project: Qpid > Issue Type: Bug > Components: Broker-J > Affects Versions: qpid-java-broker-7.0.3, qpid-java-broker-7.0.2, > qpid-java-broker-7.0.0, qpid-java-broker-7.0.1, qpid-java-6.1.7, > qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5, > qpid-java-broker-7.0.6 > Reporter: Alex Rudyy > Assignee: Alex Rudyy > Priority: Critical > Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1 > > > The Qpid Broker-J can leak closed NonBlockingConnection objects. > The heap dump analysis of impacted broker instance revealed that leaked > {{NonBlockingConnection}} objects are accumulated in > {{SelectorThread.SelectionTask#_unscheduledConnections}} belonging to AMQP > port IO pool. They have no ticker set and no state changed flag set > ({{NonBlockingConnection#isStateChanged() == false)}}. As result, the > NonBlockingConnection objects are not removed from > {{SelectorThread#_unscheduledConnections}} on invocation of > {{SelectorThread.SelectionTask#processUnscheduledConnections()}} called from > {{SelectorThread.SelectionTask#performSelect()}}. > The {{NonBlockingConnection}} and underlying model object are in closed state. > It seems that leaked {{NonBlockingConnection}} was closed as part of > invocation {{NonBlockingConnection#doWork()}}. The connection was > unregistered on {{VirtualHost}} IO pool and re-registered with port IO pool > as part of invocation {{NetworkConnectionScheduler#processConnection}} At > first, it was stored in collection > {{SelectorThread.SelectionTask#_unregisteredConnections}}. Later on, it was > moved from {{SelectorThread.SelectionTask#_unregisteredConnections}} to > {{SelectorThread.SelectionTask#_unscheduledConnections}} as part of > invocation {{SelectorThread.SelectionTask#reregisterUnregisteredConnections}} > and stack there afterwards. > The TLS transport was used in leaked connection, but, I think that connection > with plain transport can be leaked as well. > I suspect that connections were leaked in result of following scenario: > * Invocation of {{SocketChannel#read(java.nio.ByteBuffer[])}} returned > {{-1}} in {{NonBlockingConnection#readFromNetwork}}. > * The flag {{NonBlockingConnection#_closed}} was set to {{true}}. The method > {{ProtocolEngine#notifyWork()}} was not invoked to set {{state changed}} flag > to {{true}} > * The execution of {{NonBlockingConnection#doWork()}} ended up it connection > shutdown (due to {{NonBlockingConnection#_closed}} being set) and following > re-scheduling the connection on port IO scheduler. The latter resulted in > connection being put into > {{SelectorThread.SelectionTask#_unscheduledConnections}} as described above. > It seems that opening and closing frequent connections with connection life > span {{>10s}} (required for tickers to be removed) can ended-up in > connections being leaked as described in scenario above. It looks like > connections which are closed orderly or closed in result of {{IOException}} > being thrown from socket read/write operation are not effected by the defect. > The impacted broker instance can eventually crash with out of memory error. > Broker memory monitoring and periodic broker restarts can mitigate the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org