[
https://issues.apache.org/jira/browse/ARTEMIS-4797?focusedWorklogId=922592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922592
]
ASF GitHub Bot logged work on ARTEMIS-4797:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Jun/24 14:29
Start Date: 07/Jun/24 14:29
Worklog Time Spent: 10m
Work Description: joshb1050 commented on PR #4960:
URL:
https://github.com/apache/activemq-artemis/pull/4960#issuecomment-2154967424
@gemmellr That's possible...though I don't know how else to handle this. The
`connectionDestroyed` listener needs to be called for these but they aren't,
and we leak memory in the case where the exception callback is called first.
Issue Time Tracking
-------------------
Worklog Id: (was: 922592)
Time Spent: 40m (was: 0.5h)
> Failover connection references are not always cleaned up in NettyAcceptor,
> leaking memory
> -----------------------------------------------------------------------------------------
>
> Key: ARTEMIS-4797
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4797
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
> Reporter: Josh Byster
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I'm still trying to parse through exactly what conditions this occurs in,
> since I'm able to reproduce it in a very specific production setup but not in
> an isolated environment locally.
> For context, we have custom slow consumer detection that closes connection
> IDs with slow consumers. These connections are connected via failover
> transport using client ActiveMQ Classic 5.16.4 (OpenWire). This seems to be
> specific to Netty.
> It appears this specific order of events causes the connection to not get
> cleaned up and retained indefinitely on the broker. With frequent kicking of
> connections, this ends up causing the broker to eventually OOM.
> 1. Connection is created, {{ActiveMQServerChannelHandler}} is created as well
> 2. {{ActiveMQServerChannelHandler#createConnection}} is called, {{active}}
> flag is set {{true}}.
> 3. A few minutes go by, then we call
> {{ActiveMQServerControl#closeConnectionWithID}} with the connection ID.
> 4. {{ActiveMQChannelHandler#exceptionCaught}} gets called—*this is the key
> point that causes issues*. The connection is cleaned up if and only if this
> is *not* called. The root cause of the exception is
> {{AbstractChannel.close(ChannelPromise)}}, however the comment above it says
> this is normal for failover.
> 5. The {{active}} flag is set to {{false}}.
> 6. {{ActiveMQChannelHandler#channelInactive}} gets called, but does *not*
> call {{listener.connectionDestroyed}} since the {{active}} flag is false.
> 7. The connection is never removed from the {{connections}} map in
> {{NettyAcceptor}}, causing a leak and eventual OOM of the broker if it
> happens frequently enough.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
