[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855323 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 14:40 Start Date: 06/Apr/23 14:40 Worklog Time Spent: 10m Work Description: clebertsuconic merged PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418 Issue Time Tracking --- Worklog Id: (was: 855323) Time Spent: 7h 20m (was: 7h 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855306 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 13:18 Start Date: 06/Apr/23 13:18 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1499052238 @michaelandrepearce I didn't mean as an actual hidden property. I will have a JIRA open for this that will be part of the release notes, and this is actually open source. Just that I won't add it to the documentation... as most users won't care about it.. and hopefully won't need to. Over the years I have been uncommenting code when I needed to debug things... I think it's time to make that thing as part of the codebase now. Issue Time Tracking --- Worklog Id: (was: 855306) Time Spent: 7h 10m (was: 7h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855296 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 12:40 Start Date: 06/Apr/23 12:40 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1159746076 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -16,13 +16,109 @@ */ package org.apache.activemq.artemis.api.core; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.invoke.MethodHandles; +import java.time.Instant; +import java.util.ArrayList; import java.util.HashMap; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; -// import org.apache.activemq.artemis.utils.collections.ConcurrentHashSet; Issue Time Tracking --- Worklog Id: (was: 855296) Time Spent: 7h (was: 6h 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 7h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855294 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 12:30 Start Date: 06/Apr/23 12:30 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498990212 @gemmellr I will keep the squash.. they are all related anyways. Issue Time Tracking --- Worklog Id: (was: 855294) Time Spent: 6h 50m (was: 6h 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855285 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 11:40 Start Date: 06/Apr/23 11:40 Worklog Time Spent: 10m Work Description: gemmellr commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498929649 > > The idea of this feature is to be a hidden feature... > > Hmmm this worries me, I think any feature should be something we're open about have the capability documented etc. We shouldn't be hiding things so only select few know about it. > > This is ethos of open source Apache in my mind. > > Fair enough if it's something a general user may not need to know, call it like advanced troubleshooting or something but do not add something hidden Yeah, its not a 'feature-feature' in any way, and not something I think a user would normally care about so I dont think I'd ever document it personally. Its 'additional broker developer leak debug/trace details via saved stacktraces'. Issue Time Tracking --- Worklog Id: (was: 855285) Time Spent: 6h 40m (was: 6.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855262 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 10:27 Start Date: 06/Apr/23 10:27 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1159551573 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -16,13 +16,109 @@ */ package org.apache.activemq.artemis.api.core; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.invoke.MethodHandles; +import java.time.Instant; +import java.util.ArrayList; import java.util.HashMap; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; -// import org.apache.activemq.artemis.utils.collections.ConcurrentHashSet; Issue Time Tracking --- Worklog Id: (was: 855262) Time Spent: 6.5h (was: 6h 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855258 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 10:12 Start Date: 06/Apr/23 10:12 Worklog Time Spent: 10m Work Description: michaelandrepearce commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498816803 > The idea of this feature is to be a hidden feature... > Hmmm this worries me, I think any feature should be something we're open about have the capability documented etc. We shouldn't be hiding things so only select few know about it. Fair enough call it like advanced troubleshooting or something but do not add something hidden Issue Time Tracking --- Worklog Id: (was: 855258) Time Spent: 6h 20m (was: 6h 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855243 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 09:39 Start Date: 06/Apr/23 09:39 Worklog Time Spent: 10m Work Description: gemmellr commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498773294 > The idea of this feature is to be a hidden feature... > > So, I am going to use the following logger to enable this type of debugging: > > private static final Logger refLogger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass().getName() + ".REF_DEBUG"); It will certainly be hidden, people will enable it without knowing they are since it is a descendent in the regular class name logger hierarchy. Anyone simply enabling debug logging for the class (or more likely, many classes / further up the hierarchy) will get the change in actual system behaviour of the ref counting debug and saving lots of stacktraces. They would have to know about the change in behaviour and reconfigure the specific ref count logger back off in order to restore the prior behaviour and only add debug output for the normal system behaviour. Issue Time Tracking --- Worklog Id: (was: 855243) Time Spent: 6h 10m (was: 6h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855239 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 06/Apr/23 09:36 Start Date: 06/Apr/23 09:36 Worklog Time Spent: 10m Work Description: gemmellr commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498768874 > I can squash 2 commits in one for two different and related issues. I thought it was better to keep separated. No, lets not squash 'different changes' into one commit (yes I saw, its too late), lets keep different changes separate. In that spirit, the original comment was around lets also try not to make different changes entirely reliant on each other anyway by intertwining and overlapping their multiple commits, such that you cant very simply pick or see the effect of one of the different changes in isolation because the commits were mixed together rather than sequential while adding. Aside, on the squashed commit log message, it starts telling you how to enable the ref counting debug, and then also ends telling you how to enable the ref counting debug but in a less clear way. Seems like one of them could go. Issue Time Tracking --- Worklog Id: (was: 855239) Time Spent: 6h (was: 5h 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 6h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855139 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 20:25 Start Date: 05/Apr/23 20:25 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1498100344 The idea of this feature is to be a hidden feature... So, I am going to use the following logger to enable this type of debugging: private static final Logger refLogger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass().getName() + ".REF_DEBUG"); Issue Time Tracking --- Worklog Id: (was: 855139) Time Spent: 5h 50m (was: 5h 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855138 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 20:10 Start Date: 05/Apr/23 20:10 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1158962835 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +172,13 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + protected void registerDebug() { + if (debugStatus == null) { + debugStatus = new DebugState(this.toString()); + ObjectCleaner.register(this, debugStatus); + } + } Review Comment: There's no constructor defined on this class. I'm moving it to the top of the class with a note about it. Issue Time Tracking --- Worklog Id: (was: 855138) Time Spent: 5h 40m (was: 5.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855103 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 16:53 Start Date: 05/Apr/23 16:53 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1497823526 This mechanism is how I found these issues. I would like to keep this door open for future issues. Issue Time Tracking --- Worklog Id: (was: 855103) Time Spent: 5.5h (was: 5h 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855102 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 16:52 Start Date: 05/Apr/23 16:52 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1497822645 I can squash 2 commits in one for two different and related issues. I thought it was better to keep separated. Regarding the trace. I think it's better this way. Say an user starts having issues if we just set the logger we would get what we need. Issue Time Tracking --- Worklog Id: (was: 855102) Time Spent: 5h 20m (was: 5h 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855101 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 16:26 Start Date: 05/Apr/23 16:26 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1158669203 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -16,13 +16,102 @@ */ package org.apache.activemq.artemis.api.core; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.invoke.MethodHandles; +import java.util.ArrayList; +import java.util.Calendar; +import java.util.GregorianCalendar; import java.util.HashMap; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; -// import org.apache.activemq.artemis.utils.collections.ConcurrentHashSet; Issue Time Tracking --- Worklog Id: (was: 855101) Time Spent: 5h 10m (was: 5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855100 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 16:14 Start Date: 05/Apr/23 16:14 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1158738044 ## tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/interruptlm/LargeMessageFrozenTest.java: ## @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.tests.soak.interruptlm; + +import javax.jms.Connection; +import javax.jms.ConnectionFactory; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Queue; +import javax.jms.Session; +import javax.jms.TextMessage; +import java.lang.invoke.MethodHandles; +import java.util.Locale; + +import org.apache.activemq.artemis.api.core.QueueConfiguration; +import org.apache.activemq.artemis.api.core.RoutingType; +import org.apache.activemq.artemis.core.server.ActiveMQServer; +import org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory; +import org.apache.activemq.artemis.tests.util.ActiveMQTestBase; +import org.apache.activemq.artemis.tests.util.CFUtil; +import org.apache.activemq.artemis.tests.util.TcpProxy; +import org.apache.activemq.artemis.tests.util.Wait; +import org.apache.qpid.jms.JmsConnectionFactory; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class LargeMessageFrozenTest extends ActiveMQTestBase { + + private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + TcpProxy proxy; + + ActiveMQServer server; + + @Before + public void startServer() throws Exception { + server = createServer(true, true); + server.getConfiguration().addAcceptorConfiguration("alternate", "tcp://localhost:4?amqpIdleTimeout=100"); + server.start(); + + startProxy(); + } + + private void startProxy() { + proxy = new TcpProxy("localhost", 4, 3, false); + proxy.startProxy(); + } + + @After + public void stopProxy() throws Exception { + proxy.stopProxy(5000); + } + + @Test + public void testCore() throws Exception { + testFreeze("CORE"); + } + + @Test + public void testAMQP() throws Exception { + testFreeze("AMQP"); + } + + public void testFreeze(String protocol) throws Exception { + + ConnectionFactory factory; + switch (protocol.toUpperCase(Locale.ROOT)) { + case "CORE": +ActiveMQConnectionFactory artemisfactory = new ActiveMQConnectionFactory("tcp://localhost:3?connectionTTL=1000=100=1000"); +Assert.assertEquals(100, artemisfactory.getServerLocator().getClientFailureCheckPeriod()); +Assert.assertEquals(1000, artemisfactory.getServerLocator().getConnectionTTL()); +Assert.assertEquals(1000, artemisfactory.getServerLocator().getConsumerWindowSize()); +factory = artemisfactory; +break; + case "AMQP": +JmsConnectionFactory qpidFactory = new JmsConnectionFactory("amqp://localhost:3?amqp.idleTimeout=300=2"); +factory = qpidFactory; +break; + default: +factory = CFUtil.createConnectionFactory(protocol, "tcp://localhost:3"); + } + + org.apache.activemq.artemis.core.server.Queue serverQueue = server.createQueue(new QueueConfiguration(getName()).setRoutingType(RoutingType.ANYCAST).setDurable(true)); + + Connection connection = factory.createConnection(); + runAfter(connection::close); + Session session = connection.createSession(true, Session.SESSION_TRANSACTED); + Queue queue = session.createQueue(getName()); + + Assert.assertEquals(1, proxy.getInboundHandlers().size()); + Assert.assertEquals(1,
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=855099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855099 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 05/Apr/23 16:08 Start Date: 05/Apr/23 16:08 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1158730852 ## tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/interruptlm/LargeMessageFrozenTest.java: ## @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.tests.soak.interruptlm; + +import javax.jms.Connection; +import javax.jms.ConnectionFactory; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Queue; +import javax.jms.Session; +import javax.jms.TextMessage; +import java.lang.invoke.MethodHandles; +import java.util.Locale; + +import org.apache.activemq.artemis.api.core.QueueConfiguration; +import org.apache.activemq.artemis.api.core.RoutingType; +import org.apache.activemq.artemis.core.server.ActiveMQServer; +import org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory; +import org.apache.activemq.artemis.tests.util.ActiveMQTestBase; +import org.apache.activemq.artemis.tests.util.CFUtil; +import org.apache.activemq.artemis.tests.util.TcpProxy; +import org.apache.activemq.artemis.tests.util.Wait; +import org.apache.qpid.jms.JmsConnectionFactory; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class LargeMessageFrozenTest extends ActiveMQTestBase { + + private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + TcpProxy proxy; + + ActiveMQServer server; + + @Before + public void startServer() throws Exception { + server = createServer(true, true); + server.getConfiguration().addAcceptorConfiguration("alternate", "tcp://localhost:4?amqpIdleTimeout=100"); + server.start(); + + startProxy(); + } + + private void startProxy() { + proxy = new TcpProxy("localhost", 4, 3, false); + proxy.startProxy(); + } + + @After + public void stopProxy() throws Exception { + proxy.stopProxy(5000); + } + + @Test + public void testCore() throws Exception { + testFreeze("CORE"); + } + + @Test + public void testAMQP() throws Exception { + testFreeze("AMQP"); + } + + public void testFreeze(String protocol) throws Exception { + + ConnectionFactory factory; + switch (protocol.toUpperCase(Locale.ROOT)) { + case "CORE": +ActiveMQConnectionFactory artemisfactory = new ActiveMQConnectionFactory("tcp://localhost:3?connectionTTL=1000=100=1000"); +Assert.assertEquals(100, artemisfactory.getServerLocator().getClientFailureCheckPeriod()); +Assert.assertEquals(1000, artemisfactory.getServerLocator().getConnectionTTL()); +Assert.assertEquals(1000, artemisfactory.getServerLocator().getConsumerWindowSize()); +factory = artemisfactory; +break; + case "AMQP": +JmsConnectionFactory qpidFactory = new JmsConnectionFactory("amqp://localhost:3?amqp.idleTimeout=300=2"); +factory = qpidFactory; +break; + default: +factory = CFUtil.createConnectionFactory(protocol, "tcp://localhost:3"); + } + + org.apache.activemq.artemis.core.server.Queue serverQueue = server.createQueue(new QueueConfiguration(getName()).setRoutingType(RoutingType.ANYCAST).setDurable(true)); + + Connection connection = factory.createConnection(); + runAfter(connection::close); + Session session = connection.createSession(true, Session.SESSION_TRANSACTED); + Queue queue = session.createQueue(getName()); + + Assert.assertEquals(1, proxy.getInboundHandlers().size()); + Assert.assertEquals(1,
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854569 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 03/Apr/23 15:20 Start Date: 03/Apr/23 15:20 Worklog Time Spent: 10m Work Description: tabish121 commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1156111850 ## artemis-commons/src/main/java/org/apache/activemq/artemis/utils/ObjectCleaner.java: ## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.utils; + +import java.lang.ref.Cleaner; + +public class ObjectCleaner { + + private static final Cleaner cleaner; + static { + cleaner = Cleaner.create(); + } + + public static void register(Object obj, Runnable callback) { Review Comment: An issue here with this Cleaner wrapper is that it doesn't really expose a cleaner API that is compatible with the JDK Cleaner appropriate usage. You are meant to take the returned Cleanable from the Cleaner API and actually call clean on it in most cases leaving the Cleaner to handle those rare or missed cases where a cleanable resource wasn't actively closed and "cleaned" up by the user code. Just putting every resource into a Cleaner and waiting on it to actually manage the cleanup will likely result in untimely reclaim of resources and bloat in the tracking structures. Issue Time Tracking --- Worklog Id: (was: 854569) Time Spent: 4h 40m (was: 4.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854301 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 01/Apr/23 14:35 Start Date: 01/Apr/23 14:35 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1155119097 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -58,55 +108,35 @@ public int getDurableCount() { public RefCountMessage getParentRef() { return parentRef; } - // I am usually against keeping commented out code - // However this is very useful for me to debug referencing counting. - // Uncomment out anything between #ifdef DEBUG and #endif - // #ifdef DEBUG Issue Time Tracking --- Worklog Id: (was: 854301) Time Spent: 4.5h (was: 4h 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854300 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 01/Apr/23 14:35 Start Date: 01/Apr/23 14:35 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1155119030 ## artemis-server/src/main/java/org/apache/activemq/artemis/core/server/LargeServerMessage.java: ## @@ -27,6 +27,8 @@ public interface LargeServerMessage extends ReplicatedLargeMessage { + boolean DEBUG = Boolean.parseBoolean(System.getProperty("ARTEMIS_LARGE_MESSAGE_DEBUG", "false")); Review Comment: I will just use logger.isDebugEnabled(); if RefCountMessage.logger is debug, it will be using debug Maybe I will also expose the logger from RefCountMessage with RefCountMessage::isDebugEnabled() As I will be adding a method refCountMessage.addDebug(String ); (Not sure I will keep this method in the final version after I figure out the issue I'm dealing with now... apparently a message could be duplicated after a failed AMQP Consumer in some weird race, and if I add logger.debug() the error is gone... the only way I have been able to capture this was by adding a new event to the debugState, and checking if I'm trying to remove the message twice. Issue Time Tracking --- Worklog Id: (was: 854300) Time Spent: 4h 20m (was: 4h 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854208 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 14:58 Start Date: 31/Mar/23 14:58 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154583919 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java: ## @@ -344,4 +344,7 @@ public interface ActiveMQClientLogger { @LogMessage(id = 214033, value = "Cannot resolve host ", level = LogMessage.Level.ERROR) void unableToResolveHost(UnknownHostException e); + + @LogMessage(id = 214034, value = "{} is being garbage collected without removing its resources.:\nlocations:\n{}", level = LogMessage.Level.INFO) + void debugMessageNotReleased(String clazz, String debugString); Review Comment: I hadnt actually noticed that those methods are defined in the Message interface, since RefCountMessage doesnt actually indicate it implements anything itself...but things subclass it to effectively implement the Message methods, which makes it rather hard to see whats using it and what the original impl is when the Message method is being overridden in various places. Issue Time Tracking --- Worklog Id: (was: 854208) Time Spent: 4h 10m (was: 4h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854205 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 14:20 Start Date: 31/Mar/23 14:20 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154538257 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java: ## @@ -344,4 +344,7 @@ public interface ActiveMQClientLogger { @LogMessage(id = 214033, value = "Cannot resolve host ", level = LogMessage.Level.ERROR) void unableToResolveHost(UnknownHostException e); + + @LogMessage(id = 214034, value = "{} is being garbage collected without removing its resources.:\nlocations:\n{}", level = LogMessage.Level.INFO) + void debugMessageNotReleased(String clazz, String debugString); Review Comment: I tried moving CoreMessage to the server, but that would mean having to send the Server jar to the clients... I have been there before. this was the best compromise some time ago. Issue Time Tracking --- Worklog Id: (was: 854205) Time Spent: 3h 50m (was: 3h 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854206 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 14:20 Start Date: 31/Mar/23 14:20 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154538890 ## artemis-commons/src/main/java/org/apache/activemq/artemis/utils/ObjectCleaner.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.utils; + +public class ObjectCleaner { + + public static void register(Object obj, Runnable callback) { + io.netty.util.internal.ObjectCleaner.register(obj, callback); Review Comment: this will be gone before I finish my changes. Issue Time Tracking --- Worklog Id: (was: 854206) Time Spent: 4h (was: 3h 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854204 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 14:14 Start Date: 31/Mar/23 14:14 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154530742 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java: ## @@ -344,4 +344,7 @@ public interface ActiveMQClientLogger { @LogMessage(id = 214033, value = "Cannot resolve host ", level = LogMessage.Level.ERROR) void unableToResolveHost(UnknownHostException e); + + @LogMessage(id = 214034, value = "{} is being garbage collected without removing its resources.:\nlocations:\n{}", level = LogMessage.Level.INFO) + void debugMessageNotReleased(String clazz, String debugString); Review Comment: Fair point. Still feels icky having all this code and logging in -client when it isnt really used there though. Issue Time Tracking --- Worklog Id: (was: 854204) Time Spent: 3h 40m (was: 3.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854197 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 13:30 Start Date: 31/Mar/23 13:30 Worklog Time Spent: 10m Work Description: tabish121 commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154479568 ## artemis-commons/src/main/java/org/apache/activemq/artemis/utils/ObjectCleaner.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.utils; + +public class ObjectCleaner { + + public static void register(Object obj, Runnable callback) { + io.netty.util.internal.ObjectCleaner.register(obj, callback); Review Comment: Just because we don't always like how the JDK works isn't really grounds to reinvent the wheel. The Netty 4 ObjectCleaner bits are gone in Netty 5 I believe but regardless it seems like using the built in mechanisms of the JDK would be preferable. Issue Time Tracking --- Worklog Id: (was: 854197) Time Spent: 3.5h (was: 3h 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854191 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 13:08 Start Date: 31/Mar/23 13:08 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154455666 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -58,55 +108,35 @@ public int getDurableCount() { public RefCountMessage getParentRef() { return parentRef; } - // I am usually against keeping commented out code - // However this is very useful for me to debug referencing counting. - // Uncomment out anything between #ifdef DEBUG and #endif - // #ifdef DEBUG Issue Time Tracking --- Worklog Id: (was: 854191) Time Spent: 3h 20m (was: 3h 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854189 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 13:07 Start Date: 31/Mar/23 13:07 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154454266 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java: ## @@ -344,4 +344,7 @@ public interface ActiveMQClientLogger { @LogMessage(id = 214033, value = "Cannot resolve host ", level = LogMessage.Level.ERROR) void unableToResolveHost(UnknownHostException e); + + @LogMessage(id = 214034, value = "{} is being garbage collected without removing its resources.:\nlocations:\n{}", level = LogMessage.Level.INFO) + void debugMessageNotReleased(String clazz, String debugString); Review Comment: RefCountMessage is at Client. Issue Time Tracking --- Worklog Id: (was: 854189) Time Spent: 3h 10m (was: 3h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854187 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 13:06 Start Date: 31/Mar/23 13:06 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154453055 ## artemis-commons/src/main/java/org/apache/activemq/artemis/utils/ObjectCleaner.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.utils; + +public class ObjectCleaner { + + public static void register(Object obj, Runnable callback) { + io.netty.util.internal.ObjectCleaner.register(obj, callback); Review Comment: the JDK Cleaner is using Phantom References, which they might take longer to be released. While the WeakReferences are working more reliably. I looked into writing my own cleaner but the one from Netty was ready to use :) I can always fork it. Issue Time Tracking --- Worklog Id: (was: 854187) Time Spent: 2h 50m (was: 2h 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854188 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 13:06 Start Date: 31/Mar/23 13:06 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154453055 ## artemis-commons/src/main/java/org/apache/activemq/artemis/utils/ObjectCleaner.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.artemis.utils; + +public class ObjectCleaner { + + public static void register(Object obj, Runnable callback) { + io.netty.util.internal.ObjectCleaner.register(obj, callback); Review Comment: the JDK Cleaner is using Phantom References, which they might take longer to be released. While the WeakReferences (used by Netty) are working more reliably. I looked into writing my own cleaner but the one from Netty was ready to use :) I can always fork it. Issue Time Tracking --- Worklog Id: (was: 854188) Time Spent: 3h (was: 2h 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=854165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854165 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 31/Mar/23 12:12 Start Date: 31/Mar/23 12:12 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1154381133 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -16,12 +16,57 @@ */ package org.apache.activemq.artemis.api.core; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.invoke.MethodHandles; +import java.util.ArrayList; import java.util.HashMap; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; -// import org.apache.activemq.artemis.utils.collections.ConcurrentHashSet; Issue Time Tracking --- Worklog Id: (was: 854165) Time Spent: 2h 40m (was: 2.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853818 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 30/Mar/23 03:02 Start Date: 30/Mar/23 03:02 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#issuecomment-1489616826 I have been debugging with these messages, and I am making a few adjustments (Missed debug statements... etc)... this is being useful in a scenario where the Cpu Is overloaded and everything is failing on the delivering, leaving messages behind. it is working as I expected to tell me what is happening, but I'm making a few adjustments. Issue Time Tracking --- Worklog Id: (was: 853818) Time Spent: 2.5h (was: 2h 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853761 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 17:58 Start Date: 29/Mar/23 17:58 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152305657 ## tests/leak-tests/pom.xml: ## @@ -246,7 +246,8 @@ maven-surefire-plugin ${skipLeakTests} - -agentpath:${project.basedir}/target/lib/${check-leak-deploy-name} -Djgroups.bind_addr=::1 ${activemq-surefire-argline} -Dorg.apache.activemq.SERIALIZABLE_PACKAGES="java.lang,javax.security,java.util,org.apache.activemq,org.fusesource.hawtbuf" +Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853759 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 17:57 Start Date: 29/Mar/23 17:57 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152304366 ## tests/soak-tests/pom.xml: ## @@ -260,7 +260,7 @@ ${basedir}/target/classes/servers/interruptlm --java-options --Djava.rmi.server.hostname=localhost +-Djava.rmi.server.hostname=localhost -DARTEMIS_DEBUG_REF=true Review Comment: This was for the server running only. not the actual test.. But I have removed it anyway. Issue Time Tracking --- Worklog Id: (was: 853759) Time Spent: 2h 10m (was: 2h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853758 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 17:56 Start Date: 29/Mar/23 17:56 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152303303 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +93,28 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + /** Sub classes may mark fired=true when they were explicitly called. +* E.g large message removed the file upon cancellation */ + protected void markFired() { + fired = true; Review Comment: The first variable is used to control the releaseResources... The second variable is used to control if the resource leaked or not. I'm renaming it to leaked. and I can't read anything from RefCountMessage coming from RefCountDebugState (I just renamed it BTW). Having a reference back to RefCountMessage would make it impossible to release the reference. and I don't want to complicate things with a WeakReference... I would rather duplicate the variable... in this case I renamed it to leaked to make a distinction. Issue Time Tracking --- Worklog Id: (was: 853758) Time Spent: 2h (was: 1h 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853720 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 15:07 Start Date: 29/Mar/23 15:07 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152099042 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +93,28 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + /** Sub classes may mark fired=true when they were explicitly called. +* E.g large message removed the file upon cancellation */ + protected void markFired() { + fired = true; Review Comment: So is the idea that during the 'normal' countDown it marks the 'outer fired' variable, reaches out somewhere else in firing releaseComplete(), then that other places [maybe] reaches back in here eventually and calls 'markFired()' and doubly-sets the 'outer fired' variable and ALSO sets the separate 'debugStatus fired' variable to _really_ mark the message as cleaned up? The cleaner only seems to look at the latter variable when deciding whether the message was cleaned up or not, but only the markFired() 'direct call' is setting that. I looked at/thought about this code for 30mins earlier and am asking questions as I still dont understand what its expected to do normally. Its pretty impenetrable as-is. If I wasnt seeing the diff I probably wouldnt even have noticed the 2 different 'fired' variables, with one being used in 2 different ways by the two routes through the code. Issue Time Tracking --- Worklog Id: (was: 853720) Time Spent: 1h 50m (was: 1h 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853715 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 14:48 Start Date: 29/Mar/23 14:48 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152067659 ## tests/soak-tests/pom.xml: ## @@ -260,7 +260,7 @@ ${basedir}/target/classes/servers/interruptlm --java-options --Djava.rmi.server.hostname=localhost +-Djava.rmi.server.hostname=localhost -DARTEMIS_DEBUG_REF=true Review Comment: Its no longer soak testing the normal use case at all as a result though, that just seems wrong. Issue Time Tracking --- Worklog Id: (was: 853715) Time Spent: 1h 40m (was: 1.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853714 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 14:46 Start Date: 29/Mar/23 14:46 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152063360 ## tests/leak-tests/pom.xml: ## @@ -246,7 +246,8 @@ maven-surefire-plugin ${skipLeakTests} - -agentpath:${project.basedir}/target/lib/${check-leak-deploy-name} -Djgroups.bind_addr=::1 ${activemq-surefire-argline} -Dorg.apache.activemq.SERIALIZABLE_PACKAGES="java.lang,javax.security,java.util,org.apache.activemq,org.fusesource.hawtbuf" +Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853712 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 14:40 Start Date: 29/Mar/23 14:40 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152053054 ## tests/leak-tests/pom.xml: ## @@ -246,7 +246,8 @@ maven-surefire-plugin ${skipLeakTests} - -agentpath:${project.basedir}/target/lib/${check-leak-deploy-name} -Djgroups.bind_addr=::1 ${activemq-surefire-argline} -Dorg.apache.activemq.SERIALIZABLE_PACKAGES="java.lang,javax.security,java.util,org.apache.activemq,org.fusesource.hawtbuf" +Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853711 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 14:36 Start Date: 29/Mar/23 14:36 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152046672 ## tests/soak-tests/pom.xml: ## @@ -260,7 +260,7 @@ ${basedir}/target/classes/servers/interruptlm --java-options --Djava.rmi.server.hostname=localhost +-Djava.rmi.server.hostname=localhost -DARTEMIS_DEBUG_REF=true Review Comment: I actually wanted to have the soak tests to provide information about leaked messages if the test failed. Issue Time Tracking --- Worklog Id: (was: 853711) Time Spent: 1h 10m (was: 1h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853710 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 14:35 Start Date: 29/Mar/23 14:35 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1152045934 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +93,28 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + /** Sub classes may mark fired=true when they were explicitly called. +* E.g large message removed the file upon cancellation */ + protected void markFired() { + fired = true; Review Comment: if the system called the fire method directly, say, a cancellation from a bad producer, there's no need to call the fire event again on countDown... (which might not actually happen since the message is going to be GCed). I need the variable in the status as well as I can't have access to the message from the status. Issue Time Tracking --- Worklog Id: (was: 853710) Time Spent: 1h (was: 50m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853663 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 29/Mar/23 11:24 Start Date: 29/Mar/23 11:24 Worklog Time Spent: 10m Work Description: gemmellr commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1151687346 ## tests/soak-tests/pom.xml: ## @@ -260,7 +260,7 @@ ${basedir}/target/classes/servers/interruptlm --java-options --Djava.rmi.server.hostname=localhost +-Djava.rmi.server.hostname=localhost -DARTEMIS_DEBUG_REF=true Review Comment: Again, seems odd to have the soak tests default to running things in a typically non-representative manner. Perhaps more so in this case than the leak-tests, given none of these tests are actually even looking at/using the effect of the config. Perhaps just add some mechanism to do this easily on request, e.g a property that defaults empty so you can ask the tests to do this without modifying them? ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -16,12 +16,56 @@ */ package org.apache.activemq.artemis.api.core; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.invoke.MethodHandles; +import java.lang.ref.Cleaner; +import java.util.ArrayList; import java.util.HashMap; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; -// import org.apache.activemq.artemis.utils.collections.ConcurrentHashSet; Issue Time Tracking --- Worklog Id: (was: 853663) Time Spent: 50m (was: 40m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853535 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 28/Mar/23 23:55 Start Date: 28/Mar/23 23:55 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1151264320 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +76,28 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + /** Sub classes may mark fired=true when they were explicitly called. +* E.g large message removed the file upon cancellation */ + protected void markFired() { + fired = true; + if (firedAtomic != null) { + firedAtomic.set(true); + } + } + + public RefCountMessage() { + if (DEBUG_REF_COUNT) { + AtomicBoolean refFired = this.firedAtomic; + List crumbs = debugCrumbs; + crumbs.add(new Exception("new Instance")); + String clazz = this.getClass().getName(); + // notice you can't have a direct reference on the LargeMessage Itself, + // Otherwise you would get a leak from itself + // RefCountMessageLeakTest is validating this scenario + CLEANER.register(this, () -> debug(clazz, refFired, crumbs)); Review Comment: I actually amended this with a State Object. ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/Test.java: ## Review Comment: removed... it was a mistake. Issue Time Tracking --- Worklog Id: (was: 853535) Time Spent: 40m (was: 0.5h) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853533 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 28/Mar/23 22:57 Start Date: 28/Mar/23 22:57 Worklog Time Spent: 10m Work Description: tabish121 commented on code in PR #4418: URL: https://github.com/apache/activemq-artemis/pull/4418#discussion_r1151224880 ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -49,6 +76,28 @@ public int getDurableCount() { return DURABLE_REF_COUNT_UPDATER.get(this); } + /** Sub classes may mark fired=true when they were explicitly called. +* E.g large message removed the file upon cancellation */ + protected void markFired() { + fired = true; + if (firedAtomic != null) { + firedAtomic.set(true); + } + } + + public RefCountMessage() { + if (DEBUG_REF_COUNT) { + AtomicBoolean refFired = this.firedAtomic; + List crumbs = debugCrumbs; + crumbs.add(new Exception("new Instance")); + String clazz = this.getClass().getName(); + // notice you can't have a direct reference on the LargeMessage Itself, + // Otherwise you would get a leak from itself + // RefCountMessageLeakTest is validating this scenario + CLEANER.register(this, () -> debug(clazz, refFired, crumbs)); Review Comment: Even with the comment I'm not a huge fan of the lambda being used here because it is super easy for someone to later change this to capture 'this' inadvertently. I'd recommend passing the state data to a static function that returns a Runnable just as an additional safety mechanism as that prevents a capture of 'this' in the Runnable. ## artemis-core-client/src/main/java/org/apache/activemq/artemis/api/core/RefCountMessage.java: ## @@ -58,54 +107,42 @@ public int getDurableCount() { public RefCountMessage getParentRef() { return parentRef; } - // I am usually against keeping commented out code - // However this is very useful for me to debug referencing counting. - // Uncomment out anything between #ifdef DEBUG and #endif - // #ifdef DEBUG Issue Time Tracking --- Worklog Id: (was: 853533) Time Spent: 0.5h (was: 20m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=853524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853524 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 28/Mar/23 21:33 Start Date: 28/Mar/23 21:33 Worklog Time Spent: 10m Work Description: clebertsuconic opened a new pull request, #4418: URL: https://github.com/apache/activemq-artemis/pull/4418 you add the property -DARTEMIS_DEBUG_REF=true in your java arguments, And you should see Debug output when the messages are not being removed. Issue Time Tracking --- Worklog Id: (was: 853524) Time Spent: 20m (was: 10m) > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4206) Unreferenced AMQP Large Messages are not removed
[ https://issues.apache.org/jira/browse/ARTEMIS-4206?focusedWorklogId=851137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-851137 ] ASF GitHub Bot logged work on ARTEMIS-4206: --- Author: ASF GitHub Bot Created on: 15/Mar/23 12:57 Start Date: 15/Mar/23 12:57 Worklog Time Spent: 10m Work Description: clebertsuconic merged PR #4400: URL: https://github.com/apache/activemq-artemis/pull/4400 Issue Time Tracking --- Worklog Id: (was: 851137) Remaining Estimate: 0h Time Spent: 10m > Unreferenced AMQP Large Messages are not removed > > > Key: ARTEMIS-4206 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4206 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.29.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Say you crashed the server after the ack, and before the file.remove, and the > journal. record removal. > The AMQP Large Message may not be removed right away, requiring a restart of > the broker. > At this point this is really caused by ARTEMIS-4193 and only affected 2.29.0 > and no previous versions -- This message was sent by Atlassian Jira (v8.20.10#820010)