[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923986&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923986
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:24
Start Date: 21/Jun/24 14:24
Worklog Time Spent: 10m
Work Description: clebertsuconic merged PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972
Issue Time Tracking
---
Worklog Id: (was: 923986)
Time Spent: 4h 10m (was: 4h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Affects Versions: 2.35.0
>Reporter: Josh Byster
>Priority: Minor
> Fix For: 2.36.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923985&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923985
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:24
Start Date: 21/Jun/24 14:24
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2177354426
I'm actually going to merge this now...
Thanks for this.
Issue Time Tracking
---
Worklog Id: (was: 923985)
Time Spent: 4h (was: 3h 50m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Affects Versions: 2.35.0
>Reporter: Josh Byster
>Priority: Minor
> Fix For: 2.36.0
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923976
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:22
Start Date: 21/Jun/24 14:22
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1645202926
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +104,11 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.compute(binding.getAddress(), (key, value) -> {
+ Collection bindingList = value == null ? new ArrayList<>() :
value;
+ bindingList.add(binding);
Review Comment:
Yeah.. I did some testing.. I thought the compute would do a loop retry only
(I read the wrong code).
I added some testing.. and I changed your code with adding sleeps and
everything worked.. I could see the wait to lock threads.
If you could rebase against main now that I added a test to validate this?
also, you can verify check style by doing this:
mvn -Pdev install -DskipTests=true -T2
Issue Time Tracking
---
Worklog Id: (was: 923976)
Time Spent: 3h 50m (was: 3h 40m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Affects Versions: 2.35.0
>Reporter: Josh Byster
>Priority: Minor
> Fix For: 2.36.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923970
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:22
Start Date: 21/Jun/24 14:22
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2177236106
I think this can be merged.. I will leave it open for a 1 or 2 days if
anyone else wants to give some feedback, but this is +1 from me.
Issue Time Tracking
---
Worklog Id: (was: 923970)
Time Spent: 3h 40m (was: 3.5h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Affects Versions: 2.35.0
>Reporter: Josh Byster
>Priority: Minor
> Fix For: 2.36.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923963&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923963
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 18/Jun/24 21:36
Start Date: 18/Jun/24 21:36
Worklog Time Spent: 10m
Work Description: joshb1050 commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2177070523
> There are several checkstyle violations that needs addressed here as well.
Sorry, Checkstyle not working locally for me. Think I have fixed the issues
now.
Issue Time Tracking
---
Worklog Id: (was: 923963)
Time Spent: 3.5h (was: 3h 20m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923960&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923960
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 18/Jun/24 21:08
Start Date: 18/Jun/24 21:08
Worklog Time Spent: 10m
Work Description: tabish121 commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2176977615
There are several checkstyle violations that needs addressed here as well.
Issue Time Tracking
---
Worklog Id: (was: 923960)
Time Spent: 3h 20m (was: 3h 10m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923950&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923950
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 18/Jun/24 19:38
Start Date: 18/Jun/24 19:38
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1644955884
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +104,11 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.compute(binding.getAddress(), (key, value) -> {
+ Collection bindingList = value == null ? new ArrayList<>() :
value;
+ bindingList.add(binding);
Review Comment:
The `directBindingMap` is a `ConcurrentHashMap` so it will guarantee thread
safety on the `compute` by locking the node.
https://stackoverflow.com/a/59601136
Also, this should be tested already via
`WildcardAddressManagerUnitTest#testConcurrentCalls`.
Issue Time Tracking
---
Worklog Id: (was: 923950)
Time Spent: 3h 10m (was: 3h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923941&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923941
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 18/Jun/24 17:19
Start Date: 18/Jun/24 17:19
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1644813299
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +104,11 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.compute(binding.getAddress(), (key, value) -> {
+ Collection bindingList = value == null ? new ArrayList<>() :
value;
+ bindingList.add(binding);
Review Comment:
say you call this from different threads...
perhaps we should add a test having mulitple threads on this call here, and
make sure the outcome is as expected.
Issue Time Tracking
---
Worklog Id: (was: 923941)
Time Spent: 3h (was: 2h 50m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923940
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 18/Jun/24 17:18
Start Date: 18/Jun/24 17:18
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1644812334
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +104,11 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.compute(binding.getAddress(), (key, value) -> {
+ Collection bindingList = value == null ? new ArrayList<>() :
value;
+ bindingList.add(binding);
Review Comment:
Maybe I'm being naive... but as far as I remember.. the ArrayList itself is
not supposed to be used from multiple threads.
The compute will do a replay, but you could still call the compute from
multiple threads.
I feel like you should have:
```
synchronized(bindingList) {
bindingsList.add(binding);
}
```
Or another data structure to be used here.
similarly on the remove the queue, it should use a synchronized or use a
concurrent equivalent.
Issue Time Tracking
---
Worklog Id: (was: 923940)
Time Spent: 2h 50m (was: 2h 40m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923431
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 14/Jun/24 05:35
Start Date: 14/Jun/24 05:35
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1639286235
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +102,8 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.computeIfAbsent(binding.getAddress(), (unused) -> new
ArrayList<>())
+ .add(binding);
Review Comment:
Now using an `ArrayList` given that the modifications are happening in a
(synchronized) `compute` block.
Issue Time Tracking
---
Worklog Id: (was: 923431)
Time Spent: 2h 40m (was: 2.5h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923430&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923430
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 14/Jun/24 05:35
Start Date: 14/Jun/24 05:35
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1639285762
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -116,7 +121,15 @@ public Binding removeBinding(final SimpleString
uniqueName, Transaction tx) thro
return null;
}
- removeBindingInternal(binding.getA().getAddress(), uniqueName);
+ SimpleString address = binding.getA().getAddress();
+ removeBindingInternal(address, uniqueName);
+ Collection directBindings = directBindingMap.get(address);
+ if (directBindings != null) {
+ directBindings.remove(binding.getA());
+ if (directBindings.isEmpty()) {
Review Comment:
I have now used `compute` which should take advantage of the striped locking
nature of the map, and should remove the race condition mentioned.
Issue Time Tracking
---
Worklog Id: (was: 923430)
Time Spent: 2.5h (was: 2h 20m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923398
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 23:11
Start Date: 13/Jun/24 23:11
Worklog Time Spent: 10m
Work Description: tabish121 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1639034848
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +102,8 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.computeIfAbsent(binding.getAddress(), (unused) -> new
ArrayList<>())
+ .add(binding);
Review Comment:
Using a copy on write structure would seem to be adding a significant amount
of GC overhead as each new binding will copy the array and toss out the old
instance.
Issue Time Tracking
---
Worklog Id: (was: 923398)
Time Spent: 2h 20m (was: 2h 10m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923397
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 22:49
Start Date: 13/Jun/24 22:49
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1639023553
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -116,7 +121,15 @@ public Binding removeBinding(final SimpleString
uniqueName, Transaction tx) thro
return null;
}
- removeBindingInternal(binding.getA().getAddress(), uniqueName);
+ SimpleString address = binding.getA().getAddress();
+ removeBindingInternal(address, uniqueName);
+ Collection directBindings = directBindingMap.get(address);
+ if (directBindings != null) {
+ directBindings.remove(binding.getA());
+ if (directBindings.isEmpty()) {
Review Comment:
Should we perhaps put this in a `ConcurrentHashMap#compute` lambda instead?
Issue Time Tracking
---
Worklog Id: (was: 923397)
Time Spent: 2h 10m (was: 2h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923396&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923396
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 22:44
Start Date: 13/Jun/24 22:44
Worklog Time Spent: 10m
Work Description: tabish121 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1639020568
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -116,7 +121,15 @@ public Binding removeBinding(final SimpleString
uniqueName, Transaction tx) thro
return null;
}
- removeBindingInternal(binding.getA().getAddress(), uniqueName);
+ SimpleString address = binding.getA().getAddress();
+ removeBindingInternal(address, uniqueName);
+ Collection directBindings = directBindingMap.get(address);
+ if (directBindings != null) {
+ directBindings.remove(binding.getA());
+ if (directBindings.isEmpty()) {
Review Comment:
Isn't there a race condition here in that the directBindings collection
might report empty here but a new binding could be added between this check and
the next call to remove the bindings entry from the Map meaning that the
tracked state is now inconsistent with reality?
Issue Time Tracking
---
Worklog Id: (was: 923396)
Time Spent: 2h (was: 1h 50m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923376&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923376
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 18:55
Start Date: 13/Jun/24 18:55
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1638740605
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +102,8 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.computeIfAbsent(binding.getAddress(), (unused) -> new
ArrayList<>())
+ .add(binding);
Review Comment:
Good point, definitely should be thread-safe. Changed to a COW ArrayList.
Issue Time Tracking
---
Worklog Id: (was: 923376)
Time Spent: 1h 50m (was: 1h 40m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923355
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 15:59
Start Date: 13/Jun/24 15:59
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1638460217
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +102,8 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.computeIfAbsent(binding.getAddress(), (unused) -> new
ArrayList<>())
+ .add(binding);
Review Comment:
you are using ConcurrentHashMap on the directBindingsMap, however you're
using a non Concurrent list on the value (new ArrayList)
I'm wondering if we shouldn't use something more protective in multi-thread
here.
I understand it's unlikely the broker to remove or add a queue on the same
address from multiple threads.. but it's not impossible. (say an application
with a MultiCast address and applications creating and deleting subscriptions
from multiple places).
it would be a rare race, but difficult to track if it ever happened.
Shouldn't we use something more protective here.. perhaps CopyOnWrite or
anything else?
Issue Time Tracking
---
Worklog Id: (was: 923355)
Time Spent: 1h 40m (was: 1.5h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923354
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 13/Jun/24 15:58
Start Date: 13/Jun/24 15:58
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1638460217
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -100,6 +102,8 @@ public boolean addBinding(final Binding binding) throws
Exception {
if (nameMap.putIfAbsent(binding.getUniqueName(), bindingAddressPair) !=
null) {
throw ActiveMQMessageBundle.BUNDLE.bindingAlreadyExists(binding);
}
+ directBindingMap.computeIfAbsent(binding.getAddress(), (unused) -> new
ArrayList<>())
+ .add(binding);
Review Comment:
you are using ConcurrentHashMap on the directBindingsMap, however you're
using a non Concurrent list on directBindingsMap.
I'm wondering if we shouldn't use something more protective in multi-thread
here.
I understand it's unlikely the broker to remove or add a queue on the same
address from multiple threads.. but it's not impossible. (say an application
with a MultiCast address and applications creating and deleting subscriptions
from multiple places).
it would be a rare race, but difficult to track if it ever happened.
Shouldn't we use something more protective here.. perhaps CopyOnWrite or
anything else?
Issue Time Tracking
---
Worklog Id: (was: 923354)
Time Spent: 1.5h (was: 1h 20m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923169&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923169
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 18:56
Start Date: 12/Jun/24 18:56
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2163705211
at least the -Ptests testsuite (complete one) ran fine.
Issue Time Tracking
---
Worklog Id: (was: 923169)
Time Spent: 1h 20m (was: 1h 10m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923164
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 17:58
Start Date: 12/Jun/24 17:58
Worklog Time Spent: 10m
Work Description: joshb1050 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1636884171
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -160,15 +172,7 @@ public Collection getMatchingBindings(final
SimpleString address) throw
@Override
public Collection getDirectBindings(final SimpleString address)
throws Exception {
SimpleString realAddress = CompositeAddress.extractAddressName(address);
- Collection bindings = new ArrayList<>();
-
- nameMap.forEach((bindingUniqueName, bindingAddressPair) -> {
- if (bindingAddressPair.getA().getAddress().equals(realAddress)) {
-bindings.add(bindingAddressPair.getA());
- }
- });
-
- return bindings;
+ return new ArrayList<>(directBindingMap.getOrDefault(realAddress,
Collections.emptyList()));
Review Comment:
It's a copy operation, and works efficiently internally (at least in JDK
17). We could do a null check instead and return a new list if null, but this
has to be a defensive copy since the tests will otherwise fail.
```
public ArrayList(Collection c) {
Object[] a = c.toArray();
if ((size = a.length) != 0) {
if (c.getClass() == ArrayList.class) {
elementData = a;
} else {
elementData = Arrays.copyOf(a, size, Object[].class);
}
} else {
// replace with empty array.
elementData = EMPTY_ELEMENTDATA;
}
}
```
Issue Time Tracking
---
Worklog Id: (was: 923164)
Time Spent: 1h 10m (was: 1h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923157
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 17:01
Start Date: 12/Jun/24 17:01
Worklog Time Spent: 10m
Work Description: tabish121 commented on code in PR #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972#discussion_r1636813616
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/SimpleAddressManager.java:
##
@@ -160,15 +172,7 @@ public Collection getMatchingBindings(final
SimpleString address) throw
@Override
public Collection getDirectBindings(final SimpleString address)
throws Exception {
SimpleString realAddress = CompositeAddress.extractAddressName(address);
- Collection bindings = new ArrayList<>();
-
- nameMap.forEach((bindingUniqueName, bindingAddressPair) -> {
- if (bindingAddressPair.getA().getAddress().equals(realAddress)) {
-bindings.add(bindingAddressPair.getA());
- }
- });
-
- return bindings;
+ return new ArrayList<>(directBindingMap.getOrDefault(realAddress,
Collections.emptyList()));
Review Comment:
It seems rather odd to return a new ArrayList that's wrapping a potentially
immutable empty list. This should probably be made a bit smarter.
Issue Time Tracking
---
Worklog Id: (was: 923157)
Time Spent: 1h (was: 50m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923156
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 16:58
Start Date: 12/Jun/24 16:58
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2163510246
@joshb1050 there's a complete test suite that you can run with -Ptests.
The CI in Jenkins runs a limited version of the test suite for basic
validation.
For more through verification I run the entire test suite but I wouldn't
have reousrces for such a long process in the public CI from GitHub.
Issue Time Tracking
---
Worklog Id: (was: 923156)
Time Spent: 50m (was: 40m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923144&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923144
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 14:17
Start Date: 12/Jun/24 14:17
Worklog Time Spent: 10m
Work Description: joshb1050 commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2163134836
@clebertsuconic I believe I have fixed, though I wasn't able to see all of
those test failures when running locally (and it shows me as the CI passing as
well).
This is needed because each time one connection disconnects, say, with a few
thousand consumers, the `TempQueueCleanerUpper` needs to clean each of these
up, and it invokes `getDirectBindings` for each—so then this ends up being `O(k
* n)` where `k` is the number of queues on the server and `n` is the number of
consumers on the connection. It completely stops the server from functioning,
and seems to be a blocker for any setup that has a few hundred thousand plus
temp queues.
Issue Time Tracking
---
Worklog Id: (was: 923144)
Time Spent: 40m (was: 0.5h)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923137&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923137
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 13:00
Start Date: 12/Jun/24 13:00
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2162952102
there are a few test failures:
Test /
org.apache.activemq.artemis.tests.integration.cli.AddressCommandTest.testForceDeleteAddressWhenExistsQueues
Test /
org.apache.activemq.artemis.tests.integration.addressing.TwoWaysRemoveAddressTest.testDeadLock
Test /
org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testSimpleMessageReceivedOnQueueManyMessagesAndInterruptTimeout
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testUnsubscribe
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testCreateSubscriber_1
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testCreateSubscriber_2
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testDurableSubscriber
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testCreateDurableSubscriber_1
Test /
org.objectweb.jtests.jms.conform.session.TopicSessionTest.testCreateDurableSubscriber_2
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateTemporaryTopicOnQueueSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateQueueOnTopicSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateBrowserOnTopicSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testUnsubscribeOnQueueSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateTemporaryQueueOnTopicSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateDurableSubscriberOnQueueSession
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateDurableConnectionConsumerOnQueueConnection
Test /
org.objectweb.jtests.jms.conform.session.UnifiedSessionTest.testCreateTopicOnQueueSession
Test /
org.objectweb.jtests.jms.conform.topic.TemporaryTopicTest.testTemporaryTopic
Test /
org.apache.activemq.artemis.tests.integration.amqp.connect.AMQPFederationAddressPolicyTest.testFederationHandlesAddressDeletedAndConsumerRecreates
Test /
org.apache.activemq.artemis.tests.integration.amqp.connect.AMQPFederationAddressPolicyTest.testRemoteBrokerClosesFederationReceiverAfterAddressRemoved
Test /
org.apache.activemq.artemis.tests.integration.cluster.distribution.SimpleSymmetricClusterTest.testDeleteAddress
Test /
org.apache.activemq.artemis.tests.integration.cluster.distribution.SymmetricClusterWithDiscoveryTest.testStartStopServers
Why you need this optimization? Are you constantly querying for the
destination on a producer or consumer?
Issue Time Tracking
---
Worklog Id: (was: 923137)
Time Spent: 20m (was: 10m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923138&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923138
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 13:00
Start Date: 12/Jun/24 13:00
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on PR #4972:
URL:
https://github.com/apache/activemq-artemis/pull/4972#issuecomment-2162952912
Why you need this optimization? are you querying the queue every time you
produce?
Issue Time Tracking
---
Worklog Id: (was: 923138)
Time Spent: 0.5h (was: 20m)
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (ARTEMIS-4814) Remove linear iteration to get direct bindings
[
https://issues.apache.org/jira/browse/ARTEMIS-4814?focusedWorklogId=923065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923065
]
ASF GitHub Bot logged work on ARTEMIS-4814:
---
Author: ASF GitHub Bot
Created on: 12/Jun/24 03:14
Start Date: 12/Jun/24 03:14
Worklog Time Spent: 10m
Work Description: joshb1050 opened a new pull request, #4972:
URL: https://github.com/apache/activemq-artemis/pull/4972
Currently, with 500K+ queues, the cleanup step of TempQueueCleanerUpper
requires invoking WildcardAddressManager#getDirectBindings, which is O(k) in
the number of queues.
From method profiling, this can consume up to 95% of our CPU time when
needing to clean up many of these.
Add a new map to keep track of the direct bindings, and add a test assertion
that fails if we don't properly remove it.
Issue Time Tracking
---
Worklog Id: (was: 923065)
Remaining Estimate: 0h
Time Spent: 10m
> Remove linear iteration to get direct bindings
> --
>
> Key: ARTEMIS-4814
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4814
> Project: ActiveMQ Artemis
> Issue Type: Task
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, with 500K+ queues, the cleanup step of {{TempQueueCleanerUpper}}
> requires invoking {{WildcardAddressManager#getDirectBindings}}, which is O(k)
> in the number of queues.
> From method profiling, this can consume up to 95% of our CPU time when
> needing to clean up many of these.
> It would be nice to make this more efficient, which shouldn't be difficult
> given the iteration just does a simple equality check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
