[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738610#comment-17738610
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Added a test:
https://github.com/apache/activemq-artemis/pull/4529
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1, 2.28.0, 2.29.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments:
> 0001-ARTEMIS-4095-fix-delivering-message-size-accounting.patch,
> artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png, graph2.png,
> jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736635#comment-17736635
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Now, knowing that the problem is in the accounting of the persistentSize of the
delivering messages, it was easy to find the reason why it keeps growing:
incrementDeliverered() size in the test scenario above is always adding more
than decrementDeliverered(). It's possible to see it when sending a single
message and consume it with a XA Consumer: the persistentCount drops to 0, but
persistenSize stays >0. Comparing the refereneces for incrementing and
decrementing it turned out that the incrementing message has "__AMQ_CID"
property set, while decrement doesn't have it. Which in turn happens because
the fix for ARTEMIS-980 has a (meanwhile?) wrong assumption about the internal
sessions:
{code:java}
//internal session always delivers messages to noLocal advisory
consumers
//so we need to remove this property too.
message.removeProperty(MessageUtil.CONNECTION_ID_PROPERTY_NAME);{code}
The assumption is wrong because internal sessions are also used for handling
the XA-Transactions. The attached patch fixes the issue for me. [~jbertram]
would it be possible to include it into the next release? And since it's a
bugfix into a minor release too?
[^0001-ARTEMIS-4095-fix-delivering-message-size-accounting.patch]
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1, 2.28.0, 2.29.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments:
> 0001-ARTEMIS-4095-fix-delivering-message-size-accounting.patch,
> artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png, graph2.png,
> jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artyom Tarasenko updated ARTEMIS-4095:
--
Attachment: 0001-ARTEMIS-4095-fix-delivering-message-size-accounting.patch
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1, 2.28.0, 2.29.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments:
> 0001-ARTEMIS-4095-fix-delivering-message-size-accounting.patch,
> artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png, graph2.png,
> jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736122#comment-17736122
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Looking further into the issue. I turned on debugging, and initially it doesn't
show anything suspicious:
2023-06-22 11:50:37,376 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0
Then I looked up how d06459df5 (ARTEMIS-3943) calculates needsDepage() :
if (... && (queueMemorySize.getSize() + deliveringMetrics.getPersistentSize())
< maxReadBytes) && ...)
The deliveringMetrics.getPersistentSize is crucial for the decision, but is
not logged by default, so I added it to the logging:
{code:java}
---
a/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.java
+++
b/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.java
@@ -4598,1 +4598,1 @@ public class QueueImpl extends CriticalComponentImpl
implements Queue {
- logger.debug("Queue Memory Size after depage on queue=" +
this.getName() + " is " + queueMemorySize.getSize() + " with maxSize = " +
maxSize + ". Depaged " + depaged + " messages, pendingDelivery=" +
messageReferences.size() + ", intermediateMessageReferences= " +
intermediateMessageReferences.size() + ", queueDelivering=" +
deliveringMetrics.getMessageCount());
+ logger.debug("Queue Memory Size after depage on queue=" +
this.getName() + " is " + queueMemorySize.getSize() + " with maxSize = " +
maxSize + ". Depaged " + depaged + " messages, pendingDelivery=" +
messageReferences.size() + ", intermediateMessageReferences= " +
intermediateMessageReferences.size() + ", queueDelivering=" +
deliveringMetrics.getMessageCount()+ ", delivering Persistent=" +
deliveringMetrics.getPersistentSize());{code}
and now the real reason is visible:
{noformat}
2023-06-22 13:29:36,261 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 9771 with maxSize = 10485760.
Depaged 1 messages, pendingDelivery=0, intermediateMessageReferences= 1,
queueDelivering=229, delivering Persistent=20962487
2023-06-22 13:29:36,261 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 9771 with maxSize = 10485760.
Depaged 1 messages, pendingDelivery=0, intermediateMessageReferences= 1,
queueDelivering=230, delivering Persistent=20969077
2023-06-22 13:29:36,372 INFO [org.apache.activemq.artemis.core.server]
AMQ224108: Stopped paging on address 'test-channel'; size=0 bytes (0 messages);
maxSize=-1 bytes (-1 messages); globalSize=0 bytes (0 messages);
globalMaxSize=419430400 bytes (-1 messages);
2023-06-22 13:29:58,667 INFO [org.apache.activemq.artemis.core.server]
AMQ222038: Starting paging on address 'test-channel'; size=419678295 bytes
(42647 messages); maxSize=-1 bytes (-1 messages); globalSize=419678295 bytes
(42647 messages); globalMaxSize=419430400 bytes (-1 messages);
2023-06-22 13:31:07,394 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0,
queueDelivering=0, delivering Persistent=24266844
2023-06-22 13:31:37,369 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0,
queueDelivering=0, delivering Persistent=24266844
2023-06-22 13:32:07,369 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0,
queueDelivering=0, delivering Persistent=24266844
2023-06-22 13:32:37,369 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0,
queueDelivering=0, delivering Persistent=24266844
2023-06-22 13:33:07,370 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-queue#test-channel is 0 with maxSize = 10485760.
Depaged 0 messages, pendingDelivery=0, intermediateMessageReferences= 0,
queueDelivering=0, delivering Persistent=24266844
2023-06-22 13:33:37,370 DEBUG
[org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue Memory Size
after depage on queue=test-qu
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705121#comment-17705121
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Using the steps above I bisected the regression:
first bad commit: [d06459df574a66c3cdd29b3ea2addf95792a8566] ARTEMIS-3943
Adjusting defalut address settings to avoid OME from paging and flow control
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png, jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704232#comment-17704232
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
The problem is also reproducible with WildFly-20 and the following consumer:
{code:java}
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.Message;
import javax.jms.MessageListener;
import org.jboss.ejb3.annotation.ResourceAdapter;
@MessageDriven(activationConfig =
{
@ActivationConfigProperty(propertyName="destinationType",
propertyValue="javax.jms.Queue"),
@ActivationConfigProperty(propertyName="destination",
propertyValue="OrdersChannel::OrdersQueue1#OrdersChannel"),
})
@ResourceAdapter("amq-ra.rar")
public class MdbConsumer implements MessageListener {
@Override
public void onMessage(Message ignored) {
// do nothing
}
}{code}
(The code for the jms consumer is the same as in the attached
[^jmsconsumer-1.0-SNAPSHOT.jar] used in the reproducer scenario above, with one
line difference as the import for ResouceAdapter annotation has been changed
after the 4.2.3 release).
The fact that the bug is reproducible not just with the ancient JBoss, but also
with modern WildFly (and likely any other AS which use activemq-5.x.rar)
probably rises the criticality of the bug?
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png, jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artyom Tarasenko updated ARTEMIS-4095:
--
Attachment: jmeter-orders-6500-localhost.jmx
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png, jmeter-orders-6500-localhost.jmx, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artyom Tarasenko updated ARTEMIS-4095:
--
Attachment: jmsconsumer-1.0-SNAPSHOT.jar
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png, jmsconsumer-1.0-SNAPSHOT.jar
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[ https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702007#comment-17702007 ] Artyom Tarasenko edited comment on ARTEMIS-4095 at 3/17/23 9:58 PM: Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps: # Download ActiveMQ Artemis 2.28.0 from [https://activemq.apache.org/components/artemis/download/] # Uncompress archive to /path/to/apache-artemis-2.28.0 # execute cd /path/to/apache-artemis-2.28.0/bin # execute JAVA_HOME=/path/to/java/11 ./artemis create /path/to/artemis-brokeruse defaultUser as username and defaultPassword as password and allow anonymous access # edit /path/to/artemis-broker/etc/broker.xml after the line 145 insert the line 400Mb # Start ActiveMQ Artemis JAVA_HOME=/path/to/java/11 /opt/osiris/artemis-broker/bin/artemis-service start # Download jboss-4.2.3.GA-jdk6.zip from https://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/jboss-4.2.3.GA-jdk6.zip # Uncompress archive to /path/to/jboss-4.2.3.GA # cd /path/to/jboss-4.2.3.GA # rm server/all/lib/javassist.jar server/default/lib/javassist.jar client/javassist.jar # Download Javassist 3.24.1-GA from [https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # cp [javassist.jar server/default/lib/|https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # Download ActiveMQ JCA RA 5.15.3 from [https://repo1.maven.org/maven2/org/apache/activemq/activemq-rar/5.16.3/activemq-rar-5.16.3.rar] # cd /path/to/jboss-4.2.3.GA/server/default/deploy # mkdir activemq-rar-5.16.3.rar # cd activemq-rar-5.16.3.rar # unzip /path/to/activemq-rar-5.16.3.rar # edit META-INF/ra.xml remove or comment out the following sections: The location of a Trust Store to use with the Connection Factory TrustStore java.lang.String The password for the Trust Store TrustStorePassword java.lang.String The location of a Key Store to use with the Connection Factory KeyStore java.lang.String The password for the Key Store KeyStorePassword java.lang.String The Key password for the Key Store KeyStoreKeyPassword java.lang.String 19. Start JBoss AS JAVA_HOME=/path/to/java/8 JBOSS_HOME=/path/to/jboss-4.2.3.GA ./run.sh 20. Copy attached MDB jmsconsumer-1.0-SNAPSHOT.jar to /path/to/jboss-4.2.3.GA/server/default/deploy 21.Download jmeter-5.5 from [https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz] 22. Uncompress archive to /path/to/apache-jmeter-5.5 23. Excecute jmeter with the attached jmeter config /path/to/apache-jmeter-5.5/bin/jmeter -n -t jmeter-orders-6500-localhost.jmx 24. ensure that Artemis started paging grep 'AMQ222038: Starting paging' /path/to/artemis-broker/log/artemis.log 25. If the message didn't appear edit the file jmeter-orders-6500-localhost.jmx and increase the value "LoopController.loops" (is set to 6500 in the attached file). Use a value large enough to start paging. 26. Wait till the message "AMQ224108: Stopped paging" appears in the /path/to/artemis-broker/log/artemis.log 27. Repeat steps 23-26 to continuously put Artemis into paging after it gets out of it. (or use the bash script from a comment above) After some attempts (2-10) Artemis won't get out of paging: the bug reproduced. was (Author: JIRAUSER298414): Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps: # Download ActiveMQ Artemis 2.28.0 from [https://activemq.apache.org/components/artemis/download/] # Uncompress archive to /path/to/apache-artemis-2.28.0 # execute cd /path/to/apache-artemis-2.28.0/bin # execute JAVA_HOME=/path/to/java/11 ./artemis create /path/to/artemis-brokeruse defaultUser as username and defaultPassword as password and allow anonymous access # edit /path/to/artemis-broker/etc/broker.xml after the line 145 insert the line 400Mb # Start ActiveMQ Artemis JAVA_HOME=/path/to/java/11 /opt/osiris/artemis-broker/bin/artemis-service start # Download JBoss AS 4.2.3 from [https://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/] # Uncompress archive to /path/to/jboss-4.2.3.GA # cd /path/to/jboss-4.2.3.GA # rm server/all/lib/javassist.jar server/default/lib/javassist.jar client/javassist.jar # Download Javassist 3.24.1-GA from [https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # cp [javassist.jar server/default/lib/|https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # Download ActiveMQ JCA RA 5.15.3
[jira] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[ https://issues.apache.org/jira/browse/ARTEMIS-4095 ]
Artyom Tarasenko deleted comment on ARTEMIS-4095:
---
was (Author: JIRAUSER298414):
I see. Running JBoss 4.2.3 with Java 8 required for {{activemq-rar-5.16.5.rar
requires quite some steps, so I'll try to reproduce it with Wildfly.}}
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[ https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702007#comment-17702007 ] Artyom Tarasenko edited comment on ARTEMIS-4095 at 3/17/23 9:55 PM: Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps: # Download ActiveMQ Artemis 2.28.0 from [https://activemq.apache.org/components/artemis/download/] # Uncompress archive to /path/to/apache-artemis-2.28.0 # execute cd /path/to/apache-artemis-2.28.0/bin # execute JAVA_HOME=/path/to/java/11 ./artemis create /path/to/artemis-brokeruse defaultUser as username and defaultPassword as password and allow anonymous access # edit /path/to/artemis-broker/etc/broker.xml after the line 145 insert the line 400Mb # Start ActiveMQ Artemis JAVA_HOME=/path/to/java/11 /opt/osiris/artemis-broker/bin/artemis-service start # Download JBoss AS 4.2.3 from [https://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/] # Uncompress archive to /path/to/jboss-4.2.3.GA # cd /path/to/jboss-4.2.3.GA # rm server/all/lib/javassist.jar server/default/lib/javassist.jar client/javassist.jar # Download Javassist 3.24.1-GA from [https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # cp [javassist.jar server/default/lib/|https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # Download ActiveMQ JCA RA 5.15.3 from [https://repo1.maven.org/maven2/org/apache/activemq/activemq-rar/5.16.3/activemq-rar-5.16.3.rar] # cd /path/to/jboss-4.2.3.GA/server/default/deploy # mkdir activemq-rar-5.16.3.rar # cd activemq-rar-5.16.3.rar # unzip /path/to/activemq-rar-5.16.3.rar # edit META-INF/ra.xml remove or comment out the following sections: The location of a Trust Store to use with the Connection Factory TrustStore java.lang.String The password for the Trust Store TrustStorePassword java.lang.String The location of a Key Store to use with the Connection Factory KeyStore java.lang.String The password for the Key Store KeyStorePassword java.lang.String The Key password for the Key Store KeyStoreKeyPassword java.lang.String 19. Start JBoss AS JAVA_HOME=/path/to/java/8 JBOSS_HOME=/path/to/jboss-4.2.3.GA ./run.sh 20. Copy attached MDB jmsconsumer-1.0-SNAPSHOT.jar to /path/to/jboss-4.2.3.GA/server/default/deploy 21.Download jmeter-5.5 from [https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz] 22. Uncompress archive to /path/to/apache-jmeter-5.5 23. Excecute jmeter with the attached jmeter config /path/to/apache-jmeter-5.5/bin/jmeter -n -t jmeter-orders-6500-localhost.jmx 24. ensure that Artemis started paging grep 'AMQ222038: Starting paging' /path/to/artemis-broker/log/artemis.log 25. If the message didn't appear edit the file jmeter-orders-6500-localhost.jmx and increase the value "LoopController.loops" (is set to 6500 in the attached file). Use a value large enough to start paging. 26. Wait till the message "AMQ224108: Stopped paging" appears in the /path/to/artemis-broker/log/artemis.log 27. Repeat steps 23-26 to continuously put Artemis into paging after it gets out of it. (or use the bash script from a comment above) After some attempts (2-10) Artemis won't get out of paging: the bug reproduced. was (Author: JIRAUSER298414): Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps: # Download ActiveMQ Artemis 2.28.0 from [https://activemq.apache.org/components/artemis/download/] # Uncompress archive to /path/to/apache-artemis-2.28.0 # execute cd /path/to/apache-artemis-2.28.0/bin # execute JAVA_HOME=/path/to/java/11 ./artemis create /path/to/artemis-brokeruse defaultUser as username and defaultPassword as password and allow anonymous access # edit /path/to/artemis-broker/etc/broker.xml after the line 145 insert the line 400Mb # Start ActiveMQ Artemis JAVA_HOME=/path/to/java/11 /opt/osiris/artemis-broker/bin/artemis-service start # Download JBoss AS 4.2.3 from [https://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/] # Uncompress archive to /path/to/jboss-4.2.3.GA # cd /path/to/jboss-4.2.3.GA # rm server/all/lib/javassist.jar server/default/lib/javassist.jar client/javassist.jar # Download Javassist 3.24.1-GA from [https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # cp [javassist.jar server/default/lib/|https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar] # Download ActiveMQ JCA RA 5.15.3 from [https://repo1.maven.org
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702007#comment-17702007
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 3/17/23 9:52 PM:
Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps:
# Download ActiveMQ Artemis 2.28.0 from
[https://activemq.apache.org/components/artemis/download/]
# Uncompress archive to /path/to/apache-artemis-2.28.0
# execute cd /path/to/apache-artemis-2.28.0/bin
# execute JAVA_HOME=/path/to/java/11 ./artemis create
/path/to/artemis-brokeruse defaultUser as username and defaultPassword as
password and allow anonymous access
# edit /path/to/artemis-broker/etc/broker.xml
after the line 145 insert the line 400Mb
# Start ActiveMQ Artemis
JAVA_HOME=/path/to/java/11 /opt/osiris/artemis-broker/bin/artemis-service start
# Download JBoss AS 4.2.3 from
[https://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/]
# Uncompress archive to /path/to/jboss-4.2.3.GA
# cd /path/to/jboss-4.2.3.GA
# rm server/all/lib/javassist.jar server/default/lib/javassist.jar
client/javassist.jar
# Download Javassist 3.24.1-GA from
[https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar]
# cp [javassist.jar
server/default/lib/|https://github.com/jboss-javassist/javassist/releases/download/rel_3_24_1_ga/javassist.jar]
# Download ActiveMQ JCA RA 5.15.3 from
[https://repo1.maven.org/maven2/org/apache/activemq/activemq-rar/5.16.3/activemq-rar-5.16.3.rar]
# cd /path/to/jboss-4.2.3.GA/server/default/deploy
# mkdir activemq-rar-5.16.3.rar
# cd activemq-rar-5.16.3.rar
# unzip /path/to/activemq-rar-5.16.3.rar
# edit META-INF/ra.xml
remove or comment out the following sections:
The location of a Trust Store to use with the
Connection Factory
TrustStore
java.lang.String
The password for the Trust Store
TrustStorePassword
java.lang.String
The location of a Key Store to use with the
Connection Factory
KeyStore
java.lang.String
The password for the Key Store
KeyStorePassword
java.lang.String
The Key password for the Key Store
KeyStoreKeyPassword
java.lang.String
# Start JBoss AS
JAVA_HOME=/path/to/java/8 JBOSS_HOME=/path/to/jboss-4.2.3.GA ./run.sh
# Copy attached MDB jmsconsumer-1.0-SNAPSHOT.jar to
/path/to/jboss-4.2.3.GA/server/default/deploy
# Download jmeter-5.5 from
[https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz]
# Uncompress archive to /path/to/apache-jmeter-5.5
# Excecute jmeter with the attached jmeter config
/path/to/apache-jmeter-5.5/bin/jmeter -n -t jmeter-orders-6500-localhost.jmx
# ensure that Artemis started paging
grep 'AMQ222038: Starting paging' /path/to/artemis-broker/log/artemis.log
# If the message didn't appear edit the file jmeter-orders-6500-localhost.jmx
and increase the value "LoopController.loops" (is set to 6500 in the attached
file).
Use a value large enough to start paging.
# Wait till the message "AMQ224108: Stopped paging" appears in the
/path/to/artemis-broker/log/artemis.log
# Repeat steps 23-26 to continuously put Artemis into paging after it gets out
of it. (or use the bash script from a comment above) After some attempts (2-10)
Artemis won't get out of paging: the bug reproduced.
was (Author: JIRAUSER298414):
Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps:
# Download ActiveMQ Artemis 2.28.0 from
[https://activemq.apache.org/components/artemis/download/]
# Uncompress archive to /path/to/apache-artemis-2.28.0
# execute cd /path/to/apache-artemis-2.28.0/bin
# execute JAVA_HOME=/path/to/java/11 ./artemis create
/path/to/artemis-brokeruse defaultUser as username and defaultPassword as
password and allow anonymous access
# edit /path/to/artemis-broker/etc/broker.xml
after the line 145 insert the line 400Mb
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd t
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702007#comment-17702007
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Reproducing with JBoss 4.2.3.GA was easier than I expected. Here are the steps:
# Download ActiveMQ Artemis 2.28.0 from
[https://activemq.apache.org/components/artemis/download/]
# Uncompress archive to /path/to/apache-artemis-2.28.0
# execute cd /path/to/apache-artemis-2.28.0/bin
# execute JAVA_HOME=/path/to/java/11 ./artemis create
/path/to/artemis-brokeruse defaultUser as username and defaultPassword as
password and allow anonymous access
# edit /path/to/artemis-broker/etc/broker.xml
after the line 145 insert the line 400Mb
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700338#comment-17700338
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
I see. Running JBoss 4.2.3 with Java 8 required for {{activemq-rar-5.16.5.rar
requires quite some steps, so I'll try to reproduce it with Wildfly.}}
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699813#comment-17699813
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 3/13/23 7:34 PM:
The steps:
# start a JCA consumer
# use JMeter to fill the queue until the message
"AMQ222038: Starting paging" appears in the artemis log
# Wait till the JCA consumer gets the messages and the message
"AMQ224108: Stopped paging" appears in artemis.log
# Go back to step 2
was (Author: JIRAUSER298414):
The steps:
# start a JCA consumer
# use JMeter to fill the queue until the message
AMQ222038: Starting pagingappears in the artemis log
# Wait till the JCA consumer gets the messages and the message
AMQ224108: Stopped paging appears in artemis.log
# Go back to step 2
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699813#comment-17699813
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 3/13/23 7:34 PM:
The steps:
# start a JCA consumer
# use JMeter to fill the queue until the message
AMQ222038: Starting pagingappears in the artemis log
# Wait till the JCA consumer gets the messages and the message
AMQ224108: Stopped paging appears in artemis.log
# Go back to step 2
was (Author: JIRAUSER298414):
The steps:
# start a JCA consumer
# use JMeter to fill the queue until the message
AMQ222038: Starting pagingappears in the artemis log
# Wait till the JCA consumer gets the messages and the message
AMQ224108: Stopped paging appears in artemis.log
# Go back to step 2
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699813#comment-17699813
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
The steps:
# start a JCA consumer
# use JMeter to fill the queue until the message
AMQ222038: Starting pagingappears in the artemis log
# Wait till the JCA consumer gets the messages and the message
AMQ224108: Stopped paging appears in artemis.log
# Go back to step 2
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699806#comment-17699806
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
I don't know if the producer/checker script has any use to you because it's
trivial:
{code:java}
$ cat torture-artemis.bash
#/bin/bash
for i in {1..32}; do
echo "Iteration ${i}"
jmeter -n -t JMSPublisherDev-test-3000-localhost.jmx
tail -1 /opt/artemis-broker/log/artemis.log |grep --color=always
'AMQ222038: Starting paging'
if [ $? -ne 0 ]; then
echo "Not enough pressure to start paging"
exit 1
fi
sleep 230
tail -1 /opt/artemis-broker/log/artemis.log | GREP_COLORS='ms=01;32'
grep --color=always 'AMQ224108: Stopped paging'
if [ $? -ne 0 ]; then
echo "either slept too less, or we have hit it"
exit 2
fi
done{code}
We use JMeter to flood the same message (the message itself doesn't seem to
matter), ensure that artemis starts paging, then wait till the JBoss/servicemix
consumer consumes messages till artemis gets out of paging mode and then we do
it again.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699802#comment-17699802
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Yes, it's JBoss 4.2.3, sorry for the confusion. And yes we are currently
moving away from it, but doing it gradually so the connection between the old
and the new system is done via Artemis.
For JMeter producer we used activemq-client-5.16.5.jar (initially
activemq-client-5.16.3.jar, no difference). In JBoss we use
activemq-rar-5.16.5.rar (initially activemq-rar-5.16.3.rar).
JBoss 4.2.3 doesn't use camel, it is still it's predecessor, ServiceMix-3.
That's why we are rewriting the old EAI-Routes completely (with camel-quarkus
fwiw, but it's not involved in this test).
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699790#comment-17699790
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Unfortunately the test script requires our application which is a JBoss 4.3.2
application with some databases attached.
What seems to be crucial for reproducing the issue is activemq-rar. I can not
reproduce the issue just using JMeter or a camel consumer. But if you can give
me a build for testing I'd be glad to test it in our environment.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699758#comment-17699758
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Tested more versions, to find out in which version the regression has happened.
The versions 2.21.0, 2.23.1 and 2.24.0 are passing my test.
The versions 2.25.0, 2.26.0, 2.27.0, 2.27.1 2.28.0 fail the test . So the
regression seems to be happened from the release 2.25.0 onwards.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699246#comment-17699246
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Still reproducible with 2.28.0, but surprisingly I can not reproduce this bug
with 2.21.0, so this seems to be a regression.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646712#comment-17646712
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
The issue looks very similar to what is described here:
[https://stackoverflow.com/questions/67157980/side-effect-of-paging-modus-in-activemq]
The difference in the setup: we use JBoss 4.3.2 with activemq-rar-5.16.5.rar
and not Weblogic as in the report above. Also for us it's reproducible without
OpenShift, just with artemis running on the same host as JBoss, to exclude the
possible network problems.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0, 2.27.1
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645406#comment-17645406
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Is also reproducible with 2.27.1
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artyom Tarasenko updated ARTEMIS-4095:
--
Attachment: artemis-Xmx2G.png
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644376#comment-17644376
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
The bug is also reproducible using just one queue, one OpenWire consumer and
jmeter configured with OpenWire (activemq-5.16.5) producer. Running with -Xmx1G
and 200Mb requires 9 iterations to trigger
the bug, !artemis-Xmx1G.png!
with -Xmx2G and 400Mb the bug is triggered
already after 5 iterations:
!artemis-Xmx2G.png!
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, artemis-Xmx2G.png, graph1.png,
> graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artyom Tarasenko updated ARTEMIS-4095:
--
Attachment: artemis-Xmx1G.png
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: artemis-Xmx1G.png, graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643185#comment-17643185
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Manually calling a single "retryMessage(long)" on the top message in the queue,
also unblocks it.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0, 2.27.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637875#comment-17637875
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 11/24/22 4:24 PM:
-
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, we redone tests with
-1 and with
jms.redeliveryPolicy.maximumRedeliveries=-1 on the client side, but it made no
difference.
Also calling retryMessages() causes thousands
WARN AMQ222188: Unable to find target queue for node null
warnings in the log, one warning per message in the stuck queue.
was (Author: JIRAUSER298414):
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, we redone tests with
-1 and with
jms.redeliveryPolicy.maximumRedeliveries=-1 on the client side, but it made no
difference.
Also calling retryMessages() causes thousands
WARN AMQ222188: Unable to find target queue for node null
warnings in the log, one warning per message in the stuck queue.
Analyzing heap dump, the stuck queue is the only
{color:#242424}org.apache.activemq.artemis.core.server.impl.QueueImpl where
qi.deliveriesInTransit.control.state=1{color}
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637875#comment-17637875
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 11/24/22 1:19 PM:
-
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, we redone tests with
-1 and with
jms.redeliveryPolicy.maximumRedeliveries=-1 on the client side, but it made no
difference.
Also calling retryMessages() causes thousands
WARN AMQ222188: Unable to find target queue for node null
warnings in the log, one warning per message in the stuck queue.
Analyzing heap dump, the stuck queue is the only
{color:#242424}org.apache.activemq.artemis.core.server.impl.QueueImpl where
qi.deliveriesInTransit.control.state=1{color}
was (Author: JIRAUSER298414):
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, it may be important that we run
with -1
Also calling retryMessages() causes thousands
WARN AMQ222188: Unable to find target queue for node null
warnings in the log, one warning per message in the stuck queue.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637875#comment-17637875
]
Artyom Tarasenko edited comment on ARTEMIS-4095 at 11/23/22 6:19 PM:
-
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, it may be important that we run
with -1
Also calling retryMessages() causes thousands
WARN AMQ222188: Unable to find target queue for node null
warnings in the log, one warning per message in the stuck queue.
was (Author: JIRAUSER298414):
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, it may be important that we run
with -1
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637875#comment-17637875
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Manually calling retryMessages() on the stuck channel fixes consuming without
restarts of Artemis or consumer. So, at least we have a workaround now.
Since retryMessages() seems to do with DLQ, it may be important that we run
with -1
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4095) OpenWire clients are unable to consume from mutlicast queue after 2nd paging
[
https://issues.apache.org/jira/browse/ARTEMIS-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637411#comment-17637411
]
Artyom Tarasenko commented on ARTEMIS-4095:
---
Also happens with 2.27.0. Once Artemis is stuck, restarting the client consumer
doesn't help. Moreover, starting another client which uses CORE protocol,
doesn't change anything, it doesn't consume either. Also switching client
consumer to other credentials doesn't change anything. After restart of Artemis
any of this client consumers can consume messages from this address, using any
of the credentials used in the previous step.
Interestingly the bug is easily reproducible with
400Mb, but after reducing the
global-max-size to 100Mb, even after 10 paging started / paging stopped cycles,
Artemis doesn't stuck.
> OpenWire clients are unable to consume from mutlicast queue after 2nd paging
>
>
> Key: ARTEMIS-4095
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4095
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: OpenWire
>Affects Versions: 2.26.0
> Environment: Artemis, deployed through the official docker image in
> version {{2.26.0}} in an OpenShift cluster
>Reporter: Marco Bungart
>Priority: Major
> Attachments: graph1.png, graph2.png
>
>
> I observed that after artemis went into paging for the 2nd time, OpenWire
> clients were not able to read messages from their corresponding addresses.
> Restarting the applications connected as clients does not fix the issue.
> Restarting artemis, however, does fix the issue.
> Both images attached show the messages of two queues.
> - The upper (orange) line in the 1st graph shows count of messages in a queue
> to which core clients are connected.
> - The lower (blue) line in the 1stg raph shows count of messages in a queue
> to which OpenWire clients are connected.
> - in the 2nd graph, the upper (violet) line shows count of messages in a
> queue to which core clients are connected.
> - in the 2nd graph, the lower (green) line shows count of messages in a queue
> to which OpenWire clients are connected.
> I have a heap dump that we could share, showing the accumulated objects. the
> dump is about 90 MB in size If this would be helpful to have, please let me
> know.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
