Justin, Thank you for the explanation. Very helpful. I strongly felt we could ignore this error as well, but here is the impact from what I am told:
“Messages are not being distributed to other nodes in the cluster. We have a feature when a call comes into one server, then we create a database entry. That entry is then sent over Artemis to notify other server, so that call rings on users logged into respective servers.” Regards, William Crowell From: Justin Bertram <jbert...@apache.org> Date: Thursday, May 30, 2024 at 4:34 PM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: AMQ222139 Let me provide a little background on what's happening behind the scenes in this circumstance... When nodes are clustered together as they are in your case then they send "notification" messages to each other to inform them of important changes. For example, when a multicast queue (or a "local queue binding" as it is referred to internally) is created on an address that matches a cluster-connection then the node on which it is created sends a BINDING_ADDED notification message to all the other nodes in the cluster. The other nodes then add what's called a "remote queue binding" internally so that later if they receive a message on that same address they will know to send it across the cluster to the queue that was just created. This functionality is what supports cluster-wide pub/sub use-cases where, for example, a JMS or MQTT subscriber on one node will receive messages published on a completely different node in the cluster. In your specific case, a node is receiving a BINDING_ADDED notification for a remote binding which it has *already* created. Typically this is due to a misconfiguration as we've already discussed, but I suppose it's possible that there's some kind of race condition related to your use-case. You can turn on TRACE logging for the following categories to see details about notifications sent and received: - org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl - org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl Based on the activity in the cluster this could be a significant amount of logging so I recommend you direct this logging to its own file. When you receive another AMQ222139 warning message in the log you can search the logs for the remote queue binding name to see who sent the notification and when it was received previously (and hopefully some details that will shed light on exactly what is happening). In any event, you still haven't explained what the impact of the AMQ222139 warning has been on your use-case (if any). Can you clarify this point? My guess is that there's really no impact and that you can just ignore the warning. Justin On Thu, May 30, 2024 at 5:31 AM William Crowell <wcrow...@perforce.com.invalid> wrote: > More information here. I did find out the queues were temporary which I > do not think this is supported in a clustered setup. > > I did notice when we browsed the queues of the three clustered nodes where > the address is same but names are different. One of the nodes which has > two addresses which are temporary. Would this cause a problem in the > distribution of messages? > > Regards, > > William Crowell > > From: William Crowell <wcrow...@perforce.com.INVALID> > Date: Tuesday, May 28, 2024 at 5:13 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: AMQ222139 > Justin, > > This randomly happens throughout the day. We are not sure what is causing > it. > > Use case: > > We have a number of accounts, and each account gets its own topic. We > create the topics on demand as accounts are created and let Artemis delete > them after 30 min of inactivity. We can have 600 accounts active at a > time, so the topic number should be around that number. I would estimate > we produce 1-3 messages per account per second which ends up being between > 50,000,00 and 200,000,000 messages per day. > > We have a 3-node Artemis cluster. > > Regards, > > William Crowell > > From: Justin Bertram <jbert...@apache.org> > Date: Tuesday, May 28, 2024 at 4:28 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: AMQ222139 > Do you have a way to reproduce this? Can you elaborate at all on the > configuration, use-case, etc. which resulted in this? What has been the > impact? > > > Justin > > On Tue, May 28, 2024 at 3:01 PM William Crowell > <wcrow...@perforce.com.invalid> wrote: > > > Justin, > > > > I do not think I have that situation: > > > > … > > <cluster-connections> > > <cluster-connection name="my-cluster"> > > <connector-ref>artemis</connector-ref> > > <message-load-balancing>ON_DEMAND</message-load-balancing> > > <max-hops>1</max-hops> > > <static-connectors> > > <connector-ref>node0</connector-ref> > > <connector-ref>node1</connector-ref> > > </static-connectors> > > </cluster-connection> > > </cluster-connections> > > … > > > > Regards, > > > > William Crowell > > > > From: Justin Bertram <jbert...@apache.org> > > Date: Tuesday, May 28, 2024 at 3:57 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: Re: AMQ222139 > > > Where would I see if I had multiple cluster connections to the same > nodes > > using overlapping addresses? Would that be in broker.xml? > > > > Yes. That would be in broker.xml. > > > > > > Justin > > > > On Tue, May 28, 2024 at 2:39 PM William Crowell > > <wcrow...@perforce.com.invalid> wrote: > > > > > Justin, > > > > > > Where would I see if I had multiple cluster connections to the same > nodes > > > using overlapping addresses? Would that be in broker.xml? > > > > > > I thought I was running into this, but we do not use temporary queues: > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595026669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=URiUIs%2F4QCh21kf8jtrWI9o%2FBTxyIOnxMuJn%2Bm6SkcQ%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595038921%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=aSsxUMDlZuVPAYsoECKC465CtNQEE3XEB5JFMxiDiNE%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967> > ><https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595042693%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=MiS5RQgfk1I39TqoMfJLKKtDlPIKvKlbfBvCBh0K7iU%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967>> > > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595045753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=DnzAoOtWL1C1FVDQVSBAeb5l7c21Nza%2Bc8aRmaaGGOA%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595048711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=nHTCaHjXe1BAZWLBscJCYOPqqAEBbwT7Qrve9myBWFk%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967> > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-1967&data=05%7C02%7CWCrowell%40perforce.com%7C84c014c39a814e0615cd08dc80e7debe%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638526980595051710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2FArLPkFn4c%2FuwyqeV1Tt6slKe1PCAqDRlLApgYslYpc%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-1967>>>> > > > > > > This is Artemis 2.33.0. > > > > > > Regards, > > > > > > William Crowell > > > > > > From: Justin Bertram <jbert...@apache.org> > > > Date: Tuesday, May 28, 2024 at 1:29 PM > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > Subject: Re: AMQ222139 > > > As far as I know the only conditions that would result in this > situation > > > are described in the warning message. > > > > > > Do you have multiple cluster connections to the same nodes using > > > overlapping addresses? > > > > > > Do you have a way to reproduce this? Can you elaborate at all on the > > > configuration, use-case, etc. which resulted in this? What has been the > > > impact? > > > > > > Lastly, what version of ActiveMQ Artemis are you using? > > > > > > > > > Justin > > > > > > On Tue, May 28, 2024 at 7:40 AM William Crowell > > > <wcrow...@perforce.com.invalid> wrote: > > > > > > > Hi, > > > > > > > > What would cause AMQ222139? I have max-hops set to 1. > > > > > > > > 2024-05-24 17:28:04,155 WARN > [org.apache.activemq.artemis.core.server] > > > > AMQ222139: MessageFlowRecordImpl > > > > [nodeID=2135063f-0407-11ef-9fff-0242ac110002, > > > > connector=TransportConfiguration(name=artemis, > > > > > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61617&host=mqtt-7727-node1-boxview-internal, > > > > > > > > > > queueName=$.artemis.internal.sf.my-cluster.2135063f-0407-11ef-9fff-0242ac110002, > > > > > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2135063f-0407-11ef-9fff-0242ac110002, > > > > postOffice=PostOfficeImpl > > > > [server=ActiveMQServerImpl::name=mqtt-7727-node1.boxview.internal], > > > > temp=false]@3548f813, isClosed=false, reset=true]::Remote queue > binding > > > > > > fa28ea36-19f2-11ef-b6bc-0242ac1100029a5d5fea-03fb-11ef-acb5-0242ac110002 > > > > has already been bound in the post office. Most likely cause for this > > is > > > > you have a loop in your cluster due to cluster max-hops being too > large > > > or > > > > you have multiple cluster connections to the same nodes using > > overlapping > > > > addresses > > > > > > > > Regards, > > > > > > > > Bill Crowell > > > > > > > > > > > > This e-mail may contain information that is privileged or > confidential. > > > If > > > > you are not the intended recipient, please delete the e-mail and any > > > > attachments and notify us immediately. > > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click on links or open attachments unless you recognize the sender and > > know > > > the content is safe. > > > > > > > > > This e-mail may contain information that is privileged or confidential. > > If > > > you are not the intended recipient, please delete the e-mail and any > > > attachments and notify us immediately. > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click on links or open attachments unless you recognize the sender and > know > > the content is safe. > > > > > > This e-mail may contain information that is privileged or confidential. > If > > you are not the intended recipient, please delete the e-mail and any > > attachments and notify us immediately. > > > > > > > CAUTION: This email originated from outside of the organization. Do not > click on links or open attachments unless you recognize the sender and know > the content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > > CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
