Justin,
> Where exactly do you see the re-ordering of messages and how are you
> determining that they are, in fact, re-ordered? Are they re-ordered in the
> target queue itself or on the consumer(s)?
My test-suite has a single-threaded amqp-producer that produces a few
hundred-thousand sequentially numbered messages on the first node of a regular
cluster; and
there is a single threaded amqp-consumer that consumes these messages from the
second node. There are no other nodes (I reduced the cluster to these 2 nodes).
The consumer validates the sequence number.
I currently have no tools/tests to inspect the messages while they are inside
Artemis.
Note that due to the amount of messags, paging is in effect on both brokers.
> [sample]
Here is a sample from the log from my test-case:
(all messages before this are properly delivered in sequence)
WARNING: recv#1: expected message 'msg#3955', but got 'msg#3962'
WARNING: recv#1: expected message 'msg#3956', but got 'msg#3955'
WARNING: recv#1: expected message 'msg#3957', but got 'msg#3956'
WARNING: recv#1: expected message 'msg#3958', but got 'msg#3957'
WARNING: recv#1: expected message 'msg#3959', but got 'msg#3958'
WARNING: recv#1: expected message 'msg#3960', but got 'msg#3959'
WARNING: recv#1: expected message 'msg#3961', but got 'msg#3960'
WARNING: recv#1: expected message 'msg#3962', but got 'msg#3961'
(roughly the next 2000 messages are properly processed in sequence
again)
Message number 3962 appears too early by, and a few messages later it is
in-sync again.
The exact sequence number at which the effect appears is (wildly) different on
each run.
> Do you have a way that I can reproduce this behavior?
My test-suite is too large to provide directly, and it also contains
customer-specific scenarios.
I'll spend some time to create a reproducer.
But first, I'll also redo the test using a single node.
Erwin
-----Original Message-----
From: Justin Bertram <[email protected]>
Sent: Friday, May 3, 2024 8:57 PM
To: [email protected]
Subject: Re: question on message-redistribution in an Artemis cluster
EXTERNAL SENDER: Do not click any links or open any attachments unless you
trust the sender and know the content is safe.
EXPÉDITEUR EXTERNE: Ne cliquez sur aucun lien et n’ouvrez aucune pièce
jointe à moins qu’ils ne proviennent d’un expéditeur fiable, ou que vous ayez
l'assurance que le contenu provient d'une source sûre.
Where exactly do you see the re-ordering of messages and how are you
determining that they are, in fact, re-ordered? Are they re-ordered in the
target queue itself or on the consumer(s)? Do you have a way that I can
reproduce this behavior?
If I recall correctly there is just one thread that forwards messages from the
internal store-and-forward queue to the target node so I don't think that would
be the reason for this behavior.
Justin
On Fri, May 3, 2024 at 10:02 AM Dondorp, Erwin <[email protected]>
wrote:
> Intern
>
> Hello,
>
> I have an Artemis cluster with several nodes.
> In many cases, messages arrive on one node and are consumed from
> another node.
> The cluster-connections between the nodes do their work to make that
> happen.
>
> But under high-load we see messages being re-ordered between the
> broker-nodes, and I am trying to minimize (or eliminate) that effect.
> The re-ordering is usually only within a small set of messages, e.g.
> within a set of 10 to 20 messages.
> My assumption is that this is due to the use of multiple threads in
> the cluster-connections when there are a lot of messages.
> Is there a way to have more control over this?
>
> thx!
> Erwin
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor
> (gebruik door) de geadresseerde. De e-mail kan persoonlijke of
> vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging,
> verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en
> eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan.
> Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk
> verzocht degene die de e-mail verzond hiervan direct op de hoogte te
> brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie
> vennootschap<https://urldefense.com/v3/__http://www.ns.nl/emaildisclai
> mer__;!!AaIhyw!psLFP82x4zK7ZZ2qL1XdXWq58ZnIo5ZhoCIFSQ0P34fxf7Ik5pR9mjB
> IiZm5FSjzVGxGj1gD0wBPk-uRngk$ >
>
>
> Intern
>
