Hi Justin,
Thanks for the response!
Our use case is something like this. Suppose we have a nice RPC client wrapped
around message queues and their services. Some methods take a long time, and
we want to report status along the way, so we’ve arranged for such methods to
return a “task ID”:
String taskID = client.startLongThing();
while (!client.isLongThingFinished(taskID)) { wait, or print progress… }
client.getLongThingResult(taskID);
The problem we hit is that the long-running tasks are not stateless. They
start, execute and finish within the same server process. So the calls to
startLongThing(), isLongThingFinished(), and getLongThingResult() must all be
serviced by the same process. If we have multiple HA instances of a service
running in a Kubernetes cluster, the isLongThingFinished() requests will be
handle by a random server that may not know about the task.
Do you think that the “message group” attribute is a reasonable approach? In
JMS:
message.setStringProperty("JMSXGroupID", "grouplabel");
I’ve implemented and tested this and it does seem to work. Do you know of any
caveats or cautions?
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> |
[email protected]<mailto:[email protected]>
From: Justin Bertram <[email protected]>
Sent: Thursday, December 7, 2023 10:56 PM
To: [email protected]
Subject: Re: Is it possible to route message to a specific consumer?
*** [Caution] This email is from an external source. Please use caution
responding, opening attachments or clicking embedded links. ***
> Is there any kind of “consumer affinity” that can be attached to a message,
> which directs the broker to select a specific consumer?
I may be forgetting something, but I believe the answer to this question is no.
The broker is broadly designed to support decoupled, asynchronous messaging
where consumers and producers know essentially nothing about each other and the
broker just receives messages and makes them available for consumption. Message
consumption is fundamentally consumer-driven. The broker itself doesn't choose
where the messages go except in some limited sense in particular use-cases.
Even in the case of message grouping, the messages are not directed to a
specific consumer as much as they are just directed at _just one consumer (i.e.
any consumer) at a time_ since the goal is just for the messages to be
processed in order.
You can add specific properties to specific messages and then JMS consumers can
use selectors to either ignore or choose those messages for consumption and in
this way you can ensure that certain consumers get certain messages, but both
the producer and consumer need to know about the properties. Furthermore, if
all the applications use a single queue then they can start interfering with
each other. For example, if one application fails to consume its messages then
messages will begin accumulating and may prevent other applications from
getting their messages since queues are by nature first-in-first-out data
structures.
I'd need to know more about the specifics of your use-case before I could make
further recommendations.
Justin
On Thu, Dec 7, 2023 at 1:26 PM John Lilley
<[email protected]<mailto:[email protected]>>
wrote:
Oooh, maybe I answered my own question:
https://activemq.apache.org/components/artemis/documentation/1.0.0/message-grouping.html
This looks like what I’m after. Any advice around its use?
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,RLCz7uLdUVWU-4PdO0mzCFR75gph7wgFAJpvnoG5yg55RbBDZxZfAUVZs-ERB9VGZHVUZK6w6h0r3FI5bjkHA50Th_z9pS5huRB5AZIUhxN5nTIQlTbVa2sLLA,,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> |
[email protected]<mailto:[email protected]>
From: John Lilley
<[email protected]<mailto:[email protected]>>
Sent: Thursday, December 7, 2023 11:58 AM
To: [email protected]<mailto:[email protected]>
Subject: Is it possible to route message to a specific consumer?
*** [Caution] This email is from an external source. Please use caution
responding, opening attachments or clicking embedded links. ***
Greetings,
We are in the process of transforming our Java-based services into
Kubernetes-based microservice swarms with HA topology. So every service will
have multiple instances. The good news is, AMQ supports this splendidly,
because the message broker is very good about load-balancing between consumers
and handling failed consumers.
However, we have a few cases of “long running tasks” that a single service
instance is responsible for. In those cases, we need to be able to route a
request to a specific consumer instance. We could use specially-named queues
for each consumer instance, but this seems awkward and hopefully not necessary.
Is there any kind of “consumer affinity” that can be attached to a message,
which directs the broker to select a specific consumer?
If not, other ideas are certainly welcome.
We are using OpenJDK 17
This is our maven dependency
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>artemis-jms-client-all</artifactId>
<version>2.30.0</version>
</dependency>
Thanks
John
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
John Lilley
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
Data Management Chief Architect, Redpoint Global Inc.
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
M: +1 7209385761 | [email protected]
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential
and is intended solely for the use of the individual(s) to whom it is
addressed. If you believe you received this e-mail in error, please notify the
sender immediately, delete the e-mail from your computer and do not copy, print
or disclose it to anyone else. If you properly received this e-mail as a
customer, partner or vendor of Redpoint, you should maintain its contents in
confidence subject to the terms and conditions of your agreement(s) with
Redpoint.
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential
and is intended solely for the use of the individual(s) to whom it is
addressed. If you believe you received this e-mail in error, please notify the
sender immediately, delete the e-mail from your computer and do not copy, print
or disclose it to anyone else. If you properly received this e-mail as a
customer, partner or vendor of Redpoint, you should maintain its contents in
confidence subject to the terms and conditions of your agreement(s) with
Redpoint.
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential
and is intended solely for the use of the individual(s) to whom it is
addressed. If you believe you received this e-mail in error, please notify the
sender immediately, delete the e-mail from your computer and do not copy, print
or disclose it to anyone else. If you properly received this e-mail as a
customer, partner or vendor of Redpoint, you should maintain its contents in
confidence subject to the terms and conditions of your agreement(s) with
Redpoint.
