Best way to consuming same queue from two brokers?
Using Camel 2.10.3 My question: What is the best way to consume msgs from multiple brokers/queues and feed those messages through a single route? Explanation: I currently consume from an activemq broker. I have several different routes that consume messages from different queues on that broker. Now, the admins have updated to a network of brokers. The producers of the messages are using failover (there are only two brokers at the moment). Because those producers could place msgs on either broker, I want to check both brokers (I had a past issue on a different project where the producers failed over briefly and sent msgs to the backup that were never consumed even though I was using a failover url. My solution at the time was to always check/consume from both primary and backup - ignoring failover - using a second activemq config to the secondary broker). My problem is that now I have many routes and I want msgs from either the primary or backup jms queue to propagate through those routes without having to duplicate the route for each broker. For example, I DONT want to do this for each route (my actual routes are more complex than these): I realize I could make the route above a direct route and then consume from each broker and send to it, but this still seems ugly: I've considered combining into a single in-memory queue, but I've never done that before and would worry about robustness of the queue (I can't lose any messages). What is the smart way to do this? Thanks very much for any insight! -J -- View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
The NoB should be routing your messages for you, shouldn't it? On Wed, Oct 23, 2013 at 1:52 PM, dunnlow wrote: > Using Camel 2.10.3 > > My question: What is the best way to consume msgs from multiple > brokers/queues and feed those messages through a single route? > > Explanation: > > I currently consume from an activemq broker. I have several different > routes that consume messages from different queues on that broker. Now, the > admins have updated to a network of brokers. > > The producers of the messages are using failover (there are only two brokers > at the moment). Because those producers could place msgs on either broker, > I want to check both brokers (I had a past issue on a different project > where the producers failed over briefly and sent msgs to the backup that > were never consumed even though I was using a failover url. My solution at > the time was to always check/consume from both primary and backup - ignoring > failover - using a second activemq config to the secondary broker). > > My problem is that now I have many routes and I want msgs from either the > primary or backup jms queue to propagate through those routes without having > to duplicate the route for each broker. For example, I DONT want to do this > for each route (my actual routes are more complex than these): > > > > > > > > > > > > > > I realize I could make the route above a direct route and then consume from > each broker and send to it, but this still seems ugly: > > > > > > > > > > > > > > > > > I've considered combining into a single in-memory queue, but I've never done > that before and would worry about robustness of the queue (I can't lose any > messages). > > What is the smart way to do this? > > Thanks very much for any insight! > > -J > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100.html > Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Thanks James. Well, yes... sort of. If a producer sends to one broker, I can consume from either as expected...IF both brokers are ok. My concern (that I have from my relayed past experience) is that if I use the failover uri and there is a problem, that my consumer will miss some of the message produced. Maybe this isn't a legitimate concern (although, in my skepticism's defense, the admin who configured it expressed a lack in confidence about it's configuration). Thus, I think the safest way forward may be to point a consumer to each of the brokers. Considering more brokers may come online, what I guess there would be ideally it to funnel all of the brokers into a single endpoint from which I could process. Does that make sense or am I way overcomplicating it? Thanks again. James Carman wrote > The NoB should be routing your messages for you, shouldn't it? -- View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742105.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Remember that a NoB is not a "cluster", per se. There is no replication between brokers. So, if one broker goes down while there are messages sitting on it, then those messages are lost until the broker comes back online. If you're looking for high availability, then you're barking up the wrong tree. You need to look into a Master/Slave setup. On Wed, Oct 23, 2013 at 2:57 PM, dunnlow wrote: > Thanks James. Well, yes... sort of. If a producer sends to one broker, I > can consume from either as expected...IF both brokers are ok. My concern > (that I have from my relayed past experience) is that if I use the failover > uri and there is a problem, that my consumer will miss some of the message > produced. Maybe this isn't a legitimate concern (although, in my > skepticism's defense, the admin who configured it expressed a lack in > confidence about it's configuration). Thus, I think the safest way forward > may be to point a consumer to each of the brokers. Considering more brokers > may come online, what I guess there would be ideally it to funnel all of the > brokers into a single endpoint from which I could process. Does that make > sense or am I way overcomplicating it? Thanks again. > > > James Carman wrote >> The NoB should be routing your messages for you, shouldn't it? > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742105.html > Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Hi Dunnlow If you are using in memory queue, have you thought about using seda: instead of direct : ? seda would be able to store message since its asynchronous and can buffer up data in case other routes dont come up on time . But direct is highly synchronous. Might have data loss of consumers are not in sync with it. Reji -- View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742115.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Thanks Reji, I understand. My fear with SEDA is that I have a pretty high msg flow at times and if my app goes down and there are messages on the SEDA queue (that is, pulled off of various jms brokers/queues and waiting on a SEDA queue) the msgs will be lost. But I wonder, can a transaction work across that? For example, in a transaction, pull the msgs off of three different activemq instances and place in SEDA queue, but only consume one of the msgs from SEDA and then shut down the system. Can the transaction be maintained so that the two unconsumed msgs roll back to their respective brokers/queues? I feel some more testing coming on... :) Any thought? Thanks again, -J -- View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742134.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Check what James said about using Master/Slave to prevent messages being stuck on one broker too long in the event one of the brokers in your Active-Active NoB goes down. Also make sure that messages can make it back to their originating broker if there are no consumers with the ConditionalNetworkBridgeFilter: http://activemq.apache.org/networks-of-brokers.html If you're still skeptical about the routingYou can check both brokers like this: On Thu, Oct 24, 2013 at 10:21 AM, dunnlow wrote: > Thanks Reji, I understand. My fear with SEDA is that I have a pretty high > msg flow at times and if my app goes down and there are messages on the > SEDA > queue (that is, pulled off of various jms brokers/queues and waiting on a > SEDA queue) the msgs will be lost. But I wonder, can a transaction work > across that? For example, in a transaction, pull the msgs off of three > different activemq instances and place in SEDA queue, but only consume one > of the msgs from SEDA and then shut down the system. Can the transaction > be > maintained so that the two unconsumed msgs roll back to their respective > brokers/queues? I feel some more testing coming on... :) Any thought? > Thanks again, -J > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742134.html > Sent from the Camel - Users mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta