I tried setting the prefetchSize to 1 for both consumers, but unfortunately, that didn't resolve the issue. Messages still end up stuck on B2 when C2 is stopped.
On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote: > set the prefetchSize=1 on your network - to avoid orphaned messages on > B2 > > On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote: > > > Hmm, that seems like a flawed approach if the broker it visited has > > changed state and is now eligible to process those messages. How is > > this type of fail over intended to be handled with activemq? Is there > > an alternate solution that I'm missing? > > > > On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote: > >> > >> I think that by design a message will not be forwarded to a broker > >> that it > >> has already visited. > >> > >> Joe > >> http://www.ttmsolutions.com > >> > >> > >> Eric Van wrote: > >>> > >>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing) > >>> > >>> I'm running into an issue with the store and forward feature of > >>> activemq, which I using in an attempt to create a highly available > >>> distributed queue. I'm trying to figure out if the issue is a > >>> misconfiguration on my part, expected behavior of activemq, or a > >>> bug. > >>> > >>> The summary of the problem is that given 2 brokers, B1 and B2, which > >>> each have one consumer, C1 and C2, which are subscribed to the same > >>> queue. If I stop a consumer on one of the brokers, the pending > >>> messages from that broker are not always forwarded to the other > >>> broker > >>> which still has a consumer, leading to those messages getting > >>> indefinitely stuck. > >>> > >>> The steps I use to reproduce this scenario are as follows (Note: all > >>> producing and consuming is performed over the stomp transport): > >>> > >>> Couple notes about the consumers: > >>> - they have a prefetchSize of 40 > >>> - the processing of messages can take some time, so for the purposes > >>> of this exercise, I've created a simple consumer that sleeps for 10 > >>> seconds before sending the message ack (using client-individual ack > >>> mode) > >>> > >>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not > >>> yet running. > >>> 2. produce a few thousand messages to B1 > >>> Note: B1 now has a few thousand pending messages and B2 has 0. > >>> 3. start consumer C2 (listing for messages from B2) > >>> Note: messages are are successfully received and begin processing > >>> (monitoring the brokers shows pending messages decreasing). Now B2 > >>> has all the pending messages and B1 has 0. > >>> 4. start consumer C1 (listing for messages from B1) > >>> Note: no messages are received, which is another issue I have > >>> since > >>> B2 now has thousands of pending messages which C1 could help > >>> process, but instead sits idle while C2 is forced to handle all > >>> the > >>> messages. > >>> 5. stop consumer C2 > >>> Note: now I have thousands of messages sitting on B2 and 0 on B1 > >>> where a C1 is alive and ready to handle them. So at this point, > >>> despite having a consumer running, thousands of messages are stuck > >>> in the queue. > >>> 6. stop consumer C1 > >>> Note: now I have no consumers. Stopping and restarting C1 has no > >>> effect on the pending messages sitting on B1's queue. > >>> 7. stop both brokers > >>> 8. start B1, then start B2 > >>> 9. start C1 > >>> Note: now all messages have migrated from B2 to B1 and C1 is again > >>> processing messages. > >>> > >>> So after step 5, the only way to recover from the stuck messages > >>> is to > >>> restart the brokers. > >>> > >>> Below is my current connector config which I have on both brokers. > >>> I've tried playing with the various properties of the connector, but > >>> it seems as though no matter what I try the above scenario continues > >>> to occur. > >>> > >>> <networkConnector > >>> name="default-nc" > >>> uri="multicast://default?group=${broker.group}" > >>> dynamicOnly="true" > >>> networkTTL="25" > >>> suppressDuplicateQueueSubscriptions="true"/> > >>> > >>> > >>> So, is this an activemq bug? Am I mis-using activemq? Is there some > >>> other way to achieve a highly available distributed queue? > >>> > >>> Any help in this regard is greatly appreciated. > >>> > >>> -- > >>> eric > >>> > >>> > > > > -- > > eric > > Rob Davies > http://twitter.com/rajdavies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ -- eric
