Ah, that attribute isn't documented on the activemq site ;)
I tried that one out, but unfortunately I don't see any change in
behavior. I verified that the prefetchSize is indeed set to 1 by
examining the network connector attributes for both brokers via jmx.
Thank you so much for your help so far, I really do appreciate it.
I'm going to start digging through the code to see if I can determine
why these messages refuse to budge. I'll probably start by seeing if
Joe's assumption about messages not visiting a broker more than once
is the cause.
If you or anyone else has any other tips or suggestions, I'd love to
hear them.
On Thu, Oct 08, 2009 at 05:42:48PM +0100, Rob Davies wrote:
> Hi Eric,
>
> sorry for being so quick in my reply - in the broker xml config set
> the prefetchSize property on the network - e.g.
>
> <networkConnectors>
> <networkConnector
> uri="static://(tcp://localhost:61617)"
> name="Connection to 61617"
> dynamicOnly="true"
> prefetchSize="1"/>
>
> </networkConnectors>
> On 8 Oct 2009, at 17:16, Eric Van Dewoestine wrote:
>
> > I guess I'm not sure what setting you are referring to. I tried
> > setting a couple different prefetch policies on the tcp connector like
> > so:
> >
> > 1st attempt:
> > <transportConnector
> > uri="tcp://${broker.host}:0?jms.prefetchPolicy.queuePrefetch=1"
> > discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > 2nd attempt:
> > <transportConnector
> > uri="tcp://${broker.host}:0?jms.prefetchPolicy.all=1"
> > discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > But neither of them seemed to change the behavior. Are you referring
> > to another setting? Can you provide an example of the setting you are
> > referring to?
> >
> > On Thu, Oct 08, 2009 at 04:22:11PM +0100, Rob Davies wrote:
> >> do you set the prefetchSize property on the network itself ? Setting
> >> it on the consumers wouldn't make any difference
> >> On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
> >>
> >>> I tried setting the prefetchSize to 1 for both consumers, but
> >>> unfortunately, that didn't resolve the issue. Messages still end up
> >>> stuck on B2 when C2 is stopped.
> >>>
> >>> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> >>>> set the prefetchSize=1 on your network - to avoid orphaned messages
> >>>> on
> >>>> B2
> >>>>
> >>>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
> >>>>
> >>>>> Hmm, that seems like a flawed approach if the broker it visited
> >>>>> has
> >>>>> changed state and is now eligible to process those messages.
> >>>>> How is
> >>>>> this type of fail over intended to be handled with activemq? Is
> >>>>> there
> >>>>> an alternate solution that I'm missing?
> >>>>>
> >>>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>>>>>
> >>>>>> I think that by design a message will not be forwarded to a
> >>>>>> broker
> >>>>>> that it
> >>>>>> has already visited.
> >>>>>>
> >>>>>> Joe
> >>>>>> http://www.ttmsolutions.com
> >>>>>>
> >>>>>>
> >>>>>> Eric Van wrote:
> >>>>>>>
> >>>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
> >>>>>>> listing)
> >>>>>>>
> >>>>>>> I'm running into an issue with the store and forward feature of
> >>>>>>> activemq, which I using in an attempt to create a highly
> >>>>>>> available
> >>>>>>> distributed queue. I'm trying to figure out if the issue is a
> >>>>>>> misconfiguration on my part, expected behavior of activemq, or a
> >>>>>>> bug.
> >>>>>>>
> >>>>>>> The summary of the problem is that given 2 brokers, B1 and B2,
> >>>>>>> which
> >>>>>>> each have one consumer, C1 and C2, which are subscribed to the
> >>>>>>> same
> >>>>>>> queue. If I stop a consumer on one of the brokers, the pending
> >>>>>>> messages from that broker are not always forwarded to the other
> >>>>>>> broker
> >>>>>>> which still has a consumer, leading to those messages getting
> >>>>>>> indefinitely stuck.
> >>>>>>>
> >>>>>>> The steps I use to reproduce this scenario are as follows (Note:
> >>>>>>> all
> >>>>>>> producing and consuming is performed over the stomp transport):
> >>>>>>>
> >>>>>>> Couple notes about the consumers:
> >>>>>>> - they have a prefetchSize of 40
> >>>>>>> - the processing of messages can take some time, so for the
> >>>>>>> purposes
> >>>>>>> of this exercise, I've created a simple consumer that sleeps for
> >>>>>>> 10
> >>>>>>> seconds before sending the message ack (using client-individual
> >>>>>>> ack
> >>>>>>> mode)
> >>>>>>>
> >>>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are
> >>>>>>> not
> >>>>>>> yet running.
> >>>>>>> 2. produce a few thousand messages to B1
> >>>>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
> >>>>>>> 3. start consumer C2 (listing for messages from B2)
> >>>>>>> Note: messages are are successfully received and begin
> >>>>>>> processing
> >>>>>>> (monitoring the brokers shows pending messages decreasing). Now
> >>>>>>> B2
> >>>>>>> has all the pending messages and B1 has 0.
> >>>>>>> 4. start consumer C1 (listing for messages from B1)
> >>>>>>> Note: no messages are received, which is another issue I have
> >>>>>>> since
> >>>>>>> B2 now has thousands of pending messages which C1 could help
> >>>>>>> process, but instead sits idle while C2 is forced to handle all
> >>>>>>> the
> >>>>>>> messages.
> >>>>>>> 5. stop consumer C2
> >>>>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>>>>>> where a C1 is alive and ready to handle them. So at this point,
> >>>>>>> despite having a consumer running, thousands of messages are
> >>>>>>> stuck
> >>>>>>> in the queue.
> >>>>>>> 6. stop consumer C1
> >>>>>>> Note: now I have no consumers. Stopping and restarting C1 has
> >>>>>>> no
> >>>>>>> effect on the pending messages sitting on B1's queue.
> >>>>>>> 7. stop both brokers
> >>>>>>> 8. start B1, then start B2
> >>>>>>> 9. start C1
> >>>>>>> Note: now all messages have migrated from B2 to B1 and C1 is
> >>>>>>> again
> >>>>>>> processing messages.
> >>>>>>>
> >>>>>>> So after step 5, the only way to recover from the stuck messages
> >>>>>>> is to
> >>>>>>> restart the brokers.
> >>>>>>>
> >>>>>>> Below is my current connector config which I have on both
> >>>>>>> brokers.
> >>>>>>> I've tried playing with the various properties of the connector,
> >>>>>>> but
> >>>>>>> it seems as though no matter what I try the above scenario
> >>>>>>> continues
> >>>>>>> to occur.
> >>>>>>>
> >>>>>>> <networkConnector
> >>>>>>> name="default-nc"
> >>>>>>> uri="multicast://default?group=${broker.group}"
> >>>>>>> dynamicOnly="true"
> >>>>>>> networkTTL="25"
> >>>>>>> suppressDuplicateQueueSubscriptions="true"/>
> >>>>>>>
> >>>>>>>
> >>>>>>> So, is this an activemq bug? Am I mis-using activemq? Is there
> >>>>>>> some
> >>>>>>> other way to achieve a highly available distributed queue?
> >>>>>>>
> >>>>>>> Any help in this regard is greatly appreciated.
> >>>>>>>
> >>>>>>> --
> >>>>>>> eric
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> eric
> >>>>
> >>>> Rob Davies
> >>>> http://twitter.com/rajdavies
> >>>> I work here: http://fusesource.com
> >>>> My Blog: http://rajdavies.blogspot.com/
> >>>> I'm writing this: http://www.manning.com/snyder/
> >>>
> >>> --
> >>> eric
> >>
> >> Rob Davies
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
--
eric
