This is exceedingly bizarre. Now ActiveMQ is refusing to deliver ANY messages to my workers.
This is very bizarre, no code has changed. Nothing. It’s just refusing to give work. If I set the prefetch to 0 or 1, it does work for a few moments, then halts. 99% certain I’m committing all my messages. As it would make sense that nothing could be processed after that of course. Kevin On Tue, Mar 10, 2015 at 6:53 PM, Tim Bain <tb...@alumni.duke.edu> wrote: > If you make a single consumer, you'll only get one message at a time by > default (so only one thread will be doing any work). You'd have to use > client acknowledgement or selective acknowledgement to get more than one > message at a time. I'd probably leave many consumers but tune down your > prefetch buffers to something relatively small to ensure that the workload > is evenly spread and you don't have some consumers with a large prefetch > buffer worth of backlog while others sit around idle. > > But if you're seeing the broker report pending messages, then that means > that having unbalanced workloads due to large prefetch buffers isn't your > problem... Pending messages on the broker only occur when those messages > can't be dispatched to any consumer because all of their prefetch buffers > are full, which would mean that you don't have unbalanced workloads. > You'll get one or the other, not both. > > On Tue, Mar 10, 2015 at 7:42 PM, Kevin Burton <bur...@spinn3r.com> wrote: > > > I’m actually wondering if this is my issues. I’m creating one session per > > thread. So perhaps some of the threads have work to do, but they’re each > > prefetching a bunch of work when in reality a better strategy might the > to > > have one master listener and then dispatch messages to each thread. > > > > On Tue, Mar 10, 2015 at 6:37 PM, Kevin Burton <bur...@spinn3r.com> > wrote: > > > > > OK. That’s good to know. I have a large number of connections so I > have > > > to look at each one. I wonder if this could also be the issue. AKA > too > > > many connections. > > > > > > On Tue, Mar 10, 2015 at 6:19 PM, Tim Bain <tb...@alumni.duke.edu> > wrote: > > > > > >> You should be able to confirm that the prefetch buffers are empty by > > >> inspecting the JMX MBeans on the broker. Look at the consumers for > the > > >> destination, and for each one look at its DispatchedQueueSize > attribute. > > >> > > >> Keep in mind that slow consumers are identified *ONLY* if you > configure > > >> one > > >> of the abort strategies. If you didn't set that up, don't expect any > > slow > > >> consumer identification log lines. And if you did, I've never seen a > > >> situation where a consumer went slow and a log line didn't happen > (using > > >> the SlowConsumerAbortStrategy; I haven't used > > SlowAckConsumerAbortStrategy > > >> and can't vouch for it); we get those log lines pretty frequently. So > > if > > >> you're not seeing broker-side log lines about consumers being > identified > > >> as > > >> slow and then aborted, I'd bet it's simply not happening. > > >> > > >> On Tue, Mar 10, 2015 at 7:10 PM, Kevin Burton <bur...@spinn3r.com> > > wrote: > > >> > > >> > The broker. I’ll assume the prefetch brokers are empty. I’m looking > > >> into > > >> > debugging that now but I don’t have tools to introspect. > > >> > > > >> > The broker has thousands of messages. > > >> > > > >> > I just confirmed that a restart DOES improve the situation. > > >> > > > >> > It’s possible that they’re being marked as slow consumers but not > > >> *logged* > > >> > as such so I’m trying to use JMX to dump the sessions. > > >> > > > >> > On Tue, Mar 10, 2015 at 5:58 PM, Tim Bain <tb...@alumni.duke.edu> > > >> wrote: > > >> > > > >> > > Are the messages getting hung up in the broker or in the client? > > (Do > > >> the > > >> > > consumers have empty or full prefetch buffers?) > > >> > > > > >> > > On Tue, Mar 10, 2015 at 6:47 PM, Kevin Burton <bur...@spinn3r.com > > > > >> > wrote: > > >> > > > > >> > > > I’m still trying to track down some issues with ActiveMQ … > > >> > > > > > >> > > > One is that I have 5 ActiveMQ servers now, and each one has > about > > >> 3000 > > >> > > > messages pending. So 15000 messages in queues. > > >> > > > > > >> > > > These are non-persistent queues, plenty of memory and plenty of > > CPU, > > >> > but > > >> > > > the workers are just blocked waiting to receive work. > > >> > > > > > >> > > > I had a hypothesis that this could be slow workers, but after > > tuning > > >> > some > > >> > > > things I no longer receive any errors about slow workers. > > >> > > > > > >> > > > Restarting the daemons doesn’t fix things either. Anything else > > it > > >> > could > > >> > > > be? I’m a bit stumped unfortunately. > > >> > > > > > >> > > > -- > > >> > > > > > >> > > > Founder/CEO Spinn3r.com > > >> > > > Location: *San Francisco, CA* > > >> > > > blog: http://burtonator.wordpress.com > > >> > > > … or check out my Google+ profile > > >> > > > <https://plus.google.com/102718274791889610666/posts> > > >> > > > <http://spinn3r.com> > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > Founder/CEO Spinn3r.com > > >> > Location: *San Francisco, CA* > > >> > blog: http://burtonator.wordpress.com > > >> > … or check out my Google+ profile > > >> > <https://plus.google.com/102718274791889610666/posts> > > >> > <http://spinn3r.com> > > >> > > > >> > > > > > > > > > > > > -- > > > > > > Founder/CEO Spinn3r.com > > > Location: *San Francisco, CA* > > > blog: http://burtonator.wordpress.com > > > … or check out my Google+ profile > > > <https://plus.google.com/102718274791889610666/posts> > > > <http://spinn3r.com> > > > > > > > > > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
