Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-12 Thread Kevin Burton
The problem is that this is happening in production but not locally. Setting prefetchPolicy to zero fixed it for one of our tasks, but another one of our tasks is still waiting for work and I can’t figure out why. It has 6000 consumers / threads, 200k messages waiting to be processed, but none

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-12 Thread Kevin Burton
I didn’t configure one… so we’re running the stock one. I’m trying to verify it it’s just a specific task not committing messages but for some reason it processes them in batches. If it wasn’t committing the messages I would expect it to lock up permanently. One thing that could be nice for

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-11 Thread Kevin Burton
Yes. I’m sorry. I meant the stock configuration so no slow consumer strategy! On Wed, Mar 11, 2015 at 12:56 PM, Tim Bain tb...@alumni.duke.edu wrote: There isn't a stock slow consumer strategy. If you didn't configure it, you don't have one. On Wed, Mar 11, 2015 at 1:49 PM, Kevin Burton

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-11 Thread Tim Bain
There isn't a stock slow consumer strategy. If you didn't configure it, you don't have one. On Wed, Mar 11, 2015 at 1:49 PM, Kevin Burton bur...@spinn3r.com wrote: I didn’t configure one… so we’re running the stock one. I’m trying to verify it it’s just a specific task not committing messages

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-11 Thread Tim Bain
Out of curiosity, which slow consumer strategy did you configure? On Wed, Mar 11, 2015 at 11:45 AM, Kevin Burton bur...@spinn3r.com wrote: The problem is that this is happening in production but not locally. Setting prefetchPolicy to zero fixed it for one of our tasks, but another one of our

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-11 Thread Tim Bain
I'd definitely set breakpoints and step through with a debugger to try to figure out what's going on. On Mar 10, 2015 8:19 PM, Kevin Burton bur...@spinn3r.com wrote: This is exceedingly bizarre. Now ActiveMQ is refusing to deliver ANY messages to my workers. This is very bizarre, no code has

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Kevin Burton
OK. That’s good to know. I have a large number of connections so I have to look at each one. I wonder if this could also be the issue. AKA too many connections. On Tue, Mar 10, 2015 at 6:19 PM, Tim Bain tb...@alumni.duke.edu wrote: You should be able to confirm that the prefetch buffers are

Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Kevin Burton
I’m still trying to track down some issues with ActiveMQ … One is that I have 5 ActiveMQ servers now, and each one has about 3000 messages pending. So 15000 messages in queues. These are non-persistent queues, plenty of memory and plenty of CPU, but the workers are just blocked waiting to

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Tim Bain
You should be able to confirm that the prefetch buffers are empty by inspecting the JMX MBeans on the broker. Look at the consumers for the destination, and for each one look at its DispatchedQueueSize attribute. Keep in mind that slow consumers are identified *ONLY* if you configure one of the

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Tim Bain
If you make a single consumer, you'll only get one message at a time by default (so only one thread will be doing any work). You'd have to use client acknowledgement or selective acknowledgement to get more than one message at a time. I'd probably leave many consumers but tune down your prefetch

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Tim Bain
Are the messages getting hung up in the broker or in the client? (Do the consumers have empty or full prefetch buffers?) On Tue, Mar 10, 2015 at 6:47 PM, Kevin Burton bur...@spinn3r.com wrote: I’m still trying to track down some issues with ActiveMQ … One is that I have 5 ActiveMQ servers

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Kevin Burton
I’m actually wondering if this is my issues. I’m creating one session per thread. So perhaps some of the threads have work to do, but they’re each prefetching a bunch of work when in reality a better strategy might the to have one master listener and then dispatch messages to each thread. On

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Kevin Burton
This is exceedingly bizarre. Now ActiveMQ is refusing to deliver ANY messages to my workers. This is very bizarre, no code has changed. Nothing. It’s just refusing to give work. If I set the prefetch to 0 or 1, it does work for a few moments, then halts. 99% certain I’m committing all my

Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.

2015-03-10 Thread Kevin Burton
The broker. I’ll assume the prefetch brokers are empty. I’m looking into debugging that now but I don’t have tools to introspect. The broker has thousands of messages. I just confirmed that a restart DOES improve the situation. It’s possible that they’re being marked as slow consumers but not