I’m actually wondering if this is my issues. I’m creating one session per thread. So perhaps some of the threads have work to do, but they’re each prefetching a bunch of work when in reality a better strategy might the to have one master listener and then dispatch messages to each thread.
On Tue, Mar 10, 2015 at 6:37 PM, Kevin Burton <bur...@spinn3r.com> wrote: > OK. That’s good to know. I have a large number of connections so I have > to look at each one. I wonder if this could also be the issue. AKA too > many connections. > > On Tue, Mar 10, 2015 at 6:19 PM, Tim Bain <tb...@alumni.duke.edu> wrote: > >> You should be able to confirm that the prefetch buffers are empty by >> inspecting the JMX MBeans on the broker. Look at the consumers for the >> destination, and for each one look at its DispatchedQueueSize attribute. >> >> Keep in mind that slow consumers are identified *ONLY* if you configure >> one >> of the abort strategies. If you didn't set that up, don't expect any slow >> consumer identification log lines. And if you did, I've never seen a >> situation where a consumer went slow and a log line didn't happen (using >> the SlowConsumerAbortStrategy; I haven't used SlowAckConsumerAbortStrategy >> and can't vouch for it); we get those log lines pretty frequently. So if >> you're not seeing broker-side log lines about consumers being identified >> as >> slow and then aborted, I'd bet it's simply not happening. >> >> On Tue, Mar 10, 2015 at 7:10 PM, Kevin Burton <bur...@spinn3r.com> wrote: >> >> > The broker. I’ll assume the prefetch brokers are empty. I’m looking >> into >> > debugging that now but I don’t have tools to introspect. >> > >> > The broker has thousands of messages. >> > >> > I just confirmed that a restart DOES improve the situation. >> > >> > It’s possible that they’re being marked as slow consumers but not >> *logged* >> > as such so I’m trying to use JMX to dump the sessions. >> > >> > On Tue, Mar 10, 2015 at 5:58 PM, Tim Bain <tb...@alumni.duke.edu> >> wrote: >> > >> > > Are the messages getting hung up in the broker or in the client? (Do >> the >> > > consumers have empty or full prefetch buffers?) >> > > >> > > On Tue, Mar 10, 2015 at 6:47 PM, Kevin Burton <bur...@spinn3r.com> >> > wrote: >> > > >> > > > I’m still trying to track down some issues with ActiveMQ … >> > > > >> > > > One is that I have 5 ActiveMQ servers now, and each one has about >> 3000 >> > > > messages pending. So 15000 messages in queues. >> > > > >> > > > These are non-persistent queues, plenty of memory and plenty of CPU, >> > but >> > > > the workers are just blocked waiting to receive work. >> > > > >> > > > I had a hypothesis that this could be slow workers, but after tuning >> > some >> > > > things I no longer receive any errors about slow workers. >> > > > >> > > > Restarting the daemons doesn’t fix things either. Anything else it >> > could >> > > > be? I’m a bit stumped unfortunately. >> > > > >> > > > -- >> > > > >> > > > Founder/CEO Spinn3r.com >> > > > Location: *San Francisco, CA* >> > > > blog: http://burtonator.wordpress.com >> > > > … or check out my Google+ profile >> > > > <https://plus.google.com/102718274791889610666/posts> >> > > > <http://spinn3r.com> >> > > > >> > > >> > >> > >> > >> > -- >> > >> > Founder/CEO Spinn3r.com >> > Location: *San Francisco, CA* >> > blog: http://burtonator.wordpress.com >> > … or check out my Google+ profile >> > <https://plus.google.com/102718274791889610666/posts> >> > <http://spinn3r.com> >> > >> > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>