Also, I was thinking that this MIGHT be a bug with unfair scheduling. synchronized and read/write locks aren’t fair.
So it’s entirely possible that the client is scheduling work on on faster queue because they reply quicker and thus they win the lock race. This would explain why I can read from the queue server from a dedicated process, but my main worker daemon, which is rather loaded can’t process any more work. My boxes aren’t loaded THAT bad. Maybe about 50% CPU 24/7.. but it’s a lot of activemq work. So it’s entirely possible that the client is continually losing a race working with the larger queues. On Sun, Apr 19, 2015 at 8:03 PM, Kevin Burton <bur...@spinn3r.com> wrote: > Also, I”ve run with and without producer flow control and that also > doesn’t impact the situation. > > On Sun, Apr 19, 2015 at 8:01 PM, Kevin Burton <bur...@spinn3r.com> wrote: > >> Here’s the public gist of our XML config. (it needs some comment cleanup >> but that’s that we’re running with). >> >> https://gist.github.com/burtonator/b5f4228b0f0acbf05b4e >> >> We’re running 5.10.2 . I’ve reviewed the bugs fixed since then and >> nothing seems to apply to our situation. I would upgrade but we run with >> the memory persistence adapter and there’s a bug that was introduced in the >> 11.x series which impacts memory persistence and advisories. >> >> And I haven’t had time to fix that. >> >> We also have a fix applied that I developed to release lock contention >> during queue GC. It’s within the realm of probability that this could have >> introduced a bug but I feel VERY confident that this is not the issue. We >> contributed it back and there’s an outstanding pull request for this on the >> 5.10 and 5.11 branches but since I didn’t apply it against master it was >> never merged :-( >> >> I didn’t TRACE activemq but I can do this now. Should I do this on the >> broker too? the problem is that this is going to result in a VERY large >> log file because it takes about 30 minutes to reproduce this. >> >> In the mean time, I’m going to write a test to see if I can reproduce >> this by creating a similar situation of high load and a backlogged queue. >> >> On Sun, Apr 19, 2015 at 5:58 PM, Paul Gale <paul.n.g...@gmail.com> wrote: >> >>> What version of ActiveMQ are you using? Please send the contents of you >>> activemq.xml file plus details of your producer consumer and how they're >>> implemented. >>> Have you set the broker's logging to TRACE level prior to running your >>> experiments? If so, please attach or use pastebin. >>> >> -- >> >> Founder/CEO Spinn3r.com >> Location: *San Francisco, CA* >> blog: http://burtonator.wordpress.com >> … or check out my Google+ profile >> <https://plus.google.com/102718274791889610666/posts> >> <http://spinn3r.com> >> >> > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
