Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
Yes.. I think what the work around would probably do is revert to “softer” priority scheduling when it kicks in because the fall through message consumer with no selector will work. I will still try to get a fix implemented of course but I have a release i need to get out the door :) On Fri, Apr

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
It might be a good fail-safe, but hopefully you'll find something as you step through with the debugger that will highlight what the problem actually is (so you don't need to use the no-selector workaround). Workarounds are great, but fixes are better. :) But it might be a way to at least get pro

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
Sounds like a good idea. I just pushed with maxPageSize=5000 Interesting that my backlogged queues seems to be locked up at about 5000 messages. … I’m wonder if those are messages that aren’t matching selectors? I could have a plan B / fall through that accepts messages without a selector.. it

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
OK, so based on what you wrote, it sounds like the broker gets into a state where message selectors don't work properly, since the same selector matches all messages (except ones with a null priority; have you confirmed that both artemis_priority <= 9 and artemis_priority = null match their respect

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
> > > > http://imgur.com/a/2myja > > > What are the two screenshots; with and without the selector? If that's > right, then clearly zero messages are matching the selector (which means > this isn't a delivery problem, it's a selector problem), so hopefully > pulling that thread will lead to a solu

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
On Fri, Apr 24, 2015 at 2:54 PM, Tim Bain wrote: > If you start from a zero-state (broker and all clients stopped) and attach > only one consumer with your "artemis_priority <= 9" selector, do you get > any messages to it? If I restart a new broker, and start consuming messages, it will work fi

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
On Fri, Apr 24, 2015 at 3:43 PM, Kevin Burton wrote: > On Fri, Apr 24, 2015 at 2:27 PM, Tim Bain wrote: > > > If every message has at least one consumer for which the consumer's > > selector matches the message, you'll eventually process every message. > > > > That’s what I thought too, but tha

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
If you start from a zero-state (broker and all clients stopped) and attach only one consumer with your "artemis_priority <= 9" selector, do you get any messages to it? (I'm trying to eliminate any uncertainty about the state that might have been caused by adding and removing clients.) What you ju

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
OK.. but what I don’t understand is that I have at least ONE consumer that matches. So that one should keep being served (though with imperfect priority) Also, as a test, I shut down ALL consumers on the broker, then just had ONE consumer, and the selector was artemis_priority <= 9 … and this sh

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
On Fri, Apr 24, 2015 at 2:27 PM, Tim Bain wrote: > If every message has at least one consumer for which the consumer's > selector matches the message, you'll eventually process every message. > That’s what I thought too, but that doesn’t work. > Consumers that have no messages matching their

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
Keep in mind that a pause as described by that JIRA could come about because your consumer has a full prefetch buffer worth of messages that match the selector plus lots more messages in the store. If you have a backlog for any consumer, anything that can't fit in the consumer's prefetch buffer wi

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Tim Bain
If every message has at least one consumer for which the consumer's selector matches the message, you'll eventually process every message. Consumers that have no messages matching their selector in the cursor will be delayed until the messages in front of their next one get consumed, but they'll do

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
Literally JUST found this issue! Is this documented anywhere? My issue is that there *is* no sparse message distribution. Every message has a value from between 0 and 9 with none lacking that header. I even consume where the message is lacking the value. So there shouldn’t be anything left over

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Timothy Bish
On 04/24/2015 04:50 PM, Kevin Burton wrote: > I’ve been working 15 hour days for the last 2-3 weeks trying to resolve > this so if this is somewhat incoherent it’s probably due to lack of sleep > :-P > > I think we’re experiencing a bug in ActiveMQ which is VERY hard to > reproduce but happens regu

Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
Here are two screenshots of the JMX consumer stats, one without a selector and one with a selector.. you can see the one with the selector just not working. [image: Inline image 1] [image: Inline image 2] On Fri, Apr 24, 2015 at 1:50 PM, Kevin Burton wrote: > I’ve been working 15 hour days f

I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.

2015-04-24 Thread Kevin Burton
I’ve been working 15 hour days for the last 2-3 weeks trying to resolve this so if this is somewhat incoherent it’s probably due to lack of sleep :-P I think we’re experiencing a bug in ActiveMQ which is VERY hard to reproduce but happens regularly in our production setup. I can’t reproduce it in

Re: ActiveMQ topic: Messages are not delived to a client

2015-04-24 Thread Tim Bain
Is your consumer the only consumer for that topic? Disconnecting a non-durable topic consumer should result in the subscription being removed and all pending messages being DLQed/discarded. If you're not seeing that, could it be that you have another consumer attached and the in-flight messages a

Re: Message Grouping Retry Policy

2015-04-24 Thread Gary Tully
perfectly. On 24 April 2015 at 15:55, Tim Bain wrote: > Thanks for the info Gary. > > Summarizing the info in that JIRA, I believe the answer is "If you've set > 'jms.nonBlockingRedelivery=true' on your URL, message processing will > continue during the redelivery interval, and the redelivered me

Re: Message Grouping Retry Policy

2015-04-24 Thread Tim Bain
Thanks for the info Gary. Summarizing the info in that JIRA, I believe the answer is "If you've set 'jms.nonBlockingRedelivery=true' on your URL, message processing will continue during the redelivery interval, and the redelivered message will be re-processed out-of-order once the redelivery inter

Re: DLQ, cause:null

2015-04-24 Thread James Green
I'm need to understand pre-fetch limit and receive time-out interaction. We have four concurrent consumers in our route. Do each receive the messages in batches of the pre-fetch limit? At what point does the receive time-out start and end? In our case each client performs a number of db queries

Automatically Discard Expired Messages has no effect

2015-04-24 Thread wangclever
Dear all: I 'm using activemq 5.8,i config Automatically Discard Expired Messages as follow:

ActiveMQ with fileQueueCursor and get Heap size error

2015-04-24 Thread Kacu
Hi everyone, I'm struggling with activemq issue which I describe there http://stackoverflow.com/questions/29823947/activemq-filequeuecursor-heap-size but maybe one users form this forum had similar problem and could help me with it. This situation is happening when consumer break, and producer pr

Re: Message Grouping Retry Policy

2015-04-24 Thread Gary Tully
there is some good info in https://issues.apache.org/jira/browse/AMQ-1853 short answer: the ordered blocking consumer side is configurable but it is independent of jms message groups On 6 April 2015 at 11:09, cduvvuri wrote: > Would ActiveMq holds ordered messages delivery until the first messag

Re: DLQ, cause:null

2015-04-24 Thread James Green
I've added the following, "A client connection times out (perhaps the code being executed takes longer than the configured time-out period)," to the list of reasons on http://activemq.apache.org/message-redelivery-and-dlq-handling.html (should appear shortly). If someone can think of more precise

Re: DLQ, cause:null

2015-04-24 Thread Gary Tully
Tim, steady, I suggested it *may* be relevant :-) With camel and transactions - ie: spring dmlc, connection pools and cache levels - anything is possible w.r.t consumer/sessions/connection state, because there are so many variables in the mix. With activemq and prefetch, every consumer disconnect

Re: DLQ, cause:null

2015-04-24 Thread James Green
We have set the receive() timeout to 10s via the url - no DLQ issues for the past few hours, we'll wait until middle of next week before we're really confident. Does not explain why things on the machine appear slow at times just helps us deal with them. On 24 April 2015 at 11:57, Gary Tully wro

Re: DLQ, cause:null

2015-04-24 Thread Tim Bain
Gary, If I understood that JIRA correctly, the bug only occurs when the client disconnects, which doesn't sound like what James is doing (nothing in his description indicated to me that his client wasn't staying up and connected the whole time), so it doesn't sound like your fix would resolve (nor

Re: DLQ, cause:null

2015-04-24 Thread Gary Tully
to avoid the redelivered messages getting sent to the DLQ, changing the default redelivery policy max from 6 to infinite will help. You can do this in the brokerurl passed to the jms connection factory, it may also make sense to reduce the prefetch if consumers come and go without consuming the pr