Re: Broker Hangs after some time - or does nothing.
is there any possibility you could build a test case to capture your use case? On 27 July 2014 04:24, rking wrote: > Also using 5.9.1 and our broker hangs almost regularly for the past few > months. Restarting the broker fixes it, but we cannot have it happening too > often. Would be good to know if there's any solution to this. Was thinking > of upgrading to 5.10 but it seems the same problem happens there. > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4683707.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. -- http://redhat.com http://blog.garytully.com
Re: Broker Hangs after some time - or does nothing.
Also using 5.9.1 and our broker hangs almost regularly for the past few months. Restarting the broker fixes it, but we cannot have it happening too often. Would be good to know if there's any solution to this. Was thinking of upgrading to 5.10 but it seems the same problem happens there. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4683707.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
so far no luck. I see that in our dev environment. i am not sure how to fix it. if you figure out anything please share. thanks, On Fri, May 16, 2014 at 9:15 AM, rgooduru wrote: > Are you able to find the cause of this issue, I am facing same issue with > 5.9.1. We are using 5.9.1 with leveldb, everything worked properly in our > QA > but when we migrate to prod we had seen the same behavoir you were > describing. > > I was unable to capture number of connections or heap size etc from prod. I > am not able to reproduce this in QA or DEV. Is this something to do with > network connections? Any help would be appreciated. > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4681166.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: Broker Hangs after some time - or does nothing.
Are you able to find the cause of this issue, I am facing same issue with 5.9.1. We are using 5.9.1 with leveldb, everything worked properly in our QA but when we migrate to prod we had seen the same behavoir you were describing. I was unable to capture number of connections or heap size etc from prod. I am not able to reproduce this in QA or DEV. Is this something to do with network connections? Any help would be appreciated. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4681166.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
activemq instance. So if it >>>> had >>>> > memory issues how come console is still working? isn't it same process? >>>> > >>>> > thanks, >>>> > chirag >>>> > >>>> > >>>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef wrote: >>>> > >>>> >> How about a stack trace on the client? Can you look for the consumer >>>> >> threads >>>> >> and see what they are doing? >>>> >> >>>> >> Note that with producers and consumers on the same connection, it's >>>> >> possible >>>> >> to reach a deadlock if producer-flow-control kicks in because the >>>> entire >>>> >> connection is blocked, not just the one producer. There are two ways >>>> >> this >>>> >> can happen. First, if the client produces and consumes the same >>>> >> destination. Second, if the client is producing one destination and >>>> >> consuming a second destination and another client is consuming the >>>> first >>>> >> and >>>> >> producing to the second. Actually, more complicated possibilities >>>> exist >>>> >> as >>>> >> well. >>>> >> >>>> >> Your best bet is to never consume and produce on the same connection >>>> so >>>> >> that >>>> >> consumption never blocks due to producer-flow-control. In that >>>> >> scenario, >>>> >> deadlocks won't happen because consumers can always consume. >>>> >> >>>> >> Back to stack traces - if the consuming threads are blocked waiting to >>>> >> obtain a lock while producing threads hold that lock, that could >>>> >> indicate >>>> >> the problem. >>>> >> >>>> >> Anyway, slow consumption is quite often the cause of broker hangs, so >>>> >> looking for slow consumption and understanding speeding consumption >>>> (or >>>> >> slowing production) is important. If the broker JVM becomes >>>> completely >>>> >> unresponsive, try connecting jconsole or visualvm before the broker >>>> >> hangs >>>> >> and then watch memory. If the JVM is running out of Heap or Permgen >>>> >> space, >>>> >> that would explain the hung broker. >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> View this message in context: >>>> >> >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html >>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >> >>>> > >>>> > >>>> > >>>> > >>>> > ___ >>>> > If you reply to this email, your message will be added to the >>>> discussion >>>> > below: >>>> > >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html >>>> > To start a new topic under ActiveMQ - User, email >>>> > ml-node+s2283324n234180...@n4.nabble.com >>>> > To unsubscribe from ActiveMQ - User, visit >>>> > >>>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >> -- http://redhat.com http://blog.garytully.com
Re: Broker Hangs after some time - or does nothing.
; >> Note that with producers and consumers on the same connection, it's >>> >> possible >>> >> to reach a deadlock if producer-flow-control kicks in because the >>> entire >>> >> connection is blocked, not just the one producer. There are two ways >>> >> this >>> >> can happen. First, if the client produces and consumes the same >>> >> destination. Second, if the client is producing one destination and >>> >> consuming a second destination and another client is consuming the >>> first >>> >> and >>> >> producing to the second. Actually, more complicated possibilities >>> exist >>> >> as >>> >> well. >>> >> >>> >> Your best bet is to never consume and produce on the same connection >>> so >>> >> that >>> >> consumption never blocks due to producer-flow-control. In that >>> >> scenario, >>> >> deadlocks won't happen because consumers can always consume. >>> >> >>> >> Back to stack traces - if the consuming threads are blocked waiting to >>> >> obtain a lock while producing threads hold that lock, that could >>> >> indicate >>> >> the problem. >>> >> >>> >> Anyway, slow consumption is quite often the cause of broker hangs, so >>> >> looking for slow consumption and understanding speeding consumption >>> (or >>> >> slowing production) is important. If the broker JVM becomes >>> completely >>> >> unresponsive, try connecting jconsole or visualvm before the broker >>> >> hangs >>> >> and then watch memory. If the JVM is running out of Heap or Permgen >>> >> space, >>> >> that would explain the hung broker. >>> >> >>> >> >>> >> >>> >> -- >>> >> View this message in context: >>> >> >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >> >>> > >>> > >>> > >>> > >>> > ___ >>> > If you reply to this email, your message will be added to the >>> discussion >>> > below: >>> > >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html >>> > To start a new topic under ActiveMQ - User, email >>> > ml-node+s2283324n234180...@n4.nabble.com >>> > To unsubscribe from ActiveMQ - User, visit >>> > >>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >
Re: Broker Hangs after some time - or does nothing.
I tried again and this time I didnt send bunch of messages at same time. I
noticed that some message gets processed right away but some messges stays
on queue and doesnt get processed. If send few more messges let say 50
messages and it processes 40 messages and 10 msgs left on queue. I can send
more messages but not all get processed.
I am attaching config file. I have enabled producerFlowControl and removed
slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT.
On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara wrote:
> i said broker hangs bcs if i tried to send more messages or messages on
> queue doent get processed. i cannot even create queue or send message using
> webconsole.
>
> will try again and rhis time will moniter memory.
>
> On Feb 10, 2014 6:22 PM, "artnaseef" wrote:<
>>
>> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
>> symptoms.
>>
>> The messages in the DLQ are failing at the consumer - either transactions
>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
>> client calling Session.recover().
>>
>> See here for more details:
>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>>
>> You are right about the webconsole -- it will most likely stop functioning
>> once the broker runs out of memory.
>>
>> >
>> >
>> > I have producerFlowControl to false. I seperated my producer and
>> > consumers.
>> > But still I was able to reproduce issue. For some time bunch of messages
>> > started pile up in queue. Then they started to endup in DLQ. In DLQ I
>> saw
>> > most of them have these property:
>> >
>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
>> > useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay
>> =
>> > 1000}, cause:null
>> >
>> > Does this thing has anything to do with the issue?
>> >
>> > I was not monitoring heap size when it got stuck. that will be next
>> thing
>> > to do. But I can hit webconsole from that activemq instance. So if it
>> had
>> > memory issues how come console is still working? isn't it same process?
>> >
>> > thanks,
>> > chirag
>> >
>> >
>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef wrote:
>> >
>> >> How about a stack trace on the client? Can you look for the consumer
>> >> threads
>> >> and see what they are doing?
>> >>
>> >> Note that with producers and consumers on the same connection, it's
>> >> possible
>> >> to reach a deadlock if producer-flow-control kicks in because the
>> entire
>> >> connection is blocked, not just the one producer. There are two ways
>> >> this
>> >> can happen. First, if the client produces and consumes the same
>> >> destination. Second, if the client is producing one destination and
>> >> consuming a second destination and another client is consuming the
>> first
>> >> and
>> >> producing to the second. Actually, more complicated possibilities
>> exist
>> >> as
>> >> well.
>> >>
>> >> Your best bet is to never consume and produce on the same connection so
>> >> that
>> >> consumption never blocks due to producer-flow-control. In that
>> >> scenario,
>> >> deadlocks won't happen because consumers can always consume.
>> >>
>> >> Back to stack traces - if the consuming threads are blocked waiting to
>> >> obtain a lock while producing threads hold that lock, that could
>> >> indicate
>> >> the problem.
>> >>
>> >> Anyway, slow consumption is quite often the cause of broker hangs, so
>> >> looking for slow consumption and understanding speeding consumption (or
>> >> slowing production) is important. If the broker JVM becomes completely
>> >> unresponsive, try connecting jconsole or visualvm before the broker
>> >> hangs
>> >> and then watch memory. If the JVM is running out of Heap or Permgen
>> >> space,
>> >> that would explain the hung broker.
>> >>
>> >>
>> >>
>> >> --
>> >> View this mes
Re: Broker Hangs after some time - or does nothing.
i said broker hangs bcs if i tried to send more messages or messages on
queue doent get processed. i cannot even create queue or send message using
webconsole.
will try again and rhis time will moniter memory.
On Feb 10, 2014 6:22 PM, "artnaseef" wrote:<
>
> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
> symptoms.
>
> The messages in the DLQ are failing at the consumer - either transactions
> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
> client calling Session.recover().
>
> See here for more details:
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>
> You are right about the webconsole -- it will most likely stop functioning
> once the broker runs out of memory.
>
> >
> >
> > I have producerFlowControl to false. I seperated my producer and
> > consumers.
> > But still I was able to reproduce issue. For some time bunch of messages
> > started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
> > most of them have these property:
> >
> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
> > useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
> > 1000}, cause:null
> >
> > Does this thing has anything to do with the issue?
> >
> > I was not monitoring heap size when it got stuck. that will be next thing
> > to do. But I can hit webconsole from that activemq instance. So if it had
> > memory issues how come console is still working? isn't it same process?
> >
> > thanks,
> > chirag
> >
> >
> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef wrote:
> >
> >> How about a stack trace on the client? Can you look for the consumer
> >> threads
> >> and see what they are doing?
> >>
> >> Note that with producers and consumers on the same connection, it's
> >> possible
> >> to reach a deadlock if producer-flow-control kicks in because the entire
> >> connection is blocked, not just the one producer. There are two ways
> >> this
> >> can happen. First, if the client produces and consumes the same
> >> destination. Second, if the client is producing one destination and
> >> consuming a second destination and another client is consuming the first
> >> and
> >> producing to the second. Actually, more complicated possibilities exist
> >> as
> >> well.
> >>
> >> Your best bet is to never consume and produce on the same connection so
> >> that
> >> consumption never blocks due to producer-flow-control. In that
> >> scenario,
> >> deadlocks won't happen because consumers can always consume.
> >>
> >> Back to stack traces - if the consuming threads are blocked waiting to
> >> obtain a lock while producing threads hold that lock, that could
> >> indicate
> >> the problem.
> >>
> >> Anyway, slow consumption is quite often the cause of broker hangs, so
> >> looking for slow consumption and understanding speeding consumption (or
> >> slowing production) is important. If the broker JVM becomes completely
> >> unresponsive, try connecting jconsole or visualvm before the broker
> >> hangs
> >> and then watch memory. If the JVM is running out of Heap or Permgen
> >> space,
> >> that would explain the hung broker.
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> >
> > ___________
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
> > To start a new topic under ActiveMQ - User, email
> > ml-node+s2283324n234180...@n4.nabble.com
> > To unsubscribe from ActiveMQ - User, visit
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
symptoms.
The messages in the DLQ are failing at the consumer - either transactions
rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
client calling Session.recover().
See here for more details:
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
You are right about the webconsole -- it will most likely stop functioning
once the broker runs out of memory.
>
>
> I have producerFlowControl to false. I seperated my producer and
> consumers.
> But still I was able to reproduce issue. For some time bunch of messages
> started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
> most of them have these property:
>
> dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
> limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
> 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
> initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
> useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
> 1000}, cause:null
>
> Does this thing has anything to do with the issue?
>
> I was not monitoring heap size when it got stuck. that will be next thing
> to do. But I can hit webconsole from that activemq instance. So if it had
> memory issues how come console is still working? isn't it same process?
>
> thanks,
> chirag
>
>
> On Sat, Feb 8, 2014 at 12:36 AM, artnaseef wrote:
>
>> How about a stack trace on the client? Can you look for the consumer
>> threads
>> and see what they are doing?
>>
>> Note that with producers and consumers on the same connection, it's
>> possible
>> to reach a deadlock if producer-flow-control kicks in because the entire
>> connection is blocked, not just the one producer. There are two ways
>> this
>> can happen. First, if the client produces and consumes the same
>> destination. Second, if the client is producing one destination and
>> consuming a second destination and another client is consuming the first
>> and
>> producing to the second. Actually, more complicated possibilities exist
>> as
>> well.
>>
>> Your best bet is to never consume and produce on the same connection so
>> that
>> consumption never blocks due to producer-flow-control. In that
>> scenario,
>> deadlocks won't happen because consumers can always consume.
>>
>> Back to stack traces - if the consuming threads are blocked waiting to
>> obtain a lock while producing threads hold that lock, that could
>> indicate
>> the problem.
>>
>> Anyway, slow consumption is quite often the cause of broker hangs, so
>> looking for slow consumption and understanding speeding consumption (or
>> slowing production) is important. If the broker JVM becomes completely
>> unresponsive, try connecting jconsole or visualvm before the broker
>> hangs
>> and then watch memory. If the JVM is running out of Heap or Permgen
>> space,
>> that would explain the hung broker.
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> ___
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n234180...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
I have producerFlowControl to false. I seperated my producer and consumers.
But still I was able to reproduce issue. For some time bunch of messages
started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
most of them have these property:
dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
1000}, cause:null
Does this thing has anything to do with the issue?
I was not monitoring heap size when it got stuck. that will be next thing
to do. But I can hit webconsole from that activemq instance. So if it had
memory issues how come console is still working? isn't it same process?
thanks,
chirag
On Sat, Feb 8, 2014 at 12:36 AM, artnaseef wrote:
> How about a stack trace on the client? Can you look for the consumer
> threads
> and see what they are doing?
>
> Note that with producers and consumers on the same connection, it's
> possible
> to reach a deadlock if producer-flow-control kicks in because the entire
> connection is blocked, not just the one producer. There are two ways this
> can happen. First, if the client produces and consumes the same
> destination. Second, if the client is producing one destination and
> consuming a second destination and another client is consuming the first
> and
> producing to the second. Actually, more complicated possibilities exist as
> well.
>
> Your best bet is to never consume and produce on the same connection so
> that
> consumption never blocks due to producer-flow-control. In that scenario,
> deadlocks won't happen because consumers can always consume.
>
> Back to stack traces - if the consuming threads are blocked waiting to
> obtain a lock while producing threads hold that lock, that could indicate
> the problem.
>
> Anyway, slow consumption is quite often the cause of broker hangs, so
> looking for slow consumption and understanding speeding consumption (or
> slowing production) is important. If the broker JVM becomes completely
> unresponsive, try connecting jconsole or visualvm before the broker hangs
> and then watch memory. If the JVM is running out of Heap or Permgen space,
> that would explain the hung broker.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
How about a stack trace on the client? Can you look for the consumer threads and see what they are doing? Note that with producers and consumers on the same connection, it's possible to reach a deadlock if producer-flow-control kicks in because the entire connection is blocked, not just the one producer. There are two ways this can happen. First, if the client produces and consumes the same destination. Second, if the client is producing one destination and consuming a second destination and another client is consuming the first and producing to the second. Actually, more complicated possibilities exist as well. Your best bet is to never consume and produce on the same connection so that consumption never blocks due to producer-flow-control. In that scenario, deadlocks won't happen because consumers can always consume. Back to stack traces - if the consuming threads are blocked waiting to obtain a lock while producing threads hold that lock, that could indicate the problem. Anyway, slow consumption is quite often the cause of broker hangs, so looking for slow consumption and understanding speeding consumption (or slowing production) is important. If the broker JVM becomes completely unresponsive, try connecting jconsole or visualvm before the broker hangs and then watch memory. If the JVM is running out of Heap or Permgen space, that would explain the hung broker. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
5:35.803 > >> > > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1] > >> > DEBUG org.apache.activemq.ActiveMQSession - > >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null > >> > 14:15:36.803 > >> > > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1] > >> > DEBUG org.apache.activemq.ActiveMQSession - > >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null > >> > 14:15:37.803 > >> > > >> > [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1] > >> > DEBUG org.apache.activemq.ActiveMQSession - > >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null > >> > > >> > and it just keep on going and never stops. I am not worried about > >> > healthcheck stuff right now. It's just there and printing logs so > >> ignore > >> > that part. > >> > > >> > this behavior happens when activemq borker is frozen. I mean queue > >> count > >> > is > >> > stuck on 230. > >> > > >> > thanks, > >> > chirag > >> > > >> > > >> > On Wed, Feb 5, 2014 at 8:07 PM, artnaseef wrote: > >> > > >> >> There are many things to check. > >> >> > >> >> First off, I have a little confusion. You metion no log messages on > >> the > >> >> client or server and then show log messages. By server, do you mean > >> the > >> >> broker or something else? > >> >> > >> >> Troubleshooting steps: > >> >> * When flow stops, check the broker for active client connections via > >> >> the > >> >> webconsole or JMX. > >> >> * Check consumer and producer counts for the destinations. > >> >> * If the failover transport is being used, be sure to use a > >> >> ConnectionListener and TransportListener to receive notifications of > >> >> lost > >> >> connection to the broker. Note that you should get a log message if > >> the > >> >> failover transport loses connection, so this may not be necessary. > >> >> * Check broker JVM memory (especially Heap and PermGen) > >> >> > >> >> Also, if the broker is running out of memory due to having too many > >> >> messages > >> >> (i.e. slow consumer problem), that could cause the broker to stop > >> >> functioning. > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html > >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> >> > >> > > >> > > >> > > >> > > >> > ___ > >> > If you reply to this email, your message will be added to the > >> discussion > >> > below: > >> > > >> > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html > >> > To start a new topic under ActiveMQ - User, email > >> > ml-node+s2283324n234180...@n4.nabble.com > >> > To unsubscribe from ActiveMQ - User, visit > >> > > >> > http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== > >> > >> > >> > >> > >> > >> > >> -- > >> View this message in context: > >> > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > > > > > > > > > > ___ > > If you reply to this email, your message will be added to the discussion > > below: > > > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677642.html > > To start a new topic under ActiveMQ - User, email > > ml-node+s2283324n234180...@n4.nabble.com > > To unsubscribe from ActiveMQ - User, visit > > > http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== > > > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677643.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: Broker Hangs after some time - or does nothing.
about >> > healthcheck stuff right now. It's just there and printing logs so >> ignore >> > that part. >> > >> > this behavior happens when activemq borker is frozen. I mean queue >> count >> > is >> > stuck on 230. >> > >> > thanks, >> > chirag >> > >> > >> > On Wed, Feb 5, 2014 at 8:07 PM, artnaseef wrote: >> > >> >> There are many things to check. >> >> >> >> First off, I have a little confusion. You metion no log messages on >> the >> >> client or server and then show log messages. By server, do you mean >> the >> >> broker or something else? >> >> >> >> Troubleshooting steps: >> >> * When flow stops, check the broker for active client connections via >> >> the >> >> webconsole or JMX. >> >> * Check consumer and producer counts for the destinations. >> >> * If the failover transport is being used, be sure to use a >> >> ConnectionListener and TransportListener to receive notifications of >> >> lost >> >> connection to the broker. Note that you should get a log message if >> the >> >> failover transport loses connection, so this may not be necessary. >> >> * Check broker JVM memory (especially Heap and PermGen) >> >> >> >> Also, if the broker is running out of memory due to having too many >> >> messages >> >> (i.e. slow consumer problem), that could cause the broker to stop >> >> functioning. >> >> >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >> > >> > >> > >> > >> > ___ >> > If you reply to this email, your message will be added to the >> discussion >> > below: >> > >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html >> > To start a new topic under ActiveMQ - User, email >> > ml-node+s2283324n234180...@n4.nabble.com >> > To unsubscribe from ActiveMQ - User, visit >> > >> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >> >> >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > > > ___ > If you reply to this email, your message will be added to the discussion > below: > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677642.html > To start a new topic under ActiveMQ - User, email > ml-node+s2283324n234180...@n4.nabble.com > To unsubscribe from ActiveMQ - User, visit > http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677643.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
eck consumer and producer counts for the destinations. > >> * If the failover transport is being used, be sure to use a > >> ConnectionListener and TransportListener to receive notifications of > >> lost > >> connection to the broker. Note that you should get a log message if the > >> failover transport loses connection, so this may not be necessary. > >> * Check broker JVM memory (especially Heap and PermGen) > >> > >> Also, if the broker is running out of memory due to having too many > >> messages > >> (i.e. slow consumer problem), that could cause the broker to stop > >> functioning. > >> > >> > >> > >> > >> -- > >> View this message in context: > >> > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > > > > > > > > > > ___ > > If you reply to this email, your message will be added to the discussion > > below: > > > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html > > To start a new topic under ActiveMQ - User, email > > ml-node+s2283324n234180...@n4.nabble.com > > To unsubscribe from ActiveMQ - User, visit > > > http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== > > > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: Broker Hangs after some time - or does nothing.
Hmm, that is very interesting. Is the client a C++ (activemq-cpp) client?
So, first off, the "Connection reset" error means a client is losing
connection to the broker without cleaning up first - not sure why, but
that itself isn't indicative of your problem. Keep it in mind because it
may be triggering the problem, but it's not the cause.
This looks like it could be a bug in the producer, but that's premature to
assert. Can you look at the producer code to see how it handles failed
sends? It looks like the producer may be getting stuck on a failure.
Check how the producer handles connection to the broker when the
connection fails. If the client is using the failover transport, that
shouldn't be an issue - although it depends on the transport settings.
>
>
> Thanks for reply.
>
> -I checked consumer and produce count on queue and they are 99(c) and 7(p)
> respectively.
> - I see this warning messgage randomly in broker logs: "2014-02-06
> 13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669 failed:
> java.net.SocketException: Connection reset |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///[ip]:63669@61616"
>
> and on client side I see following debug messages:
>
> 14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
> message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
> messageId = null, originalDestination = null, originalTransactionId =
> null,
> producerId = null, destination = null, transactionId = null, expiration =
> 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
> correlationId = null, replyTo = null, persistent = false, type = null,
> priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null,
> compressed = false, userID = null, content = null, marshalledProperties =
> null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> null, readOnlyProperties = false, readOnlyBody = false, droppable = false,
> text = }
> 14:15:33.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:34.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:35.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:36.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:37.803
> [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>
> and it just keep on going and never stops. I am not worried about
> healthcheck stuff right now. It's just there and printing logs so ignore
> that part.
>
> this behavior happens when activemq borker is frozen. I mean queue count
> is
> stuck on 230.
>
> thanks,
> chirag
>
>
> On Wed, Feb 5, 2014 at 8:07 PM, artnaseef wrote:
>
>> There are many things to check.
>>
>> First off, I have a little confusion. You metion no log messages on the
>> client or server and then show log messages. By server, do you mean the
>> broker or something else?
>>
>> Troubleshooting steps:
>> * When flow stops, check the broker for active client connections via
>> the
>> webconsole or JMX.
>> * Check consumer and producer counts for the destinations.
>> * If the failover transport is being used, be sure to use a
>> ConnectionListener and TransportListener to receive notifications of
>> lost
>> connection to the broker. Note that you should get a log message if the
>> failover transport loses connection, so this may not be necessary.
>> * Check broker JVM memory (especially Heap and PermGen)
>>
>> Also, if the broker is running out of memory due to having too many
>> messages
>> (i.e. slow consumer problem), that could cause the broker to stop
>> functioning.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> ___
> If you
Re: Broker Hangs after some time - or does nothing.
Thanks for reply.
-I checked consumer and produce count on queue and they are 99(c) and 7(p)
respectively.
- I see this warning messgage randomly in broker logs: "2014-02-06
13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669 failed:
java.net.SocketException: Connection reset |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///[ip]:63669@61616"
and on client side I see following debug messages:
14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
messageId = null, originalDestination = null, originalTransactionId = null,
producerId = null, destination = null, transactionId = null, expiration =
0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
correlationId = null, replyTo = null, persistent = false, type = null,
priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null,
compressed = false, userID = null, content = null, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
null, readOnlyProperties = false, readOnlyBody = false, droppable = false,
text = }
14:15:33.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:34.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:35.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:36.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:37.803
[com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
and it just keep on going and never stops. I am not worried about
healthcheck stuff right now. It's just there and printing logs so ignore
that part.
this behavior happens when activemq borker is frozen. I mean queue count is
stuck on 230.
thanks,
chirag
On Wed, Feb 5, 2014 at 8:07 PM, artnaseef wrote:
> There are many things to check.
>
> First off, I have a little confusion. You metion no log messages on the
> client or server and then show log messages. By server, do you mean the
> broker or something else?
>
> Troubleshooting steps:
> * When flow stops, check the broker for active client connections via the
> webconsole or JMX.
> * Check consumer and producer counts for the destinations.
> * If the failover transport is being used, be sure to use a
> ConnectionListener and TransportListener to receive notifications of lost
> connection to the broker. Note that you should get a log message if the
> failover transport loses connection, so this may not be necessary.
> * Check broker JVM memory (especially Heap and PermGen)
>
> Also, if the broker is running out of memory due to having too many
> messages
> (i.e. slow consumer problem), that could cause the broker to stop
> functioning.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Broker Hangs after some time - or does nothing.
Ah, it's possible to change the subject. Got it. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677514.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
There are many things to check. First off, I have a little confusion. You metion no log messages on the client or server and then show log messages. By server, do you mean the broker or something else? Troubleshooting steps: * When flow stops, check the broker for active client connections via the webconsole or JMX. * Check consumer and producer counts for the destinations. * If the failover transport is being used, be sure to use a ConnectionListener and TransportListener to receive notifications of lost connection to the broker. Note that you should get a log message if the failover transport loses connection, so this may not be necessary. * Check broker JVM memory (especially Heap and PermGen) Also, if the broker is running out of memory due to having too many messages (i.e. slow consumer problem), that could cause the broker to stop functioning. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Broker Hangs after some time - or does nothing.
I am using Activemq-5.10.SNAPSHOT and implementing with replicated leveldb. After few messages consumer stop processing messages. Messages just stay on queue. nothing happens. Client cannot send anymore messges. I dont see any exception on client side. I dont see any exception on server logs. I saw some exception in logs : 2014-02-05 14:01:28,247 | DEBUG | Running WriteCheck[tcp:// 10.190.225.214:44179] | org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ InactivityMonitor Worker 2014-02-05 14:01:28,285 | DEBUG | Transport Connection to: tcp:// 10.190.237.125:58514 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.237.125:58514@61616 java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275) at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:724) 2014-02-05 14:01:28,285 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=activemq,connector=clientConnectors,connectorName=openwire,connectionViewType=clientId,connectionName=ID_RHEL6u2-001-1-38845-1391630198629-0_1 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport: tcp:///10.190.237.125:58514@61616 2014-02-05 14:01:28,285 | DEBUG | Transport Connection to: tcp:// 10.190.237.125:58519 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.237.125:58519@61616 java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275) at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:724) 2014-02-05 14:01:28,285 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=activemq,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//10.190.237.125_58514 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport: tcp:///10.190.237.125:58514@61616 but after that everything in normal. any suggestions where to start troubleshooting. thanks, chirag
