I tried again and this time I didnt send bunch of messages at same time. I noticed that some message gets processed right away but some messges stays on queue and doesnt get processed. If send few more messges let say 50 messages and it processes 40 messages and 10 msgs left on queue. I can send more messages but not all get processed.
I am attaching config file. I have enabled producerFlowControl and removed slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT. On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <chiragpuj...@gmail.com>wrote: > i said broker hangs bcs if i tried to send more messages or messages on > queue doent get processed. i cannot even create queue or send message using > webconsole. > > will try again and rhis time will moniter memory. > > On Feb 10, 2014 6:22 PM, "artnaseef" <a...@artnaseef.com> wrote:< >> >> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed >> symptoms. >> >> The messages in the DLQ are failing at the consumer - either transactions >> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE >> client calling Session.recover(). >> >> See here for more details: >> http://activemq.apache.org/message-redelivery-and-dlq-handling.html >> >> You are right about the webconsole -- it will most likely stop functioning >> once the broker runs out of memory. >> >> > >> > >> > I have producerFlowControl to false. I seperated my producer and >> > consumers. >> > But still I was able to reproduce issue. For some time bunch of messages >> > started pile up in queue. Then they started to endup in DLQ. In DLQ I >> saw >> > most of them have these property: >> > >> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy >> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor = >> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1, >> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false, >> > useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay >> = >> > 1000}, cause:null >> > >> > Does this thing has anything to do with the issue? >> > >> > I was not monitoring heap size when it got stuck. that will be next >> thing >> > to do. But I can hit webconsole from that activemq instance. So if it >> had >> > memory issues how come console is still working? isn't it same process? >> > >> > thanks, >> > chirag >> > >> > >> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <a...@artnaseef.com> wrote: >> > >> >> How about a stack trace on the client? Can you look for the consumer >> >> threads >> >> and see what they are doing? >> >> >> >> Note that with producers and consumers on the same connection, it's >> >> possible >> >> to reach a deadlock if producer-flow-control kicks in because the >> entire >> >> connection is blocked, not just the one producer. There are two ways >> >> this >> >> can happen. First, if the client produces and consumes the same >> >> destination. Second, if the client is producing one destination and >> >> consuming a second destination and another client is consuming the >> first >> >> and >> >> producing to the second. Actually, more complicated possibilities >> exist >> >> as >> >> well. >> >> >> >> Your best bet is to never consume and produce on the same connection so >> >> that >> >> consumption never blocks due to producer-flow-control. In that >> >> scenario, >> >> deadlocks won't happen because consumers can always consume. >> >> >> >> Back to stack traces - if the consuming threads are blocked waiting to >> >> obtain a lock while producing threads hold that lock, that could >> >> indicate >> >> the problem. >> >> >> >> Anyway, slow consumption is quite often the cause of broker hangs, so >> >> looking for slow consumption and understanding speeding consumption (or >> >> slowing production) is important. If the broker JVM becomes completely >> >> unresponsive, try connecting jconsole or visualvm before the broker >> >> hangs >> >> and then watch memory. If the JVM is running out of Heap or Permgen >> >> space, >> >> that would explain the hung broker. >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >> > >> > >> > >> > >> > _______________________________________________ >> > If you reply to this email, your message will be added to the discussion >> > below: >> > >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html >> > To start a new topic under ActiveMQ - User, email >> > ml-node+s2283324n234180...@n4.nabble.com >> > To unsubscribe from ActiveMQ - User, visit >> > >> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >> >> >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >
<?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans"; xmlns:amq="http://activemq.apache.org/schema/core"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd";> <!-- Allows us to use system properties as variables in this configuration file --> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <value>file:${activemq.conf}/credentials.properties</value> </property> </bean> <!-- The <broker> element is used to configure the ActiveMQ broker. --> <broker xmlns="http://activemq.apache.org/schema/core"; brokerName="activemq" dataDirectory="/opt/gwx/activemqdata"> <destinationPolicy> <policyMap> <policyEntries> <policyEntry producerFlowControl="false" topic=">"> <pendingMessageLimitStrategy> <constantPendingMessageLimitStrategy limit="1000"/> </pendingMessageLimitStrategy> </policyEntry> <policyEntry producerFlowControl="true" queue=">"> <deadLetterStrategy> <individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true"/> </deadLetterStrategy> </policyEntry> <!-- <policyEntry queue=">" > --> <!-- <slowConsumerStrategy> --> <!-- <abortSlowConsumerStrategy abortConnection="true"/> --> <!-- </slowConsumerStrategy> --> <!-- </policyEntry> --> </policyEntries> </policyMap> </destinationPolicy> <managementContext> <managementContext createConnector="false"/> </managementContext> <persistenceAdapter> <replicatedLevelDB bind="tcp://0.0.0.0:0" directory="/opt/gwx/activemqdata/leveldb" replicas="4" zkAddress="host-1:2181,host-1:2182,host-2:2181,host-2:2182" zkPassword="password" zkPath="/opt/gwx/activemqdata"/> </persistenceAdapter> <systemUsage> <systemUsage sendFailIfNoSpace="true"> <memoryUsage> <memoryUsage limit="256 mb"/> </memoryUsage> <storeUsage> <storeUsage limit="2 gb"/> </storeUsage> <tempUsage> <tempUsage limit="512 mb"/> </tempUsage> </systemUsage> </systemUsage> <transportConnectors> <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB --> <transportConnector name="openwire" uri="tcp://0.0.0.0:${openwirePort}?maximumConnections=1000&wireformat.maxFrameSize=104857600"/> <transportConnector name="amqp" uri="amqp://0.0.0.0:${amqpPort}?maximumConnections=1000&wireformat.maxFrameSize=104857600"/> </transportConnectors> <!-- destroy the spring context on shutdown to stop jetty --> <shutdownHooks> <bean xmlns="http://www.springframework.org/schema/beans"; class="org.apache.activemq.hooks.SpringContextHook"/> </shutdownHooks> </broker> <!-- Enable web consoles, REST and Ajax APIs and demos Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details --> <import resource="jetty.xml"/> </beans>
