Hi, We use Spring DMLC to consume from one or more Queues. Async send and auto acknowledge are used. KahaDB is used for persistence. StoreUsage is set to 1g. But when this happens, the size is only around 200mb. So does not appear to be a flow control induced deadlock. There are multiple DMLC running on the same host. Each listener, consumes a message and depending the type of message, may again push few more messages to the queue. The input rate is around 200 rps, and each consumed message pushing 10 or more messages, the load on broker may be 2000 rps. The consumer pushes the messages back to broker using JMSTemplate. Both DMLC and JMSTemplate share the same PooledConnectionFactory. Did not see any exception on either the broker or consumer logs. Spring version is 2.5.6. ActiveMQ version is 5.5.1. Both are running on rhel 5.
Here are the consumer thread dumps: Of the 1173 threads 1013 are in this state: Thread 31315: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.Object.wait() @bci=2, line=485 (Compiled frame) - org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(long) @bci=46, line=87 (Compiled frame) - org.apache.activemq.ActiveMQMessageConsumer.dequeue(long) @bci=19, line=452 (Compiled frame) - org.apache.activemq.ActiveMQMessageConsumer.receive(long) @bci=44, line=575 (Compiled frame) - org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(javax.jms.MessageConsumer) @bci=23, line=405 (Compiled frame) - org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(java.lang.Object, javax.jms.Session, javax.jms.MessageConsumer, org.springframework.transaction.TransactionStatus) @bci=122, line=308 (Compiled frame) - org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(java.lang.Object, javax.jms.Session, javax.jms.MessageConsumer) @bci=97, line=261 (Interpreted frame) - org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener() @bci=17, line=982 (Interpreted frame) - org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop() @bci=130, line=974 (Interpreted frame) - org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run() @bci=51, line=876 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread dump from broker: There are around 40 pairs of these: Thread 9762: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.Object.wait() @bci=2, line=485 (Compiled frame) - org.apache.activemq.thread.DedicatedTaskRunner.runTask() @bci=134, line=111 (Compiled frame) - org.apache.activemq.thread.DedicatedTaskRunner$1.run() @bci=11, line=42 (Interpreted frame) Thread 9760: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Compiled frame) - org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill() @bci=31, line=50 (Compiled frame) - org.apache.activemq.transport.tcp.TcpTransport$2.fill() @bci=9, line=576 (Compiled frame) - org.apache.activemq.transport.tcp.TcpBufferedInputStream.read() @bci=12, line=58 (Compiled frame) - org.apache.activemq.transport.tcp.TcpTransport$2.read() @bci=9, line=561 (Compiled frame) - java.io.DataInputStream.readInt() @bci=4, line=370 (Compiled frame) - org.apache.activemq.openwire.OpenWireFormat.unmarshal(java.io.DataInput) @bci=10, line=269 (Compiled frame) - org.apache.activemq.transport.tcp.TcpTransport.readCommand() @bci=8, line=227 (Compiled frame) - org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=1, line=219 (Compiled frame) - org.apache.activemq.transport.tcp.TcpTransport.run() @bci=47, line=202 (Compiled frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Then this: Thread 21583: (state = IN_JAVA) - java.util.AbstractList$Itr.next() @bci=12, line=345 (Compiled frame; information may be imprecise) - org.apache.activemq.broker.region.Queue.doActualDispatch(java.util.List) @bci=171, line=1822 (Compiled frame) - org.apache.activemq.broker.region.Queue.doDispatch(java.util.List) @bci=54, line=1772 (Compiled frame) - org.apache.activemq.broker.region.Queue.pageInMessages(boolean) @bci=6, line=1932 (Compiled frame) - org.apache.activemq.broker.region.Queue.iterate() @bci=502, line=1440 (Compiled frame) - org.apache.activemq.thread.DedicatedTaskRunner.runTask() @bci=67, line=104 (Compiled frame) - org.apache.activemq.thread.DedicatedTaskRunner$1.run() @bci=11, line=42 (Interpreted frame) Have attached the thread dumps for both consumer and broker. Does not appear to be any deadlocks looking at thread dumps. But I may be wrong. Any pointers would be of great help. Thanks Preetam broker_threads.txt <http://activemq.2283324.n4.nabble.com/file/n4657796/broker_threads.txt> consumer_threads.txt <http://activemq.2283324.n4.nabble.com/file/n4657796/consumer_threads.txt> -- View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-not-consuming-though-Queue-has-items-ActiveMQ-5-5-1-and-Spring-DMLC-JMStemplate-tp4657796.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
