[
https://issues.apache.org/jira/browse/AMQ-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223149#comment-13223149
]
Claudio Tagliola commented on AMQ-3755:
---------------------------------------
We had a hunch we had to move to a multiple VMTransport implementation,
however, I have some follow-up questions:
>From the documentation it is not clear what the limitations of the VMTransport
>exactly are, it talks about a JMS Client, but that is a bit of a vague notion
>inside the same JVM. It seems, one of the effects of this limitation is that
>the receiveNoWait call will start blocking, which is not obvious from the
>documentation.
Furthermore, if we move to multiple VMTransports, will the backup broker be
able to continue without blocking or will the bottleneck then surface inside
the broker?
It will take some time to get this modified and get it deployed in our
production environment.
> receiveNoWait blocked on same mutex as sending threads
> ------------------------------------------------------
>
> Key: AMQ-3755
> URL: https://issues.apache.org/jira/browse/AMQ-3755
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Reporter: Claudio Tagliola
>
> We have a server processing requests with an internal broker, connected with
> a vm transport. At a certain load and a high number of clients (approx. 350
> clients) we see our server congested towards the internal VM broker. By
> looking at the stacktraces, it seems the incoming reading thread is waiting
> for a lock obtained by one of several other threads. This resulted in a short
> period (around 20-30 seconds) of very sluggish response times towards our
> clients. Response times of _any_ request through ActiveMQ go from a normal
> <50ms response time towards 2 seconds.
> Below is an extract of a stackdump while this happened. The first stacktrace
> is the thread trying to read incoming messages with a receiveNoWait, but it
> is blocked by the first thread, which the second thread owns. The third and
> fourth are two other processes, all waiting for the same mutex. These four
> processes are - from our server's perspective - not related to each other:
> {noformat}
> "RoundRobinReceiver" prio=10 tid=0x00007f1030483800 nid=0x2a59 waiting for
> monitor entry [0x00007f1075f5b000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> - waiting to lock <0x00000006bb6028b0> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
> at
> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1863)
> at
> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2029)
> at
> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2024)
> at
> org.apache.activemq.ActiveMQMessageConsumer.afterMessageIsConsumed(ActiveMQMessageConsumer.java:871)
> - locked <0x0000000715bda898> (a java.util.LinkedList)
> at
> org.apache.activemq.ActiveMQMessageConsumer.receiveNoWait(ActiveMQMessageConsumer.java:617)
> at
> com.foo.bar.impl.server.services.jms.JMSRoundRobinListener.readFromConsumer(JMSRoundRobinListener.java:200)
> at
> com.foo.bar.impl.server.services.jms.JMSRoundRobinListener.roundRobinOverQueueConsumers(JMSRoundRobinListener.java:173)
> at
> com.foo.bar.impl.server.services.jms.JMSRoundRobinListener.receiveMessagesUntilInterrupt(JMSRoundRobinListener.java:109)
> at
> com.foo.bar.impl.server.services.jms.JMSRoundRobinListener.access$200(JMSRoundRobinListener.java:22)
> at
> com.foo.bar.impl.server.services.jms.JMSRoundRobinListener$RoundRobinReceiver.run(JMSRoundRobinListener.java:248)
> at java.lang.Thread.run(Thread.java:722)
> "ResultPublisher1" prio=10 tid=0x00007f0f20003000 nid=0x2b31 waiting on
> condition [0x00007f1060d59000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006bb600410> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
> at
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:94)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> - locked <0x00000006bb6028b0> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
> at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1744)
> - locked <0x00000006bc4bd448> (a java.lang.Object)
> at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
> at
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
> at
> com.foo.bar.impl.server.services.jms.MappingResultPublisher.publishResponse(MappingResultPublisher.java:66)
> at
> com.foo.bar.impl.server.services.jms.QueuedMappingResultPublisher.executeJob(QueuedMappingResultPublisher.java:44)
> at
> com.foo.bar.impl.server.services.mapper.AbstractQueuedMappingJobExecutor.processJob(AbstractQueuedMappingJobExecutor.java:120)
> at
> com.foo.bar.impl.server.services.mapper.AbstractQueuedMappingJobExecutor.access$100(AbstractQueuedMappingJobExecutor.java:23)
> at
> com.foo.bar.impl.server.services.mapper.AbstractQueuedMappingJobExecutor$1.run(AbstractQueuedMappingJobExecutor.java:105)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> "RemoteResolveRequestService-ReplierThread-1" prio=10 tid=0x00007f1030485000
> nid=0x2a5a waiting for monitor entry [0x00007f1075e5a000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> - waiting to lock <0x00000006bb6028b0> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
> at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1744)
> - locked <0x00000006baf131f8> (a java.lang.Object)
> at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
> at
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
> at
> com.foo.bar.remoteservices.server.RpcReplierTask.sendReply(RpcReplierTask.java:129)
> at
> com.foo.bar.remoteservices.server.RpcReplierTask.run(RpcReplierTask.java:103)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> "PersistEntityMessagePublisherThread" daemon prio=10 tid=0x00007f1030b70800
> nid=0x2a4b waiting for monitor entry [0x00007f1076c68000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> - waiting to lock <0x00000006bb6028b0> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
> at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1744)
> - locked <0x00000006bc542e90> (a java.lang.Object)
> at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
> at
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
> at
> com.foo.bar.impl.server.services.PersistEntityMessageBackgroundPublisher.sendMessage(PersistEntityMessageBackgroundPublisher.java:140)
> at
> com.foo.bar.impl.server.services.PersistEntityMessageBackgroundPublisher.sendMessage(PersistEntityMessageBackgroundPublisher.java:132)
> at
> com.foo.bar.impl.server.services.PersistEntityMessageBackgroundPublisher.forwardMessagesToTopic(PersistEntityMessageBackgroundPublisher.java:101)
> at
> com.foo.bar.impl.server.services.PersistEntityMessageBackgroundPublisher.access$000(PersistEntityMessageBackgroundPublisher.java:22)
> at
> com.foo.bar.impl.server.services.PersistEntityMessageBackgroundPublisher$1.run(PersistEntityMessageBackgroundPublisher.java:49)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
> My question is: why is the receiving of messages blocked by the other threads
> and why do they share a single mutex? Is there anything I can do to alleviate
> this problem?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
