Stirling Chow created AMQ-4151:
----------------------------------
Summary: Duplicate non-persistent messages that are sent to a
queue are either dispatched multiple times (i.e., not detected as duplicates)
or cause the queue size to be miscalculated.
Key: AMQ-4151
URL: https://issues.apache.org/jira/browse/AMQ-4151
Project: ActiveMQ
Issue Type: Bug
Reporter: Stirling Chow
Symptom
=======
We have a virtual topic that is shared by a network of brokers. Each broker
has consumers that process the corresponding Consumer.*.VirtualTopic.> queue.
While testing the effects of memory limits on our system, we encountered
AMQ-4148, which resulted in duplicate subscriptions being made to the virtual
topic. These duplicate subcriptions were attempting to enqueue the same topic
message multiple times to the corresponding Consumer.*.VirtualTopic.>.
Although logic exists in queues to handle duplicate messages, we noticed lots
of strange behaviour with duplicate handling. We decided to investigate
further and found a race condition whereby duplicate non-persistent messages
that were correctly ignored resulted in the queue size appearing to be
non-empty when in fact there were no messages.
While our original investigation was prompted by AMQ-4148, the test case we
attached to this ticket does not rely on the misbehaviour caused by AMQ-4148,
but can occur during normal operation of network bridges when there are
multiple consumers for a virtual topic (as is the case, e.g., when conduit
subscriptions are disabled).
Cause
=====
If a virtual topic has multiple consumers, then sending a message to the topic
results in multiple dispatches of the same message (one to each consumer). If
the multiple consumers originate from the same remote broker (e.g., because
conduit subscriptions are disabled), then a single message sent to the virtual
topic on broker1 will result in multiple duplicate messages being sent to the
virtual topic on broker2.
If broker2 has corresponding Consumer.*.VirtualTopic.> queue, then the multiple
duplicate messages sent to the virtual topic on broker2 will result in multiple
duplicate messages being sent to the queue.
If the messages are non-persistent, the only logic that prevents each duplicate
message from being dispatched to a consumer is as follows:
{code:title=Queue.java}
private PendingList doPageInForDispatch(boolean force) throws Exception {
...
// Only add new messages, not already pagedIn to avoid multiple
// dispatch attempts
pagedInMessagesLock.writeLock().lock();
try {
if(isPrioritizedMessages()) {
resultList = new PrioritizedPendingList();
} else {
resultList = new OrderedPendingList();
}
for (QueueMessageReference ref : result) {
if (!pagedInMessages.containsKey(ref.getMessageId())) {
pagedInMessages.put(ref.getMessageId(), ref);
resultList.addMessageLast(ref);
} else {
ref.decrementReferenceCount();
}
}
} finally {
pagedInMessagesLock.writeLock().unlock();
}
...
{code}
If the consumers are fast and acknowledge the initial message dispatch before
the next duplicate message is sent to the queue, then the
{{pagedInMessages.constainsKey(...)}} check will *not* prevent the duplicate
message from being dispatched since the message ID will have already been
removed as part of the acknowledgement.
This is problem 1: duplicate detection fails if acknowledgements are quick
If the consumers are slow and don't acknowledge the initial message dispatch
before the next duplicate message is sent to the queue, then duplicate
detection will correctly ignore the message. However, by this time, the
duplicate message has already incremented the queue size:
{code:title=Queue.java}
final void messageSent(final ConnectionContext context, final Message msg)
throws Exception {
destinationStatistics.getEnqueues().increment();
destinationStatistics.getMessages().increment();
messageDelivered(context, msg);
{code}
The call to {{messageSent}} is made by the thread sending the message to the
queue. This thread is completely unaware that the message was ignored since
the call to {{doPageInForDispatch}} is done by a separate thread (e.g., the
taskrunner calling {{iterate()}}.
Since the queue size is incremented, but the duplicate message is never
dispatched, there is no subsequent acknowledgement ot reduce the queue size.
As a result, each duplicate message that is ignored remains counted in the
queue size even though it's no longer in the queue.
This is problem 2: if duplication detection succeeds, the queue size
incorrectly counts the duplicate/ignored message.
Which problem gets exhibited depends on a race condition between the thread(s)
enqueueing the duplicate virtual topic subscriptions and consumer threads
acknowledging the dispatches.
While the unit test's methodology of creating multiple virtual topic consumers
is somewhat contrived, there must be other circumstances under which duplicate
messages are sent to queues (otherwise, why would there be logic to handle this
case?) In this context, the test case reveals a problem with the handling of
duplicate messages.
Solution
========
None at this time. We worked around the problem by patching AMQ-4148 to
prevent the duplicate subscriptions that cause this bug. However, our concern
has been raised about AMQ's generally handling of duplicate messages sent to a
queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
