[
https://issues.apache.org/activemq/browse/AMQ-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stirling Chow updated AMQ-2552:
-------------------------------
Description:
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message
that does expire is added to the end of the queue, the added message will not
be processed by the Broker's expiration logic until some of the 400+ unexpired
messages are consumed. A unit test is provided to demonstrate this issue.
These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
Motivation
========
Our application uses message expiration to trigger actions when messages are
left in a queue for too long a period. In many cases, there are no consumers
on the queue containing the messages that might expire. Prior versions of AMQ
only expired messages when an attempt was made to dispatch them to a consumer,
but AMQ-1112 introduced the notion of periodic, automatic expiration of pending
messages (see the expireMessagesPeriod policy). With this periodic expiration,
our expectation is that expired messages will be detected and processed in a
timely fashion, regardless of the presence of consumers on the queue.
Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages()
method:
>>
doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
<<
doBrowse(browsedMessages, this.getMaxExpirePageSize());
The purpose of the change was to limit the number of messages that would be
paged into memory from the message store. The side effect is that only the
first 400 messages in the queue are ever inspected by expireMessages(). If
these 400 messages do not contain the messages that are expiring and there are
no active consumers to move the expiring messages forward, then expiration will
never occur. With AMQ 5.3, successive calls to expireMessages() would
eventually pull all messages into the page so they could be checked for
expiration.
It should also be noted that the test cases can be modified so there is a
consumer (with a selector that doesn't choose any of the messages), and they
still fail showing that the issue is not purely the result of having *no*
consumers.
Resolution
=========
I don't think the browse idiom works very well for the automated expiration
task. It was my initial assumption (and expectation) that the
expireMessagesPeriod task would process *all* expired messages in the queue
*each time* the task fired. A related issue to limiting the number of messages
that are examined during each expiration task is that expiring messages at the
end of the queue will not be handled in a timely fashion since they must wait
for multiple firings of the expiration task (this is on 5.3 code; 5.3.1 code
will never get to those messages).
If the desire/need is to scan all messages to determine which have expired, the
existing paging strategy is limited because it would require all messages to be
brought from store into memory, thus defeating the whole point of paging. An
alternative might be to have a sliding page window that would allow batches of
messages to be brought in while clearing out the previous batch.
Another solution might be to have the expiration task page in directly from the
store, but this is problematic since non-persistent messages would still have
to come in from the pending message cursor. In addition, a complete scan of
the store might be non-performant for large queues.
I think the most satisfactory solution for minimizing unnecessary processing
and providing timely expiration of messages is to implement a proper heap that
indexes messages with expiration values and allows immediate access to *only*
the messages that have expired. The references to these expired messages could
then be used to remove the messages directly from the cursors and the store
rather than having to iterate over these structures.
was:
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message
that does expire is added to the end of the queue, the added message will not
be processed by the Broker's expiration logic until some of the 400+ unexpired
messages are consumed. A unit test is provided to demonstrate this issue.
These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
Motivation
========
Our application uses message expiration to trigger actions when messages are
left in a queue for too long a period. In many cases, there are no consumers
on the queue containing the messages that might expire. Prior versions of AMQ
only expired messages when an attempt was made to dispatch them to a consumer,
but AMQ-1112 introduced the notion of periodic, automatic expiration of pending
messages (see the expireMessagesPeriod policy). With this periodic expiration,
our expectation is that expired messages will be detected and processed in a
timely fashion, regardless of the presence of consumers on the queue.
Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages()
method:
>>
doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
<<
doBrowse(browsedMessages, this.getMaxExpirePageSize());
The purpose of the change was to limit the number of messages that would be
paged into memory from the message store. The side effect is that only the
first 400 messages in the queue are ever inspected by expireMessages(). If
these 400 messages do not contain the messages that are expiring and there are
no active consumers to move the expiring messages forward, then expiration will
never occur. With AMQ 5.3, successive calls to expireMessages() would
eventually pull all messages into the page so they could be checked for
expiration.
It should also be noted that the test cases can be modified so there is a
consumer (with a selector that doesn't choose any of the messages), and they
still fail showing that the issue is not purely the result of having *no*
consumers.
Resolution
=========
I don't think the browse idiom works very well for the automated expiration
task. It was my initial assumption (and expectation) that the
expireMessagesPeriod task would process *all* expired messages in the queue
*each time* the task fired. A related issue to limiting the number of messages
that are examined during each expiration task is that expiring messages at the
end of the queue will not be handled in a timely fashion since they must wait
for multiple firings of the expiration task (this is on 5.3 code; 5.3.1 code
will never get to those messages).
If the desire/need is to scan all messages to determine which have expired, the
existing paging strategy is limited because it would require all messages to be
brought from store into memory, thus defeating the whole point of paging. An
alternative might be to have a sliding page window that would allow batches of
messages to be brought in while clearing out the previous batch.
Another solution might be to have the expiration task page in directly from the
store, but this is problematic since non-persistent messages would still have
to come in from the pending message cursor. In addition, a complete scan of
the store might be non-performant for large queues.
I think the most satisfactory solution for minimizing unecessary processing and
providing timely expiration of messages is to implement a proper heap that
indexes messages with expiration values and allows immediate access to *only*
the messages that have expired. The references to these expired messages could
then be used to remove the messages from the cursors and the store.
> Automatic message expiration may not occur on a queue with no (or slow)
> consumers.
> ----------------------------------------------------------------------------------
>
> Key: AMQ-2552
> URL: https://issues.apache.org/activemq/browse/AMQ-2552
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Environment: ActiveMQ 5.3.1 Snapshot r893661
> Reporter: Stirling Chow
> Attachments: AMQTest.java
>
>
> Symptom
> ========
> If a queue is loaded with 400+ messages that never expire, and then a message
> that does expire is added to the end of the queue, the added message will not
> be processed by the Broker's expiration logic until some of the 400+
> unexpired messages are consumed. A unit test is provided to demonstrate this
> issue.
> These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
> Motivation
> ========
> Our application uses message expiration to trigger actions when messages are
> left in a queue for too long a period. In many cases, there are no consumers
> on the queue containing the messages that might expire. Prior versions of
> AMQ only expired messages when an attempt was made to dispatch them to a
> consumer, but AMQ-1112 introduced the notion of periodic, automatic
> expiration of pending messages (see the expireMessagesPeriod policy). With
> this periodic expiration, our expectation is that expired messages will be
> detected and processed in a timely fashion, regardless of the presence of
> consumers on the queue.
> Cause
> ======
> AMQ-2481 removed forced paging from the periodically-called expireMessages()
> method:
> >>
> doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> <<
> doBrowse(browsedMessages, this.getMaxExpirePageSize());
> The purpose of the change was to limit the number of messages that would be
> paged into memory from the message store. The side effect is that only the
> first 400 messages in the queue are ever inspected by expireMessages(). If
> these 400 messages do not contain the messages that are expiring and there
> are no active consumers to move the expiring messages forward, then
> expiration will never occur. With AMQ 5.3, successive calls to
> expireMessages() would eventually pull all messages into the page so they
> could be checked for expiration.
> It should also be noted that the test cases can be modified so there is a
> consumer (with a selector that doesn't choose any of the messages), and they
> still fail showing that the issue is not purely the result of having *no*
> consumers.
> Resolution
> =========
> I don't think the browse idiom works very well for the automated expiration
> task. It was my initial assumption (and expectation) that the
> expireMessagesPeriod task would process *all* expired messages in the queue
> *each time* the task fired. A related issue to limiting the number of
> messages that are examined during each expiration task is that expiring
> messages at the end of the queue will not be handled in a timely fashion
> since they must wait for multiple firings of the expiration task (this is on
> 5.3 code; 5.3.1 code will never get to those messages).
> If the desire/need is to scan all messages to determine which have expired,
> the existing paging strategy is limited because it would require all messages
> to be brought from store into memory, thus defeating the whole point of
> paging. An alternative might be to have a sliding page window that would
> allow batches of messages to be brought in while clearing out the previous
> batch.
> Another solution might be to have the expiration task page in directly from
> the store, but this is problematic since non-persistent messages would still
> have to come in from the pending message cursor. In addition, a complete
> scan of the store might be non-performant for large queues.
> I think the most satisfactory solution for minimizing unnecessary processing
> and providing timely expiration of messages is to implement a proper heap
> that indexes messages with expiration values and allows immediate access to
> *only* the messages that have expired. The references to these expired
> messages could then be used to remove the messages directly from the cursors
> and the store rather than having to iterate over these structures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
