On 5 Sep 2006, at 20:04, Fateev, Maxim wrote:
Cool, it is really close to the design I was working on. I think
there are a few issues with it that should be addressed:
1. Queue should not keep list of references to all messages in
flight. It leads to requirement that MessageStore should deal with
redriving messages that weren't acked. It also means that messages
should never be dispatched directly to consumers from producers as
it happens right now in Queue.send(...) method. They should always
be put into MessageStore.
Messages are always put into storage - but they are additionally
pushed to subscribers for dispatching. I was going to implement a
mixed approach - where consumers that can keep up with the producer
will not be impeded by waiting for a poll, but slower/non-active
consumers would ultimately rely on a poll from the store.
2. Such MessageStore API is not really compatible with message
selectors in case of Queues. For example if you have single
consumer with message selector that selects one message from 1000
and you decide to keep 500 messages in memory then this consumer
will not get its messages ever. IMHO message selectors in Queues
cannot be implemented really efficiently without keeping all
messages in memory and thus should not be supported.
Message selectors for Queues will always be an issue - but
unfortunately the use of a message selector on a Queue is part of the
JMS specification. We could make recommendations that if using
selectors, we recommend that there should always be a set of active
consumers able to select all messages from the Queue.
3. It is not compatible with current subscription mechanism.
Subscriptions should either stop accepting messages when prefetch
buffers are full or poll for messages when needed. But current
implementation expects that ALL messages are pushed to every
subscription.
Yes - am changing - so the PendingMessageCursor can in effect ignore
new messages from a producer if it is full - or the usage manager has
reached it's limit - and poll for them later.
3. getNextMessage(...) should block if there are no messages in a
queue. Otherwise it would require busy polling.
depends on the implementation - more messages available can be
triggered from producer etc - providing the PendingMessageCursor is
intialized with state and maintains it.
4. I'm not sure about semantic of getNextMessage(...,
RecoveryListener) API. Will it create a thread per request? Will
single blocked listener block the whole queue? I think "List
getNextMessage(..., int count)" would have cleaner semantic.
I think a Listener allows for more flexibility - in allowing the
PendingMessageCursor to stop a request if the usage memory is at its
limit for example
I think the proposed changes are too big and potentially dangerous
to be put as fixes to existing Queue and Topic implementations. I
would recommend implementing alternative Queue and Topic that use
this new Store and keep existing ones until new one is stable.
Recently introduced "DestinationFactory" can be used to initialize
alternative destination implementations.
Difficult call - I'm trying not to change the existing code too much
- as I know there's been alot of bug fixes in the
PrefetchSubscription/DurableTopicSubscriber etc. code, for timing
issues, transactions etc - I'm tempted not to loose that if possible :)
-----Original Message-----
From: Rob Davies [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 05, 2006 11:28 AM
To: [email protected]
Subject: Re: Paging support
There will be some changes to the message store api for the following:
getNextMessage(String clientId, String subsName) - next message
that hasn't been acknowledged getNextMessage(String clientId,
String subsName, MessageId from,
RecoveryListener) - allow batches of messages to be processed
etc.
In addition the pending LinkedList in Prefetch Subscription has
been replaced with a PendingMessageCursor interface - to allow
different implementations to be plugged in.
So a durable subscriber will not have to be recovered - but can
retrieve messages in batches - taking into account the available
memory in the message store.
There is a FilePendingMessageCursor - which can be used optionally
for the temporary offline storage for non-durable Topic subscribers
On 5 Sep 2006, at 18:35, Fateev, Maxim wrote:
Rob,
Would you give overview of the proposed design for the paging support
in ActiveMQ? The AMQ-845 doesn't provide any specifics.
-----Original Message-----
From: Rob Davies [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 05, 2006 10:22 AM
To: [email protected]
Subject: Re: Paging support
A message reference is kept in memory (which consists of a messageId
and information like sequence Id to enable faster retrieval from
storage).
There is some work currently in progress in enable paging support in
ActiveMQ - see http://issues.apache.org/activemq/browse/AMQ-845
On 5 Sep 2006, at 18:07, pradeep wrote:
Maxim,
Can you clarify this
Are you storing just the message id or the whole message in memory?
Your
help will be highly appreciated.
Thanks
pradeep
Fateev, Maxim wrote:
It would be interesting to see some numbers about memory
utilization
per message in backlog.
IMHO the main problem with keeping references in memory is that it
needs recovery phase on broker startup. It can take a while when
backlog is really big.
-----Original Message-----
From: James Strachan [mailto:[EMAIL PROTECTED]
Sent: Sunday, September 03, 2006 11:51 PM
To: [email protected]
Subject: Re: Paging support
Note that MessageReferences != message contents; they are mostly
just a MessageId which is pretty small.
On 8/29/06, Fateev, Maxim <[EMAIL PROTECTED]> wrote:
AFAIK messages are moved from journal to permantent storage
(JDBC) on
every checkpoint. The problem is that references for every message
are still kept in memory.
IMHO it is inherent limitation of MessageStore API as it is
defined
now.
The only way to get message is by id. And the only way to have id
is to keep it in memory.
I'm looking into changing MessageStore API to one that instead of
Message getMessage(MessageId identity)
Would provide
Message getNextMessage()
method eliminating need to keep references in memory.
-----Original Message-----
From: Anders Bengtsson [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 29, 2006 1:23 AM
To: [email protected]
Subject: Re: Paging support
gnodet wrote:
If you don' t use any persistent store, messages must be kept in
memory, so you are obvisouly limited by the available mem.
Yes, this is what I would expect.
If you use a jdbc store, messages can be removed from memory and
store for later consumption.
Yes, except it doesn't actually work!
It writes things to the store, but it is still very much
limited by
memory for some reason. (Someone reported this as AMQ-845 earlier,
but with no response).
--
View this message in context:
http://www.nabble.com/Paging-support-tf2163517.html#a6034790
Sent from the ActiveMQ - User forum at Nabble.com.
--
James
-------
http://radio.weblogs.com/0112098/
--
View this message in context: http://www.nabble.com/Paging-support-
tf2163517.html#a6156162
Sent from the ActiveMQ - User forum at Nabble.com.
