Sure, I'll put some notes together. I don't seem to have permissions to add any pages - can you give me access or alternatively I can put the pages on my own confluence and get someone to move them later.
Colin. > -----Original Message----- > From: Carl Trieloff [mailto:[EMAIL PROTECTED] > Sent: 24 January 2007 18:07 > To: [email protected] > Subject: Re: Message history and replay > > > Colin, > > Would you be willing to create a page on the wiki for this > cwiki.apache.org/qpid and lay out all the use cases there so > we don't loose them. then we can add designs etc there also > as we explore this topic. > > Carl. > > > Colin Crist wrote: > > I believe the customary apache response here is: > > > > +1 > > > > Colin. > > > > > >> -----Original Message----- > >> From: John O'Hara [mailto:[EMAIL PROTECTED] > >> Sent: 23 January 2007 16:56 > >> To: [email protected] > >> Subject: Re: Message history and replay > >> > >> Two options: > >> 1) Make the 'rewind' accessible to the client on connect > >> 2) Make it an administrative function (reset next message to X) > >> > >> Both are valid. > >> Also retentions of several days should be valid subject to > resource > >> constraints. > >> > >> > >> Also Queues should have attributes related to this: > >> a) Archive Retention Period *0*, x seconds, y messages > >> b) Rewindable Y / *N* > >> c) Client Can Rewind Y / *N* > >> d) Archive treatment - *Delete* or Copy-out > >> e) Archive interval - *30s* (queue garbage collect period) > >> f) Archive TTL-deleted messages - Y / *N* > >> > >> Defaults in *. > >> > >> Queues are smart resources. We need to spec out what they can do > >> (and what > >> *subset* of that we propose to standardise over on the AMQP WG). > >> Here's a great place to put value added features. > >> > >> John > >> > >> On 23/01/07, Colin Crist <[EMAIL PROTECTED]> wrote: > >> > >>> I had thought a replay queue is where the messages would be > >>> > >> put I'm no > >> > >>> longer convinced. > >>> > >>> In another mail John O'Hara uses the the reading email analogy. > >>> Continuing this to its logical conclusion could mean that > replay is > >>> actually an option when you open the queue for reading. It > >>> > >> avoids the > >> > >>> client having to muck about with another queue. > >>> > >>> This is not the traditional view of a queue but is more > >>> > >> general and I > >> > >>> like it. > >>> > >>> So, how about a queue contains all messages put on it > since it was > >>> last "purged". When a queue is opened for reading, it can > be opened > >>> with a selector and one part of that selector is whether > or not the > >>> message has been commited by a consumer or not. > >>> > >>> A consumer could then always open the queue and request all > >>> > >> messages, > >> > >>> committed or not, from a known point, e.g. a trade ID or > >>> > >> message ID. > >> > >>> The key thing here I suppose is that this type of consumer > >>> > >> is writing > >> > >>> the message in some other form into a database and so > >>> > >> always knows the > >> > >>> last thing written. > >>> A nice benefit is that if the database is really hosed you > >>> > >> could get > >> > >>> replay from start of day. This would definitely help > >>> > >> implement DR for > >> > >>> purely message based systems. > >>> > >>> So, can I specialise queues and layer this functionality on > >>> > >> top of the > >> > >>> spec in the APID implementation with some kind of > provider specific > >>> header property? > >>> > >>> Colin. > >>> > >>> > >>>> -----Original Message----- > >>>> From: Marnie McCormack [mailto:[EMAIL PROTECTED] > >>>> Sent: 22 January 2007 19:54 > >>>> To: [email protected]; [EMAIL PROTECTED] > >>>> Subject: Re: Message history and replay > >>>> > >>>> I've been trying to imagine the scenarios in which a user > >>>> > >> will make > >> > >>>> use of a ReplayExchange. > >>>> > >>>> I thought that the most common usage would be the following: > >>>> > >>>> - the receiving application has lost some messages sent > to it and > >>>> needs them to be resent from a point in time or a message ID > >>>> > >>>> (The only other recovery scenario I thought likely was broker > >>>> failure, which would be handled by restarted (with > >>>> persistence) and the undelivered messages would already > remain in > >>>> the BDB store.) > >>>> > >>>> So, the bit I'm not sure about is this - you have no way > >>>> > >> of knowing > >> > >>>> in advance the point at which replay might be needed > from (unless > >>>> using client acks - which work with ordinary queues ?). > >>>> > >>>> Thus your recipient app starts up again (or recognises it > >>>> > >> has lost > >> > >>>> messages) and then wants to replay. So, the queue it is > >>>> > >> reading from > >> > >>>> is rebound to a replay exchange (with a marker > >>>> ID) ? Does that mean that it is actually reading from a > different > >>>> queue than normal operation, or that the queue is bound to a > >>>> different exchange type ? > >>>> > >>>> The logic behind this could be quite complex - in that > >>>> > >> the balance > >> > >>>> between normal operation and replay operation in subtle > and would > >>>> involve the receiving client understanding the AMQP behaviour at > >>>> quite a low level ? > >>>> > >>>> Apologies if I've missed a simple flow for replay :-) > >>>> > >>>> Regards, > >>>> Marnie > >>>> > >>>> On 1/22/07, Carl Trieloff <[EMAIL PROTECTED]> wrote: > >>>> > >>>>> This has been asked for before. I would interested to see if an > >>>>> exchange could be used. I also think this might bring up > >>>>> > >>>> the topic of > >>>> > >>>>> chained exchanges again. > >>>>> > >>>>> Colin, do you have any interest in creating a prototype > >>>>> > >>>> ReplyExchange > >>>> > >>>>> to see how well it works? > >>>>> Carl. > >>>>> > >>>>> Alan Conway wrote: > >>>>> > >>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote: > >>>>>> > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> A quick question... > >>>>>>> > >>>>>>> A common recovery scenario is requesting replay on a > >>>>>>> > >> queue from > >> > >>>>>>> a > >>>>>>> > >>>>> given, > >>>>> > >>>>>>> known message, either by its ID or by something in the > >>>>>>> > >>>> header. Its > >>>> > >>>>> saves all > >>>>> > >>>>>>> that mucking about with XA and makes recovery the > >>>>>>> > >> normal way of > >> > >>>>>>> system startup rather then some exceptional case. > >>>>>>> > >>>>>>> This means when a message has reached all its recipients > >>>>>>> > >>>> it should > >>>> > >>>>> never be > >>>>> > >>>>>>> marked for removal from the durable message store. > >>>>>>> > >>>>>>> As a user, I would also like to control when messages > >>>>>>> > >>>> get removed > >>>> > >>>>>>> as > >>>>>>> > >>>>> part of > >>>>> > >>>>>>> my end of day or even weekly processs, probably > >>>>>>> > >> based on header > >> > >>>>> property > >>>>> > >>>>>>> selectors. > >>>>>>> > >>>>>>> I don't believe this is supported in AMQP in the wire > >>>>>>> > >>>> protocol (nor > >>>> > >>>>> perhaps > >>>>> > >>>>>>> should it be) - is it maybe something that could be done > >>>>>>> > >>>> via some > >>>> > >>>>>>> standardised command messages to a "replay" exchange? > >>>>>>> > >>>>>>> Any thoughts on implementation difficulty? > >>>>>>> > >>>>>>> > >>>>>> You could represent this in AMQP protocol using a custom > >>>>>> > >>>> exchange type. > >>>> > >>>>>> A ReplayExchange stores all messages sent to it (until > >>>>>> > >>>> cleaned up by > >>>> > >>>>>> some administrative function.) > >>>>>> > >>>>>> When binding a queue to a replay exchange you MAY specify an > >>>>>> argument "replay-from=N". A queue so bound will be > >>>>>> > >>>> "pre-filled" with > >>>> > >>>>>> the range of message starting from the Nth message up > to the most > >>>>>> recent message sent to the exchange. Thereafter the > >>>>>> > >>>> exchange behaves > >>>> > >>>>>> like a fan-out exchange, each new message being added to > >>>>>> > >>>> all the bound queues. > >>>> > >>>>>> That gives you replay entirely within the standard > >>>>>> > >>>> protocol, using a > >>>> > >>>>>> standard extension point to add this as proprietary broker > >>>>>> functionality. > >>>>>> > >>>>>> Note that to combine replay with other exchange types > >>>>>> > >> requires > >> > >>>>>> exchange chaining which has been discussed several times > >>>>>> > >>>> but is not > >>>> > >>>>>> yet an agreed part of the protocol - this may be > another use case > >>>>>> for that discussion > >>>>>> > >>>>>> Cheers, > >>>>>> Alan. > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>> > >>> > > > > > > > >
