Hi Colin, Had a little more time to think about this and I believe that there are potentially two distinct solutions, with pros & cons for each, which possibly meet the needs of different applications/architectures.
Robert Greig & I discussed briefly and don't entirely agree on the pros/cons :-) *Replay with only BDBStore* If your application has performance as its only really significant requirement, then a solution which provided a fairly limited set of operations for BDBStore replaying would be appropriate. So, you might perhaps have a tool that can archive data (using a header property) or replay it based on a property, not using SQL since that's not the BDBStore way. However, it'd be implementation expensive to try and provide anything more feature rich. *Replay with BDBStore and an RDBMS* So, if you assume that for many users currently replaying data they're probably using an RDBMS to do it (store/replay/archive/report etc), then they might find the BDBStore approach fast but not as sophisticated as their existing architecture. In that case, it might be nice to be able to setup BDBStore to archive to an RDBMS and be able to replay from thet RDBMS (mySQL or whatever). The users can then migrate over any SQL scripts they use to select data from replay/check the state of their archive/run reports etc - at the cost of a less performant recovery model. Could batch up the archive to RDBMS from BDBStore to make it more efficient/more robust if RDBMS becomes unavailable. Just some thoughts - hth. Marnie On 1/18/07, Colin Crist <[EMAIL PROTECTED]> wrote:
Typo. ^On the consumer side, WebMethods (!) for example in their^On the producer side, WebMethods (!) for example in their^ Sry. Colin. > -----Original Message----- > From: Colin Crist [mailto:[EMAIL PROTECTED] > Sent: 18 January 2007 23:16 > To: '[email protected]' > Subject: RE: Message history and replay > > > > > > As a user, I would also like to control when messages get > > removed as > > > part of my end of day or even weekly processs, probably based on > > > header property selectors. > > > > Yes, very interesting and useful use case. It would improve > throughput > > for many apps since people would not have to commit the > whole message > > to their database before sending it to the message broker. > > On the consumer side, WebMethods (!) for example in their > core messaging platform can let you find out the last message > a client published when it reconnects. I didn't use it when I > was in WebMethods hell as our systems did not require such > behaviour but I found it interesting and could see scenarios > when it may help minimise downsteam duplicates and shorten > producer recovery time. > > I don't think this works for AMQP as it does not maintain the > exchange route history - or am I wrong? > > Personally, I can live without it and its best to keep > recovery downstream IMO. It's the consumer end that has to > deal with duplicates after all but I've found many systems > that on recovery will resend too much - all trades from start > of day being a common example that can give unexpected load > on the middleware and general network buzz. > > The lesson I found is that some producers are very simple and > when you outsource or buy in products you've just got to live > with them! > > > > I don't believe this is supported in AMQP in the wire > protocol (nor > > > perhaps should it be) - is it maybe something that could be > > done via > > > some standardised command messages to a "replay" exchange? > > > > > Any thoughts on implementation difficulty? > > > > In terms of our underlying architecture, this would be relatively > > easy. We have a delivery table that simply maps from message id to > > queue, and the message is stored separately. Messages are removed > > automatically when a ref count hits zero. The removal process is > > actually handled in the store module so it could do something else > > easily (e.g if you had a custom store). > > > The admin API could be enhanced to support re-enqueuing of > message ids > > or hard delete of messages. > > I'd see this kind of behaviour to be message based and > perhaps standardised in some addendum to the AMQP > specification that defines optional exchanges and payload > formats. Of course it makes sense to map it to an API but > outside of the core specification to keep it decoupled as > much as possible and reduce dependencies and so on. If I were > to have a client in a language not in the Qpid toolkit, it > would be good to have the messages and interaction well > defined so another implementation could leverage and help > drive it into a standard. > > Its hardly new and there are always things to learn from the > old dogs of middleware (with most of the current market share...) > > http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg2400066 > 8&loc=en_US&cs=utf-8&lang=en > > The same argument goes for configuration, management and > monitoring functionality too :) I'd like to see a market for > 3rd party tooling that works with all AMQP based products. A > whole other conversation when you and the spec are hitting 1.0... > > Hope this is not too far leftfield for this stage of the game > and you don't mind me throwing in my opinions... > > Regards, > > Colin. >
