So - the more I think about this, the more I think a custom store might be the best way to approach this. For safety I think the approach I would take is that on asking for the message to be removed from the DB, I would write the blob id into a separate table (in the same transaction as the delete). Then we can have a separate thread that is iterating over the table of "blobs to be deleted", and for every row it finds it can call the blob store to delete, and only remove the row from the "blobs to be deleted" table when it gets confirmation that the blob has, indeed, been removed from the external store. In this way we can separate the deletion of the message data from interaction with the external blob store (so, e.g. if the blob store is unreachable it doesn't delay anything else), and on restart the store can just start deleting the blobs by id of anything that is left in the table.
-- Rob On Wed, 11 Mar 2020 at 10:33, VERMEULEN Olivier <olivier.vermeu...@murex.com> wrote: > from many queues on the same broker > ________________________________ > From: Rob Godfrey <rob.j.godf...@gmail.com> > Sent: Wednesday, March 11, 2020 10:01:43 AM > To: users@qpid.apache.org <users@qpid.apache.org> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener > > On Wed, 11 Mar 2020 at 09:46, VERMEULEN Olivier < > olivier.vermeu...@murex.com> > wrote: > > > Why don't we just trigger the existing MessageDeleteListeners at the > > beginning of the remove method instead of doing it at the end ? > > > > > https://github.com/apache/qpid-broker-j/blob/master/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670 > > > > > https://github.com/apache/qpid-broker-j/blob/master/bdbstore/src/main/java/org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java#L1199 > > > > This way we handle both the normal use case and the recovery one and the > > headers will still be accessible from the StoredMessage instance (need to > > add a getter in StorableMessageMetaData). > > I can provide a pull-request if you want. > > > > I still don't think this will work well for getting the header data - you > really need to be operating on the AbstractServerMessage not the > StoredMessage if you want to access the (protocol dependent implementation) > headers / content. > > I think the current approach (and others that I have briefly thought about) > will also fall down in the case of "orphaned" content bodies that are being > deleted on broker restart (basically because the delete of the content - > deliberately - not in a transaction with the delete of the per-queue > enqueue records, an abrupt shutdown of the broker can leave message data in > the store which is no longer referenced by any queues. In this case the > recovery code identifies the orphans and deletes them - but this is a > database level operation rather than on the message model - and you can't > assume the headers will be available there either (they are stored in a > separate table). > > In a previous mail, you talked about "multicast" - is the expectation that > the underlying blob may be being referenced from many queues on the same > broker? or on queues on many different brokers? > > -- Rob > > > > > > WDYT? > > Olivier > > > > From: VERMEULEN Olivier <olivier.vermeu...@murex.com> > > Sent: mardi 10 mars 2020 20:08 > > To: users@qpid.apache.org > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener > > > > The idea is to avoid any performance impact on the messaging when big > > messages arrive. > > So whenever a producer tries to send a big message (more than 10KB for > us) > > we put the payload in the blob store with a unique ID and we pass this ID > > in the message instead of the original payload. > > When a consumer receives such a message we then retrieve the payload from > > the blob store. > > The problem is that we can't expect the consumer to handle the cleanup of > > the blob store since it might not be the only consumer receiving this > > message (multicast for example or even a message being routed to 2 queues > > through a topic). > > The only one that really knows when the payload can be removed from the > > blob store is the broker. > > That's why I was looking for a hook that would be called whenever a > > message is or will be deleted from the store (this includes the recovery > > process). > > After that whether the hook is at the queue or virtual host level doesn't > > really matter... > > > > Olivier > > ________________________________ > > From: Rob Godfrey <rob.j.godf...@gmail.com<mailto: > rob.j.godf...@gmail.com > > >> > > Sent: Tuesday, March 10, 2020 6:26:39 PM > > To: users@qpid.apache.org<mailto:users@qpid.apache.org> < > > users@qpid.apache.org<mailto:users@qpid.apache.org>> > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener > > > > Yeah - I was wondering if it might be better to add hooks into the point > at > > which remove() is called (i.e. > > > > > https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127 > > ), > > because a listener on the here could actually work with the message > object > > where the headers are available... However this would miss the other > point > > at which messages are removed from the store - in the store recovery > > process (e.g. > > > > > https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227 > > ). > > Unfortunately at this point we wouldn't have access to the headers. > > > > Can you explain a bit more how your blob storage is supposed to work. > > Perhaps there is another way that we can add a hook that would help here > > (e.g. on a per queue basis have some sort of "dequeue" trigger which is > > passed the message it is about to be permanently removed from the queue). > > > > -- Rob > > > > On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier < > > olivier.vermeu...@murex.com<mailto:olivier.vermeu...@murex.com>> > > wrote: > > > > > The cast works fine in my case but the problem is that everything is > > NULL, > > > including the meta-data. > > > Indeed the message is cleaned before the listeners are called... > > > > > > > > > https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670 > > > > > > Olivier > > > > > > -----Original Message----- > > > From: Rob Godfrey <rob.j.godf...@gmail.com<mailto: > > rob.j.godf...@gmail.com>> > > > Sent: lundi 9 mars 2020 09:58 > > > To: users@qpid.apache.org<mailto:users@qpid.apache.org> > > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener > > > > > > On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier < > > > olivier.vermeu...@murex.com<mailto:olivier.vermeu...@murex.com>> > > > wrote: > > > > > > > Hello Rob, > > > > > > > > "Are you looking to have Broker-J trigger the deletion on the blob at > > > > the point at which it believes it no longer needs the message?" > > > > Yes that's exactly what I'm trying to do. > > > > Note that I managed to do it by creating my own vhost, just wanted to > > > > check if there was a less intrusive way of doing it... > > > > I agree that we shouldn't be able to modify the message but today I > > > > can't even read it easily. > > > > What I'm interested in are the headers but from a StoredMessage > > > > instance I have to go through internal classes and package-protected > > > > classes to get > > > > them: > > > > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader(); > > > > getHeader being package-protected here... > > > > > > > > > > Note that this will only work if the message is an "Internal" message > > (i.e. > > > the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 > or > > > AMQP 1.0 message sent over the wire). The store API is basically built > > to > > > be as unaware of the form of the message as possible, and the api here > is > > > basically designed to allow the store to clean up a message which was > > > potentially stored in multiple queues, but which has now been removed > > from > > > all of them. As such it's not really designed to make it easy for you > to > > > access the headers (or content - since the "content" will include > > structure > > > for 1.0 messages). > > > > > > > > > > Did I miss something? > > > > > > > > > > No - I think to do this safely we'd have to expose some new API here. > > > > > > -- Rob > > > > > > > > > > > > > > Thanks, > > > > Olivier > > > > > > > > -----Original Message----- > > > > From: Rob Godfrey <rob.j.godf...@gmail.com<mailto: > > rob.j.godf...@gmail.com>> > > > > Sent: vendredi 6 mars 2020 17:54 > > > > To: users@qpid.apache.org<mailto:users@qpid.apache.org> > > > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener > > > > > > > > Hi Olivier > > > > > > > > I'm not entirely sure what your exact idea here is - are you looking > > > > to have Broker-J trigger the deletion on the blob at the point at > > > > which it believes it no longer needs the message? Or maybe to update > > > > the blob store to make clear that the broker no longer needs to > retain > > > > a copy of the message? > > > > > > > > Clearly the API is not designed for allowing extensions here - and I > > > > think if we were to try to make it an extension / interception point > > > > then we would have to be careful in limiting what could be done here > > > > (e.g. offering only some read-only view of the message). > > > > > > > > Without such an interception extension mechanism, you are correct I > > > > think that you would need to create a new vhost type (potentially > > > > using a new store type of your own implementation) > > > > > > > > -- Rob > > > > > > > > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier < > > > > olivier.vermeu...@murex.com<mailto:olivier.vermeu...@murex.com>> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > We're trying to implement the claim-check pattern on top of our > > > > > broker > > > > > + dispatch-router topology. > > > > > So we have a blob store and the idea is for each message above 10KB > > > > > to store the payload in it and to only keep an ID in the message. > > > > > The tricky part is the delete of the payload in the blob store, > > > > > especially since some of our use cases are using multicast... > > > > > When looking at the Broker-J source code I found this > > > > > MessageDeleteListener which would be really helpful for this use > > case: > > > > > > > > > > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a > > > > > e7 > > > > > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto > > > > > re > > > > > /MessageStore.java#L84 > > > > > > > > > > My first question would be, is it ok to use it? > > > > > Second, so far the only way I found to register a new listener is > to > > > > > define my own VirtualHost, is there an easier way ? > > > > > > > > > > Thanks, > > > > > Olivier > > > > > ******************************* > > > > > This e-mail contains information for the intended recipient only. > It > > > > > may contain proprietary material or confidential information. If > you > > > > > are not the intended recipient you are not authorized to > distribute, > > > > > copy or use this e-mail or any attachment to it. Murex cannot > > > > > guarantee that it is virus free and accepts no responsibility for > > > > > any loss or damage arising from its use. If you have received this > > > > > e-mail in error please notify immediately the sender and delete the > > > > > original email received, any attachments and all copies from your > > > system. > > > > > > > > > ******************************* > > > > This e-mail contains information for the intended recipient only. It > > > > may contain proprietary material or confidential information. If you > > > > are not the intended recipient you are not authorized to distribute, > > > > copy or use this e-mail or any attachment to it. Murex cannot > > > > guarantee that it is virus free and accepts no responsibility for any > > > > loss or damage arising from its use. If you have received this e-mail > > > > in error please notify immediately the sender and delete the original > > > > email received, any attachments and all copies from your system. > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org<mailto: > > users-unsubscr...@qpid.apache.org> For > > > > additional commands, e-mail: users-h...@qpid.apache.org<mailto: > > users-h...@qpid.apache.org> > > > > > > > > > > > ******************************* > > > This e-mail contains information for the intended recipient only. It > may > > > contain proprietary material or confidential information. If you are > not > > > the intended recipient you are not authorized to distribute, copy or > use > > > this e-mail or any attachment to it. Murex cannot guarantee that it is > > > virus free and accepts no responsibility for any loss or damage arising > > > from its use. If you have received this e-mail in error please notify > > > immediately the sender and delete the original email received, any > > > attachments and all copies from your system. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org<mailto: > > users-unsubscr...@qpid.apache.org> > > > For additional commands, e-mail: users-h...@qpid.apache.org<mailto: > > users-h...@qpid.apache.org> > > > > > > > > ******************************* > > This e-mail contains information for the intended recipient only. It may > > contain proprietary material or confidential information. If you are not > > the intended recipient you are not authorized to distribute, copy or use > > this e-mail or any attachment to it. Murex cannot guarantee that it is > > virus free and accepts no responsibility for any loss or damage arising > > from its use. If you have received this e-mail in error please notify > > immediately the sender and delete the original email received, any > > attachments and all copies from your system. > > > ******************************* > This e-mail contains information for the intended recipient only. It may > contain proprietary material or confidential information. If you are not > the intended recipient you are not authorized to distribute, copy or use > this e-mail or any attachment to it. Murex cannot guarantee that it is > virus free and accepts no responsibility for any loss or damage arising > from its use. If you have received this e-mail in error please notify > immediately the sender and delete the original email received, any > attachments and all copies from your system. >
