Hello Rob, Thanks for the answer. I started looking at the code to see if there is something I can do about these 2 commits. But before going any further I'd like your input on the below, to see if what I'm trying to do could work or if I'm missing something (which I'm surely are) https://github.com/apache/qpid-broker-j/compare/master...overmeulen:feature/jdbc-message-store-commits
Regards, Olivier -----Original Message----- From: Rob Godfrey <rob.j.godf...@gmail.com> Sent: vendredi 14 septembre 2018 16:06 To: users@qpid.apache.org Cc: AYOUBI Ali <ali.ayo...@murex.com> Subject: Re: [Broker-J] JDBC message store performance On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <olivier.vermeu...@murex.com> wrote: > Hello, > > We ran a performance test with a bunch of brokers and an Oracle > database to store the messages. > We noticed that the database was a bit overloaded with commits. > Looking at the logs we saw that sending a message was triggering 1 > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA, > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving a > message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1 for > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT). > I debugged a bit the code and saw that in > AbstractVirtualHost.executeTransaction the delete on > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a "post > commit" operation, explaining why we have 2 commits. > Is it something expected? Do you think we could reduce this to 1 > commit when receiving a message? > It's not unexpected - basically the issue is that the broker needs to cope with the possibility that the same message is being stored on multiple queues. The first commit is deleting the referencing of the message from the given queue. The second commit is occuring after the message has been definitively removed from the queue, and the store has determined that there are no more references, so it is ok to remove the message. This is driven by reference-counting of the message, and has historically been a place of many potential race conditions. I'm sure it is possible to optimise the code in some way, but it may not be "easy". For the BDB store this doesn't matter as much as the actual synchronisation to disk of these operations is coalesced, obviously this is more of an issue for the JDBC store. -- Rob > > Thanks, > Olivier > ******************************* > > This e-mail contains information for the intended recipient only. It > may contain proprietary material or confidential information. If you > are not the intended recipient you are not authorised to distribute, > copy or use this e-mail or any attachment to it. Murex cannot > guarantee that it is virus free and accepts no responsibility for any > loss or damage arising from its use. If you have received this e-mail > in error please notify immediately the sender and delete the original > email received, any attachments and all copies from your system. > ******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
