> On March 25, 2014, 9:20 p.m., Gordon Sim wrote: > > Ultimately the question as to the acceptability of the performance impact > > is something only users can answer. However the penalty would be high as > > the update is synchronous and all bdb transactions will be serialised. > > > > For a per-message operation such as this, a more performant solution would > > be to use the mechanism used for recording enqueues. Perhaps for example a > > special queue could be used onto which messages representing the changes to > > the sequence number could be enqueued (and the message representing the old > > value then dequeued, like with an LVQ). On recovery the last sequence would > > be recovered from the special queue and used to update the exchange. > > > > This is somewhat half-baked, but might be an interesting angle to explore > > some more as a simple(ish) solution. > > Pavel Moravec wrote: > I like the idea about LVQ queue, that would use store much efficiently. I > agree direct bdb access would be terribly slow while async store much more > efficient. > > Is there a way how to hide such queue from an end-user, or is it > acceptable to have it available/visible as a regular queue? (maybe worth > marking it as exclusive to prevent message consumption or deletion by a > mistake) > > Also, it is worth having the queue name configurable via a new broker > option, I guess. > > Until I get some feedback within few days, I will focus on this direction. > > Gordon Sim wrote: > I think if we keep it out of the queue registry and (optionally) don't > set up a management object for it, then it would not be visible externally. I > agree you would want to make sure that it couldn't be accessed by regular > producers/consumers or other operations. > > Gordon Sim wrote: > One issue with hiding the queue may be in recovery - we might need a > special pattern or some special flag on the queue to handle it differently on > recovery and not register it in the registry.
Persistent LVQ queues can be quite ugly when some keys get updated multiple times every second and some are updated only few times a day. With legacy store you can end up with full queue (because the broker will not be able to reuse journals containing few messages from exchanges with few updates) and my guess is that with linear store, you can end up with a very huge journal just to keep few integers. To get it working reliably, you would probably need to have one LVQ queue per exchange, so that the queue actually contains always only one message. - Jakub ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19566/#review38533 ----------------------------------------------------------- On March 23, 2014, 11:52 a.m., Pavel Moravec wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/19566/ > ----------------------------------------------------------- > > (Updated March 23, 2014, 11:52 a.m.) > > > Review request for qpid, Gordon Sim and Kim van der Riet. > > > Bugs: QPID-5642 > https://issues.apache.org/jira/browse/QPID-5642 > > > Repository: qpid > > > Description > ------- > > Elegant (but not performance optimal) way of patch: > > 1) In Exchange::PreRoute::PreRoute, update exchange in store (whole entry). > 2) The update method is dealed very similarly like > MessageStoreImpl::create(db_ptr db, IdSequence& seq, const > qpid::broker::Persistable& p) method, i.e. calls BufferValue that calls > Exchange::encode. > > Here the code can be unified by merging MessageStoreImpl::update > intoMessageStoreImpl::create method where the code almost duplicates. > > However I do not see the patch as performance efficient, as with every > message preroute, new qpid::framing::Buffer is filled in Exchange::encode > method, data are copied from it to char* BufferValue::data and even then they > are really written to the BDB. While in fact we just update the only one > number in the Buffer. > > I tried to come up with less performance invasive approach (for those > interested, see > https://bugzilla.redhat.com/attachment.cgi?id=877576&action=diff - if you > dont have access there, let me write), that keeps qpid::framing::Buffer for > every durable exchange with sequencing enabled, but it returned (among > others) into the need of changing the way store encodes/decodes Exchange > instance (change in Exchange::encode / decode methods). What would make the > broker backward incompatible. > > Is the performance penalty (due to Exchange::encode method called for every > message preroute) acceptable? > > Is it worth merging MessageStoreImpl::update intoMessageStoreImpl::create > method? > > > Diffs > ----- > > /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStore.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStoreModule.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStoreModule.cpp 1577475 > /trunk/qpid/cpp/src/qpid/broker/NullMessageStore.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/NullMessageStore.cpp 1577475 > /trunk/qpid/cpp/src/qpid/legacystore/MessageStoreImpl.h 1577475 > /trunk/qpid/cpp/src/qpid/legacystore/MessageStoreImpl.cpp 1577475 > /trunk/qpid/cpp/src/qpid/linearstore/MessageStoreImpl.h 1577475 > /trunk/qpid/cpp/src/qpid/linearstore/MessageStoreImpl.cpp 1577475 > /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.h 1577475 > /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1577475 > > Diff: https://reviews.apache.org/r/19566/diff/ > > > Testing > ------- > > - reproducer from JIRA verified > - automated tests passed (except for those known to fail due to QPID-5641 > (valgrind & legacystore) > > > Thanks, > > Pavel Moravec > >
