> On March 25, 2014, 9:20 p.m., Gordon Sim wrote: > > Ultimately the question as to the acceptability of the performance impact > > is something only users can answer. However the penalty would be high as > > the update is synchronous and all bdb transactions will be serialised. > > > > For a per-message operation such as this, a more performant solution would > > be to use the mechanism used for recording enqueues. Perhaps for example a > > special queue could be used onto which messages representing the changes to > > the sequence number could be enqueued (and the message representing the old > > value then dequeued, like with an LVQ). On recovery the last sequence would > > be recovered from the special queue and used to update the exchange. > > > > This is somewhat half-baked, but might be an interesting angle to explore > > some more as a simple(ish) solution. > > Pavel Moravec wrote: > I like the idea about LVQ queue, that would use store much efficiently. I > agree direct bdb access would be terribly slow while async store much more > efficient. > > Is there a way how to hide such queue from an end-user, or is it > acceptable to have it available/visible as a regular queue? (maybe worth > marking it as exclusive to prevent message consumption or deletion by a > mistake) > > Also, it is worth having the queue name configurable via a new broker > option, I guess. > > Until I get some feedback within few days, I will focus on this direction. > > Gordon Sim wrote: > I think if we keep it out of the queue registry and (optionally) don't > set up a management object for it, then it would not be visible externally. I > agree you would want to make sure that it couldn't be accessed by regular > producers/consumers or other operations. > > Gordon Sim wrote: > One issue with hiding the queue may be in recovery - we might need a > special pattern or some special flag on the queue to handle it differently on > recovery and not register it in the registry. > > Jakub Scholz wrote: > Persistent LVQ queues can be quite ugly when some keys get updated > multiple times every second and some are updated only few times a day. With > legacy store you can end up with full queue (because the broker will not be > able to reuse journals containing few messages from exchanges with few > updates) and my guess is that with linear store, you can end up with a very > huge journal just to keep few integers. To get it working reliably, you would > probably need to have one LVQ queue per exchange, so that the queue actually > contains always only one message.
agreed - Gordon ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19566/#review38533 ----------------------------------------------------------- On March 23, 2014, 11:52 a.m., Pavel Moravec wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/19566/ > ----------------------------------------------------------- > > (Updated March 23, 2014, 11:52 a.m.) > > > Review request for qpid, Gordon Sim and Kim van der Riet. > > > Bugs: QPID-5642 > https://issues.apache.org/jira/browse/QPID-5642 > > > Repository: qpid > > > Description > ------- > > Elegant (but not performance optimal) way of patch: > > 1) In Exchange::PreRoute::PreRoute, update exchange in store (whole entry). > 2) The update method is dealed very similarly like > MessageStoreImpl::create(db_ptr db, IdSequence& seq, const > qpid::broker::Persistable& p) method, i.e. calls BufferValue that calls > Exchange::encode. > > Here the code can be unified by merging MessageStoreImpl::update > intoMessageStoreImpl::create method where the code almost duplicates. > > However I do not see the patch as performance efficient, as with every > message preroute, new qpid::framing::Buffer is filled in Exchange::encode > method, data are copied from it to char* BufferValue::data and even then they > are really written to the BDB. While in fact we just update the only one > number in the Buffer. > > I tried to come up with less performance invasive approach (for those > interested, see > https://bugzilla.redhat.com/attachment.cgi?id=877576&action=diff - if you > dont have access there, let me write), that keeps qpid::framing::Buffer for > every durable exchange with sequencing enabled, but it returned (among > others) into the need of changing the way store encodes/decodes Exchange > instance (change in Exchange::encode / decode methods). What would make the > broker backward incompatible. > > Is the performance penalty (due to Exchange::encode method called for every > message preroute) acceptable? > > Is it worth merging MessageStoreImpl::update intoMessageStoreImpl::create > method? > > > Diffs > ----- > > /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStore.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStoreModule.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/MessageStoreModule.cpp 1577475 > /trunk/qpid/cpp/src/qpid/broker/NullMessageStore.h 1577475 > /trunk/qpid/cpp/src/qpid/broker/NullMessageStore.cpp 1577475 > /trunk/qpid/cpp/src/qpid/legacystore/MessageStoreImpl.h 1577475 > /trunk/qpid/cpp/src/qpid/legacystore/MessageStoreImpl.cpp 1577475 > /trunk/qpid/cpp/src/qpid/linearstore/MessageStoreImpl.h 1577475 > /trunk/qpid/cpp/src/qpid/linearstore/MessageStoreImpl.cpp 1577475 > /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.h 1577475 > /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1577475 > > Diff: https://reviews.apache.org/r/19566/diff/ > > > Testing > ------- > > - reproducer from JIRA verified > - automated tests passed (except for those known to fail due to QPID-5641 > (valgrind & legacystore) > > > Thanks, > > Pavel Moravec > >
