Re: survey: c++ broker and queue depth statistics
On 09/27/2013 11:47 AM, Gordon Sim wrote: The c++ broker reports a queue depth in terms of total bytes, as well as the number of messages. For 0-10 the bytes statistic is calculated by aggregating only the content size (i.e. the size of the body segment). For 1.0 it is the whole message including properties etc (i.e. the payload of the transfer 'performative'). So the size will be different depending on the protocol used in sending it and this difference can be quite marked. E.g. in an extreme case where there are many headers but no content, the bytes reported if sent over 0-10 would be 0 whereas if sent over 1.0 could easily be several hundred bytes. The question is what to do about this. The options are (a) accept that they are inconsistent between versions, (b) modify the 1.0 path to only record the application-data or (c) modify the 0-10 path to include the size of the header segment. While (c) seems to me to be logical the most 'correct', it would be a difference in behaviour. It would mean for example that any queue limits would be hit earlier. One could argue that would be an improvement, but it may cause issues for systems when upgrading. The purpose of this mail is to solicit some feedback from users as to which of these options (or indeed other options that have not occurred to me) would be preferable. Option (c) appears to be the preferred solution. I've put a patch for that up for review for anyone interested: https://reviews.apache.org/r/14427/ - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
I vote c. It is an improvement in behaviour. On 09/27/2013 06:47 AM, Gordon Sim wrote: The c++ broker reports a queue depth in terms of total bytes, as well as the number of messages. For 0-10 the bytes statistic is calculated by aggregating only the content size (i.e. the size of the body segment). For 1.0 it is the whole message including properties etc (i.e. the payload of the transfer 'performative'). So the size will be different depending on the protocol used in sending it and this difference can be quite marked. E.g. in an extreme case where there are many headers but no content, the bytes reported if sent over 0-10 would be 0 whereas if sent over 1.0 could easily be several hundred bytes. The question is what to do about this. The options are (a) accept that they are inconsistent between versions, (b) modify the 1.0 path to only record the application-data or (c) modify the 0-10 path to include the size of the header segment. While (c) seems to me to be logical the most 'correct', it would be a difference in behaviour. It would mean for example that any queue limits would be hit earlier. One could argue that would be an improvement, but it may cause issues for systems when upgrading. The purpose of this mail is to solicit some feedback from users as to which of these options (or indeed other options that have not occurred to me) would be preferable. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
I would vote for (c) option. Gordon Sim wrote > The c++ broker reports a queue depth in terms of total bytes, as well as > the number of messages. > > For 0-10 the bytes statistic is calculated by aggregating only the > content size (i.e. the size of the body segment). For 1.0 it is the > whole message including properties etc (i.e. the payload of the transfer > 'performative'). > > So the size will be different depending on the protocol used in sending > it and this difference can be quite marked. E.g. in an extreme case > where there are many headers but no content, the bytes reported if sent > over 0-10 would be 0 whereas if sent over 1.0 could easily be several > hundred bytes. > > The question is what to do about this. The options are (a) accept that > they are inconsistent between versions, (b) modify the 1.0 path to only > record the application-data or (c) modify the 0-10 path to include the > size of the header segment. > > While (c) seems to me to be logical the most 'correct', it would be a > difference in behaviour. It would mean for example that any queue limits > would be hit earlier. One could argue that would be an improvement, but > it may cause issues for systems when upgrading. > > The purpose of this mail is to solicit some feedback from users as to > which of these options (or indeed other options that have not occurred > to me) would be preferable. > > --Gordon. > > - > To unsubscribe, e-mail: > users-unsubscribe@.apache > For additional commands, e-mail: > users-help@.apache -- View this message in context: http://qpid.2158936.n2.nabble.com/survey-c-broker-and-queue-depth-statistics-tp7598657p7598792.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
+1 Ted Ross wrote > I agree with Pavel and Jakub. Tying the "bytes" depth as closely as > possible to actual memory consumption is probably a good thing. We > could even consider counting the message overhead (the memory used that > is not part of the actual message). > > -Ted > > On 09/27/2013 08:01 AM, Pavel Moravec wrote: >> I would vote for (c) option. >> >> Negs: >> - "backward incompatible" change >> >> Pros: >> - the incompatibility requires just a change in one type of number during >> one upgrade >> - calculation of message size should be unified - assume a queue >> enqueuing messages from 1.0 and 0-10 client >> - limiting queue in "size" units is IMHO useful to limit memory >> consumption of the queue - where counting message headers in the limit >> makes more sense >> >> Kind regards, >> Pavel >> >> >> >> - Original Message - >>> From: "Gordon Sim" < > gsim@ > > >>> To: > users@.apache >>> Sent: Friday, September 27, 2013 1:04:17 PM >>> Subject: Re: survey: c++ broker and queue depth statistics >>> >>> On 09/27/2013 11:47 AM, Gordon Sim wrote: >>>> The c++ broker reports a queue depth in terms of total bytes, as well >>>> as >>>> the number of messages. >>>> >>>> For 0-10 the bytes statistic is calculated by aggregating only the >>>> content size (i.e. the size of the body segment). For 1.0 it is the >>>> whole message including properties etc (i.e. the payload of the >>>> transfer >>>> 'performative'). >>>> >>>> So the size will be different depending on the protocol used in sending >>>> it and this difference can be quite marked. E.g. in an extreme case >>>> where there are many headers but no content, the bytes reported if sent >>>> over 0-10 would be 0 whereas if sent over 1.0 could easily be several >>>> hundred bytes. >>>> >>>> The question is what to do about this. The options are (a) accept that >>>> they are inconsistent between versions, (b) modify the 1.0 path to only >>>> record the application-data or (c) modify the 0-10 path to include the >>>> size of the header segment. >>> I should point out that neither (b) or (c) give identical sizes for the >>> same application data due to minor differences in the two type systems. >>> However the difference would be fairly small and relatively fixed >>> whereas for (a) it can vary hugely depending on the exact message. >>> >>> An option (d) would be to try and come up with a scheme where the size >>> would be always identical. I'm not hugely keen on this as it would add >>> overhead (e.g. calculating the size of a set of 0-10 headers if encoded >>> in the 1.0 type scheme). >>> >>>> While (c) seems to me to be logical the most 'correct', it would be a >>>> difference in behaviour. It would mean for example that any queue >>>> limits >>>> would be hit earlier. One could argue that would be an improvement, but >>>> it may cause issues for systems when upgrading. >>>> >>>> The purpose of this mail is to solicit some feedback from users as to >>>> which of these options (or indeed other options that have not occurred >>>> to me) would be preferable. >>>> >>>> --Gordon. >>>> >>>> - >>>> To unsubscribe, e-mail: > users-unsubscribe@.apache >>>> For additional commands, e-mail: > users-help@.apache >>>> >>> >>> - >>> To unsubscribe, e-mail: > users-unsubscribe@.apache >>> For additional commands, e-mail: > users-help@.apache >>> >>> >> - >> To unsubscribe, e-mail: > users-unsubscribe@.apache >> For additional commands, e-mail: > users-help@.apache >> > > > - > To unsubscribe, e-mail: > users-unsubscribe@.apache > For additional commands, e-mail: > users-help@.apache -- View this message in context: http://qpid.2158936.n2.nabble.com/survey-c-broker-and-queue-depth-statistics-tp7598657p7598791.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: survey: c++ broker and queue depth statistics
c. > -Original Message- > From: Gordon Sim [mailto:g...@redhat.com] > Sent: Friday, September 27, 2013 6:47 AM > To: users@qpid.apache.org > Subject: survey: c++ broker and queue depth statistics > > The c++ broker reports a queue depth in terms of total bytes, as well as the > number of messages. > > For 0-10 the bytes statistic is calculated by aggregating only the content > size > (i.e. the size of the body segment). For 1.0 it is the whole message including > properties etc (i.e. the payload of the transfer 'performative'). > > So the size will be different depending on the protocol used in sending it and > this difference can be quite marked. E.g. in an extreme case where there are > many headers but no content, the bytes reported if sent over 0-10 would be > 0 whereas if sent over 1.0 could easily be several hundred bytes. > > The question is what to do about this. The options are (a) accept that they > are inconsistent between versions, (b) modify the 1.0 path to only record the > application-data or (c) modify the 0-10 path to include the size of the header > segment. > > While (c) seems to me to be logical the most 'correct', it would be a > difference in behaviour. It would mean for example that any queue limits > would be hit earlier. One could argue that would be an improvement, but it > may cause issues for systems when upgrading. > > The purpose of this mail is to solicit some feedback from users as to which of > these options (or indeed other options that have not occurred to me) would > be preferable. > > --Gordon. > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional > commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
On 09/27/2013 01:01 PM, Pavel Moravec wrote: I would vote for (c) option. Thanks, Pavel! Negs: - "backward incompatible" change Pros: - the incompatibility requires just a change in one type of number during one upgrade And I'd note this is a worst case scenario also. For many current uses the queue depth limit if specified in bytes is unlikely to be so precisely calibrated as to cause problems. Many uses may not even notice the change. - calculation of message size should be unified - assume a queue enqueuing messages from 1.0 and 0-10 client - limiting queue in "size" units is IMHO useful to limit memory consumption of the queue - where counting message headers in the limit makes more sense - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
On 09/27/2013 12:52 PM, Jakub Scholz wrote: Assuming there are no plans to abandon AMQP 0.10, I would say that in my opinion the option c) is worth the effort both on Qpid side as well as for the users side. The option d) seems to be unnecessary. At least with our use cases we never plan the queue sizes exactly to the last byte and I assume the difference between the 0.10 and 1.0 will be rather small. Thanks for the response, Jakub! From some quick experiments, the difference is around 40 bytes. Though this can vary a little depending on the exact message 9 e.g. which properties are set ttl, priority etc, it doesn't vary much with content size or with the number of headers. (In one test I compared messages with 1, 5 and 10 equal sized headers and either 10 or 100 bytes content in both protocols and the range of differences was 40-43 bytes). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
I agree with Pavel and Jakub. Tying the "bytes" depth as closely as possible to actual memory consumption is probably a good thing. We could even consider counting the message overhead (the memory used that is not part of the actual message). -Ted On 09/27/2013 08:01 AM, Pavel Moravec wrote: I would vote for (c) option. Negs: - "backward incompatible" change Pros: - the incompatibility requires just a change in one type of number during one upgrade - calculation of message size should be unified - assume a queue enqueuing messages from 1.0 and 0-10 client - limiting queue in "size" units is IMHO useful to limit memory consumption of the queue - where counting message headers in the limit makes more sense Kind regards, Pavel - Original Message - From: "Gordon Sim" To: users@qpid.apache.org Sent: Friday, September 27, 2013 1:04:17 PM Subject: Re: survey: c++ broker and queue depth statistics On 09/27/2013 11:47 AM, Gordon Sim wrote: The c++ broker reports a queue depth in terms of total bytes, as well as the number of messages. For 0-10 the bytes statistic is calculated by aggregating only the content size (i.e. the size of the body segment). For 1.0 it is the whole message including properties etc (i.e. the payload of the transfer 'performative'). So the size will be different depending on the protocol used in sending it and this difference can be quite marked. E.g. in an extreme case where there are many headers but no content, the bytes reported if sent over 0-10 would be 0 whereas if sent over 1.0 could easily be several hundred bytes. The question is what to do about this. The options are (a) accept that they are inconsistent between versions, (b) modify the 1.0 path to only record the application-data or (c) modify the 0-10 path to include the size of the header segment. I should point out that neither (b) or (c) give identical sizes for the same application data due to minor differences in the two type systems. However the difference would be fairly small and relatively fixed whereas for (a) it can vary hugely depending on the exact message. An option (d) would be to try and come up with a scheme where the size would be always identical. I'm not hugely keen on this as it would add overhead (e.g. calculating the size of a set of 0-10 headers if encoded in the 1.0 type scheme). While (c) seems to me to be logical the most 'correct', it would be a difference in behaviour. It would mean for example that any queue limits would be hit earlier. One could argue that would be an improvement, but it may cause issues for systems when upgrading. The purpose of this mail is to solicit some feedback from users as to which of these options (or indeed other options that have not occurred to me) would be preferable. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
I would vote for (c) option. Negs: - "backward incompatible" change Pros: - the incompatibility requires just a change in one type of number during one upgrade - calculation of message size should be unified - assume a queue enqueuing messages from 1.0 and 0-10 client - limiting queue in "size" units is IMHO useful to limit memory consumption of the queue - where counting message headers in the limit makes more sense Kind regards, Pavel - Original Message - > From: "Gordon Sim" > To: users@qpid.apache.org > Sent: Friday, September 27, 2013 1:04:17 PM > Subject: Re: survey: c++ broker and queue depth statistics > > On 09/27/2013 11:47 AM, Gordon Sim wrote: > > The c++ broker reports a queue depth in terms of total bytes, as well as > > the number of messages. > > > > For 0-10 the bytes statistic is calculated by aggregating only the > > content size (i.e. the size of the body segment). For 1.0 it is the > > whole message including properties etc (i.e. the payload of the transfer > > 'performative'). > > > > So the size will be different depending on the protocol used in sending > > it and this difference can be quite marked. E.g. in an extreme case > > where there are many headers but no content, the bytes reported if sent > > over 0-10 would be 0 whereas if sent over 1.0 could easily be several > > hundred bytes. > > > > The question is what to do about this. The options are (a) accept that > > they are inconsistent between versions, (b) modify the 1.0 path to only > > record the application-data or (c) modify the 0-10 path to include the > > size of the header segment. > > I should point out that neither (b) or (c) give identical sizes for the > same application data due to minor differences in the two type systems. > However the difference would be fairly small and relatively fixed > whereas for (a) it can vary hugely depending on the exact message. > > An option (d) would be to try and come up with a scheme where the size > would be always identical. I'm not hugely keen on this as it would add > overhead (e.g. calculating the size of a set of 0-10 headers if encoded > in the 1.0 type scheme). > > > While (c) seems to me to be logical the most 'correct', it would be a > > difference in behaviour. It would mean for example that any queue limits > > would be hit earlier. One could argue that would be an improvement, but > > it may cause issues for systems when upgrading. > > > > The purpose of this mail is to solicit some feedback from users as to > > which of these options (or indeed other options that have not occurred > > to me) would be preferable. > > > > --Gordon. > > > > - > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > > For additional commands, e-mail: users-h...@qpid.apache.org > > > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: survey: c++ broker and queue depth statistics
Hi Gordon, Assuming there are no plans to abandon AMQP 0.10, I would say that in my opinion the option c) is worth the effort both on Qpid side as well as for the users side. The option d) seems to be unnecessary. At least with our use cases we never plan the queue sizes exactly to the last byte and I assume the difference between the 0.10 and 1.0 will be rather small. Thanks & Regards Jakub On Fri, Sep 27, 2013 at 1:04 PM, Gordon Sim wrote: > On 09/27/2013 11:47 AM, Gordon Sim wrote: > >> The c++ broker reports a queue depth in terms of total bytes, as well as >> the number of messages. >> >> For 0-10 the bytes statistic is calculated by aggregating only the >> content size (i.e. the size of the body segment). For 1.0 it is the >> whole message including properties etc (i.e. the payload of the transfer >> 'performative'). >> >> So the size will be different depending on the protocol used in sending >> it and this difference can be quite marked. E.g. in an extreme case >> where there are many headers but no content, the bytes reported if sent >> over 0-10 would be 0 whereas if sent over 1.0 could easily be several >> hundred bytes. >> >> The question is what to do about this. The options are (a) accept that >> they are inconsistent between versions, (b) modify the 1.0 path to only >> record the application-data or (c) modify the 0-10 path to include the >> size of the header segment. >> > > I should point out that neither (b) or (c) give identical sizes for the > same application data due to minor differences in the two type systems. > However the difference would be fairly small and relatively fixed whereas > for (a) it can vary hugely depending on the exact message. > > An option (d) would be to try and come up with a scheme where the size > would be always identical. I'm not hugely keen on this as it would add > overhead (e.g. calculating the size of a set of 0-10 headers if encoded in > the 1.0 type scheme). > > > While (c) seems to me to be logical the most 'correct', it would be a >> difference in behaviour. It would mean for example that any queue limits >> would be hit earlier. One could argue that would be an improvement, but >> it may cause issues for systems when upgrading. >> >> The purpose of this mail is to solicit some feedback from users as to >> which of these options (or indeed other options that have not occurred >> to me) would be preferable. >> >> --Gordon. >> >> --**--**- >> To unsubscribe, e-mail: >> users-unsubscribe@qpid.apache.**org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >> > > --**--**- > To unsubscribe, e-mail: > users-unsubscribe@qpid.apache.**org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: survey: c++ broker and queue depth statistics
On 09/27/2013 11:47 AM, Gordon Sim wrote: The c++ broker reports a queue depth in terms of total bytes, as well as the number of messages. For 0-10 the bytes statistic is calculated by aggregating only the content size (i.e. the size of the body segment). For 1.0 it is the whole message including properties etc (i.e. the payload of the transfer 'performative'). So the size will be different depending on the protocol used in sending it and this difference can be quite marked. E.g. in an extreme case where there are many headers but no content, the bytes reported if sent over 0-10 would be 0 whereas if sent over 1.0 could easily be several hundred bytes. The question is what to do about this. The options are (a) accept that they are inconsistent between versions, (b) modify the 1.0 path to only record the application-data or (c) modify the 0-10 path to include the size of the header segment. I should point out that neither (b) or (c) give identical sizes for the same application data due to minor differences in the two type systems. However the difference would be fairly small and relatively fixed whereas for (a) it can vary hugely depending on the exact message. An option (d) would be to try and come up with a scheme where the size would be always identical. I'm not hugely keen on this as it would add overhead (e.g. calculating the size of a set of 0-10 headers if encoded in the 1.0 type scheme). While (c) seems to me to be logical the most 'correct', it would be a difference in behaviour. It would mean for example that any queue limits would be hit earlier. One could argue that would be an improvement, but it may cause issues for systems when upgrading. The purpose of this mail is to solicit some feedback from users as to which of these options (or indeed other options that have not occurred to me) would be preferable. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
