Jun, Thanks for the pointer to the docs. It would be great if more detail like this discussion were included in the Wiki. The detail of each metric is not always clear.
For our case (trying to monitor and alert on if the consumer is falling far behind the producer on a specific topic), what would you recommend? Should we compare BrokerAllTopicStat.[topic].getMessagesIn to ConsumerTopicStat.[topic].getMessagesPerTopic? On Wed, Apr 30, 2014 at 1:13 AM, Jun Rao <jun...@gmail.com> wrote: > Let's say a consumer wants to fetch 1000 bytes and there are two messages > in the broker each of 600 bytes. The broker will give 1000 bytes to the > consumer. The consumer will only be able to use the first 600 bytes since > the remaining 400 bytes doesn't form a full message. > > 0.7 jmx are documented in > https://cwiki.apache.org/confluence/display/KAFKA/Operations > > thanks, > > Jun > > > On Tue, Apr 29, 2014 at 9:51 PM, Arnaud Lawson <arnaud.law...@velos.io > >wrote: > > > Thank you Jun, > > > > What could a partial message be? A consumer fetching for messages on an > > empty queue and receiving a message representing that fact? > > > > As an example below i just ran a test using the producer & consumer > shells > > and JMXTerm. NumProduceRequests went from 13, 14 to 15 as i added values > > one by one to the queue for a test topic but then NumFetchRequests varies > > greatly. I am not sure why. I know there is only one consumer (the shell > > consumer) for this test kafka 0.7 instance and replication is not > available > > for this version. *Essentially** we would like to be able to measure the > > rate of consumption for some particular topic and make sure that this > rate > > is about the same as the rate of production for that topic. Which JMX > > metrics would be appropriate for this for Kafka 0.7?* Any suggestions > would > > be helpful. Thanks. > > > > > > #mbean = kafka:type=kafka.SocketServerStats: > > NumProduceRequests = 13; (Attribute exposed for management) > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > #mbean = kafka:type=kafka.SocketServerStats: > > NumFetchRequests = 4763; (Attribute exposed for management) > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > > #mbean = kafka:type=kafka.SocketServerStats: > > NumProduceRequests = 14; (Attribute exposed for management) > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > #mbean = kafka:type=kafka.SocketServerStats: > > NumFetchRequests = 4776; (Attribute exposed for management) > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > > #mbean = kafka:type=kafka.SocketServerStats: > > NumProduceRequests = 15; (Attribute exposed for management) > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > #mbean = kafka:type=kafka.SocketServerStats: > > NumFetchRequests = 4904; (Attribute exposed for management) > > > > > > > > > > On Wed, Apr 30, 2014 at 12:18 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > Is the data consumed more than once from different consumers? If that's > > not > > > the case, byteout could be a bit larger than bytein since we could > > return a > > > partial message to the consumer. The partial message will be ignored in > > the > > > consumer though. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Tue, Apr 29, 2014 at 2:13 PM, Arnaud Lawson <arnaud.law...@velos.io > > > >wrote: > > > > > > > We are using Kafka 0.7.1 and i believe replication isn't available > for > > > this > > > > version (correct me if i am wrong)... > > > > > > > > Basically we want to be able to measure the rate of consumption for > > some > > > > particular topic and make sure that this rate is about the same as > the > > > rate > > > > of production for that topic. Which JMX metrics could be appropriate > > for > > > > this - for Kafka 0.7? Thanks. > > > > > > > > > > > > On Tue, Apr 29, 2014 at 4:28 PM, Bae, Jae Hyeon <metac...@gmail.com> > > > > wrote: > > > > > > > > > Are you using 0.8? BytesOut will include the traffic for > replication. > > > If > > > > > you have no consumer and replication factor is 2, BytesOut should > be > > > > > exactly double of BytesIn. > > > > > > > > > > > > > > > On Tue, Apr 29, 2014 at 1:26 PM, Arnaud Lawson < > > arnaud.law...@velos.io > > > > > >wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > After graphing the cumulative values of Bytesin and Bytesout for > > one > > > of > > > > > our > > > > > > Kafka topics, we've noticed that the Bytesout values are > sometimes > > > > > greater > > > > > > than the Bytesin values. We are not really sure how this would be > > > > > possible > > > > > > if the max that a consumer can consume would be what has been put > > on > > > > the > > > > > > queue for that topic. Any explanations? Thanks. > > > > > > > > > > > > -- > > > > > > > > > > > > Arnaud Lawson > > > > > > , > > > > > > > > > > > > Systems Operations Engineer > > > > > > *VELOS* > > > > > > Accelerating Machine Learning. > > > > > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > > > C: (240) 393 - 6703 > > > > > > F: 646.349.4063 > > > > > > E: arnaud.law...@velos.io > > > > > > W: www.velos.io > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Arnaud Lawson > > > > , > > > > > > > > Systems Operations Engineer > > > > *VELOS* > > > > Accelerating Machine Learning. > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > C: (240) 393 - 6703 > > > > F: 646.349.4063 > > > > E: arnaud.law...@velos.io > > > > W: www.velos.io > > > > > > > > > > > > > > > -- > > > > Arnaud Lawson > > , > > > > Systems Operations Engineer > > *VELOS* > > Accelerating Machine Learning. > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > C: (240) 393 - 6703 > > F: 646.349.4063 > > E: arnaud.law...@velos.io > > W: www.velos.io > > > -- Arnaud Lawson , Systems Operations Engineer *VELOS* Accelerating Machine Learning. 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 C: (240) 393 - 6703 F: 646.349.4063 E: arnaud.law...@velos.io W: www.velos.io
