Re: Get Latest Offset for Specific Topic for All Partition

2015-02-10 Thread Gwen Shapira
Yes, its the size of the log in bytes. We can improve on documentation a bit :) Metric is defined in Log.scala: newGauge("Size", new Gauge[Long] { def value = size }, tags) so the value is "size", and: /** * The size of the log in bytes */ def size: Long = logSegmen

Re: Get Latest Offset for Specific Topic for All Partition

2015-02-09 Thread Bhavesh Mistry
Hi Gwen, This JMX stats is good for calculate injection rate per partition. I do not have to depend on ZK to figuring out who is leader what is latest offset. One quick question, what is Size # ? is it # of bytes particular partition has on disk ? Unfortunately, MBean description is very l

Re: Get Latest Offset for Specific Topic for All Partition

2015-02-05 Thread Gwen Shapira
You can use the metrics Kafka publishes. I think the relevant metrics are: Log.LogEndOffset Log.LogStartOffset Log.size Gwen On Thu, Feb 5, 2015 at 11:54 AM, Bhavesh Mistry wrote: > HI All, > > I just need to get the latest offset # for topic (not for consumer group). > Which API to get this i

Re: Get Latest Offset for Specific Topic for All Partition

2015-02-05 Thread Joel Koshy
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIaccuratelygetoffsetsofmessagesforacertaintimestampusingOffsetRequest? However, you will need to issue a TopicMetadataRequest first to discover the leaders for all the partitions and then issue the offset request. On Thu, Feb 05, 2015

Get Latest Offset for Specific Topic for All Partition

2015-02-05 Thread Bhavesh Mistry
HI All, I just need to get the latest offset # for topic (not for consumer group). Which API to get this info ? My use case is to analyze the data injection rate to each of partition is uniform or not (close). For this, I am planing to dump the latest offset into graphite for each partition an