Interesting. We were able to get this from JMX for Kafka 0.7.2 - here's a snapshot for one of our Kafka clusters: https://apps.sematext.com/spm-reports/s/93EtvhnOz0
Is getting this from ZK instead of JMX better? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Mar 8, 2014 at 7:33 AM, Pierre Andrews <pie...@quantifind.com>wrote: > It's been developed with 0.8 in mind. Giving a quick look at how 0.7 stores > data in ZK, it looks like it's slightly different and might not work > straight away. > > If you are intersted, have a look at: > > https://github.com/quantifind/KafkaOffsetMonitor/blob/master/src/main/scala/com/quantifind/kafka/OffsetGetter.scala > > This is where the data is read from ZK and Kafka. If you have a 0.7 setup, > you might be able to adapt it to get the right offsets, you would probably > not have to change anything else in the code. > > P > > > On Sat, Mar 8, 2014 at 2:48 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com > > wrote: > > > Hi, > > > > Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info > > needed for computing the lag? > > > > Thanks, > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews <pie...@quantifind.com > > >wrote: > > > > > Hello everyone, > > > > > > at Quantifind, we are big users of Kafka and we like it a lot! > > > In a few use cases, we had to figure out if a queue was growing and how > > its > > > consumers were behaving. There are a few command-line tools to try to > > > figure out what's going on, but it's not always easy to debug and to > see > > > what has happened while everyone was sleeping. > > > > > > To be able to monitor our kafka queues and consumers, we thus > developed a > > > tiny web app that could tell us the log size of each topic in the > brokers > > > and the offsets of each consumers. That's very similar to what the > kafka > > > ConsumerOffsetChecker tool* is doing, but instead of having a one off > > > snapshot, our app keeps an history and displays a nice graph of what's > > > going on. > > > > > > You can find screenshots and more details here: > > > http://quantifind.github.io/KafkaOffsetMonitor/ > > > > > > the code is on github: > > > https://github.com/quantifind/KafkaOffsetMonitor > > > > > > If you have kafka 0.8 setup, it's very easy to use: > > > > > > 1- download the current jar > > > > > > > > > http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar > > > 2- run it, pointing at your kafka brokers: > > > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \ > > > com.quantifind.kafka.offsetapp.OffsetGetterWeb \ > > > --zk zk-server1,zk-server2 \ > > > --port 8080 \ > > > --refresh 10.seconds \ > > > --retain 2.days > > > 3- open your browser and point it to localhost:8080 > > > > > > You can run it locally or host it on a server if you prefer. > > > > > > It's all open source and we'll be happy to receive issue report and > pull > > > requests. > > > > > > I hope that you like it and that it can find some uses in the rest of > the > > > Kafka community. > > > > > > Best > > > > > > Pierre > > > > > > PS: we are aware of https://github.com/claudemamo/kafka-web-consolebut > > > are > > > currently offering slightly different features. Hopefully we can merge > > the > > > projects in the future. > > > > > > * here: > > > > > > > > > https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala > > > > > >
