Hi Patricio, thank you!
I don't know how kafka-spout works. Right now, the monitor uses the value defined in kafka.utils.ZkUtils.ConsumersPath to grab the list of consumers in ZK. Again, that could be a parameter to the app somewhere to be more flexible and deal with separate zk path. P On Sat, Mar 8, 2014 at 12:42 AM, Patricio Echagüe <patric...@gmail.com> wrote: > > Great work! > > In addition to Dan's question, does it work with storm kafka-spout which > uses a separate zk path? > > Sent from my Nexus 4. > On Mar 7, 2014 7:18 PM, "Dan Hoffman" <hoffman...@gmail.com> wrote: > > > Hi - am I right in that for this tool to be effective, consumers must be > > using the high level consumer or otherwise keeping their offsets in > > zookeeper? Is there any way to track performance without that? > > > > > > On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <steve.mo...@gmail.com> wrote: > > > > > Very nice > > > > > > > On Mar 7, 2014, at 11:55, Pierre Andrews <pie...@quantifind.com> > > wrote: > > > > > > > > Claude, we should join forces ;) > > > > > > > > > > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <claude.m...@gmail.com> > > > wrote: > > > >> > > > >> Awesome!!! ;-) > > > >> > > > >> Claude > > > >> > > > >> > > > >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews < pie...@quantifind.com > > > >>> wrote: > > > >> > > > >>> Great! Thanks! > > > >>> > > > >>> > > > >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <jay.kr...@gmail.com> > > > wrote: > > > >>>> > > > >>>> This is really useful! I added it to the ecosystem page: > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem > > > >>>> > > > >>>> -Jay > > > >>>> > > > >>>> > > > >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews < > > > pie...@quantifind.com > > > >>>>> wrote: > > > >>>> > > > >>>>> Hello everyone, > > > >>>>> > > > >>>>> at Quantifind, we are big users of Kafka and we like it a lot! > > > >>>>> In a few use cases, we had to figure out if a queue was growing and > > > >> how > > > >>>> its > > > >>>>> consumers were behaving. There are a few command-line tools to try > > to > > > >>>>> figure out what's going on, but it's not always easy to debug and > > to > > > >>> see > > > >>>>> what has happened while everyone was sleeping. > > > >>>>> > > > >>>>> To be able to monitor our kafka queues and consumers, we thus > > > >>> developed a > > > >>>>> tiny web app that could tell us the log size of each topic in the > > > >>> brokers > > > >>>>> and the offsets of each consumers. That's very similar to what the > > > >>> kafka > > > >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one > > off > > > >>>>> snapshot, our app keeps an history and displays a nice graph of > > > >> what's > > > >>>>> going on. > > > >>>>> > > > >>>>> You can find screenshots and more details here: > > > >>>>> http://quantifind.github.io/KafkaOffsetMonitor/ > > > >>>>> > > > >>>>> the code is on github: > > > >>>>> https://github.com/quantifind/KafkaOffsetMonitor > > > >>>>> > > > >>>>> If you have kafka 0.8 setup, it's very easy to use: > > > >>>>> > > > >>>>> 1- download the current jar > > > >> > > > > > http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar > > > >>>>> 2- run it, pointing at your kafka brokers: > > > >>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \ > > > >>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \ > > > >>>>> --zk zk-server1,zk-server2 \ > > > >>>>> --port 8080 \ > > > >>>>> --refresh 10.seconds \ > > > >>>>> --retain 2.days > > > >>>>> 3- open your browser and point it to localhost:8080 > > > >>>>> > > > >>>>> You can run it locally or host it on a server if you prefer. > > > >>>>> > > > >>>>> It's all open source and we'll be happy to receive issue report and > > > >>> pull > > > >>>>> requests. > > > >>>>> > > > >>>>> I hope that you like it and that it can find some uses in the rest > > of > > > >>> the > > > >>>>> Kafka community. > > > >>>>> > > > >>>>> Best > > > >>>>> > > > >>>>> Pierre > > > >>>>> > > > >>>>> PS: we are aware of > > > >> https://github.com/claudemamo/kafka-web-consolebut > > > >>>>> are > > > >>>>> currently offering slightly different features. Hopefully we can > > > >> merge > > > >>>> the > > > >>>>> projects in the future. > > > >>>>> > > > >>>>> * here: > > > >> > > > > > https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala > > > >> > > > > >
