Hi Harsh, Thanks a lot for your answer.
So you mean that I should look for kafka spout offsets in the zookeeper servers which Storm uses for Nimbus/Supervisors communications, is that right ? My problem is to find the zk path of these offsets: in the zookeeper servers associated to my Kafka brokers, I have for example offsets stored in the following path: /consumers/<consumer group id>/offsets However, when I access Storm's zookeeper's nodes using zkCli.sh, I do not find any /consumers path. What is the path of the zk nodes which contain the offset of the Kafka spouts? Best regards, Alexandre Vermeerbergen 2017-03-28 8:12 GMT+02:00 Harsh Choudhary <shry.ha...@gmail.com>: > The storm stores its offset in the Zookeeper, it is connected to. So, you > won't find the offset information for the storm clients in the same place > where other clients' offsets are. You must have provided topic id in Spout > Config in Storm, this is where it stores the offset information in its > Zookeeper. > > *Cheers!* > > Harsh Choudhary / Software Engineer > > Blog / express.harshti.me > > [image: Facebook] <https://facebook.com/shry.harsh> [image: Twitter] > <https://twitter.com/har_ssh> [image: Google Plus] > <https://plus.google.com/107567038912927268680> > <https://in.linkedin.com/in/choudharyharsh> [image: Linkedin] > <https://in.linkedin.com/in/choudharyharsh> [image: Instagram] > <https://instagram.com/harsh.choudhary> > <https://www.pinterest.com/shryharsh/>[image: 500px] > <https://500px.com/harshchoudhary> [image: github] > <https://github.com/shry15harsh> > > On Tue, Mar 28, 2017 at 11:27 AM, Alexandre Vermeerbergen < > avermeerber...@gmail.com> wrote: > >> Hello, >> >> I need to programmatically monitor the offset of my Kafka Spouts (based >> on storm-kafka, not storm-kafka-client). >> >> I already have a probe checking the offsets of all consumers of my Kafka >> Brokers cluster. It takes into account both Zookeeper-based old consumers, >> but also newer ones managing their offsets directly in Kafka brokers. >> >> But my probe do not "see" any of the consumers corresponding to Kafka >> spouts. >> >> So my question is: where does Kafka spouts (from storm-kafka) store their >> offsets as Kafka consumers? >> >> Tested configurations: >> * Storm 1.0.1, Storm 1.0.3 and Storm 1.1.0 rc3, >> * Kafka Brokers : 0.1.10.1 >> >> >> Best regards, >> Alexandre Vermeerbergen >> >> >
