On Tue, Mar 28, 2017 at 1:07 PM, Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote:
> Hi Harsh, > > Thanks a lot for your answer. > > So you mean that I should look for kafka spout offsets in the zookeeper > servers which Storm uses for Nimbus/Supervisors communications, is that > right ? > Yes > > My problem is to find the zk path of these offsets: in the zookeeper > servers associated to my Kafka brokers, I have for example offsets stored > in the following path: /consumers/<consumer group id>/offsets > > However, when I access Storm's zookeeper's nodes using zkCli.sh, I do not > find any /consumers path. > > What is the path of the zk nodes which contain the offset of the Kafka > spouts? > There is no consumers folder but a folder name which is same as topic-id, you provided in Spout Config. It will be like /topic-id > > Best regards, > Alexandre Vermeerbergen > > > 2017-03-28 8:12 GMT+02:00 Harsh Choudhary <shry.ha...@gmail.com>: > >> The storm stores its offset in the Zookeeper, it is connected to. So, you >> won't find the offset information for the storm clients in the same place >> where other clients' offsets are. You must have provided topic id in Spout >> Config in Storm, this is where it stores the offset information in its >> Zookeeper. >> >> *Cheers!* >> >> Harsh Choudhary / Software Engineer >> >> Blog / express.harshti.me >> >> [image: Facebook] <https://facebook.com/shry.harsh> [image: Twitter] >> <https://twitter.com/har_ssh> [image: Google Plus] >> <https://plus.google.com/107567038912927268680> >> <https://in.linkedin.com/in/choudharyharsh> [image: Linkedin] >> <https://in.linkedin.com/in/choudharyharsh> [image: Instagram] >> <https://instagram.com/harsh.choudhary> >> <https://www.pinterest.com/shryharsh/>[image: 500px] >> <https://500px.com/harshchoudhary> [image: github] >> <https://github.com/shry15harsh> >> >> On Tue, Mar 28, 2017 at 11:27 AM, Alexandre Vermeerbergen < >> avermeerber...@gmail.com> wrote: >> >>> Hello, >>> >>> I need to programmatically monitor the offset of my Kafka Spouts (based >>> on storm-kafka, not storm-kafka-client). >>> >>> I already have a probe checking the offsets of all consumers of my Kafka >>> Brokers cluster. It takes into account both Zookeeper-based old consumers, >>> but also newer ones managing their offsets directly in Kafka brokers. >>> >>> But my probe do not "see" any of the consumers corresponding to Kafka >>> spouts. >>> >>> So my question is: where does Kafka spouts (from storm-kafka) store >>> their offsets as Kafka consumers? >>> >>> Tested configurations: >>> * Storm 1.0.1, Storm 1.0.3 and Storm 1.1.0 rc3, >>> * Kafka Brokers : 0.1.10.1 >>> >>> >>> Best regards, >>> Alexandre Vermeerbergen >>> >>> >> >
