Re: [storm-kafka] where is stored Kafka Spout consummer's offset?

2017-03-28 Thread Harsh Choudhary
On Tue, Mar 28, 2017 at 1:07 PM, Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote: > Hi Harsh, > > Thanks a lot for your answer. > > So you mean that I should look for kafka spout offsets in the zookeeper > servers which Storm uses for Nimbus/Supervisors communications, is that > right

Re: [storm-kafka] where is stored Kafka Spout consummer's offset?

2017-03-28 Thread Alexandre Vermeerbergen
Hi Harsh, Thanks a lot for your answer. So you mean that I should look for kafka spout offsets in the zookeeper servers which Storm uses for Nimbus/Supervisors communications, is that right ? My problem is to find the zk path of these offsets: in the zookeeper servers associated to my Kafka brok

Re: [storm-kafka] where is stored Kafka Spout consummer's offset?

2017-03-27 Thread Harsh Choudhary
The storm stores its offset in the Zookeeper, it is connected to. So, you won't find the offset information for the storm clients in the same place where other clients' offsets are. You must have provided topic id in Spout Config in Storm, this is where it stores the offset information in its Zooke

[storm-kafka] where is stored Kafka Spout consummer's offset?

2017-03-27 Thread Alexandre Vermeerbergen
Hello, I need to programmatically monitor the offset of my Kafka Spouts (based on storm-kafka, not storm-kafka-client). I already have a probe checking the offsets of all consumers of my Kafka Brokers cluster. It takes into account both Zookeeper-based old consumers, but also newer ones managing