Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Mohammad Kargar
I will. Thanks anyway Mohammad On Feb 14, 2017 7:24 PM, "Cody Koeninger" wrote: > Not sure what to tell you at that point - maybe compare what is > present in ZK to a known working group. > > On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar > wrote: > >

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Cody Koeninger
Not sure what to tell you at that point - maybe compare what is present in ZK to a known working group. On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar wrote: > Yes offset nodes are in zk and I can get the values. > > On Feb 14, 2017 6:54 PM, "Cody Koeninger"

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Mohammad Kargar
Yes offset nodes are in zk and I can get the values. On Feb 14, 2017 6:54 PM, "Cody Koeninger" wrote: > Are there offset nodes in ZK? > > for a consumer group named mygroup, a topic named test, and partition > 0 you should be able to connect to ZK and do e.g. > > get

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Cody Koeninger
Are there offset nodes in ZK? for a consumer group named mygroup, a topic named test, and partition 0 you should be able to connect to ZK and do e.g. get /consumers/mygroup/offsets/test/0 If those don't exist, those are the ZK nodes you would need to make sure get created / updated from your

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Cody Koeninger
Can you explain what wasn't successful and/or show code? On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar wrote: > As explained here, direct approach of integration between spark streaming > and kafka does not update offsets in Zookeeper, hence Zookeeper-based Kafka >

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread bryan . jeffrey
Mohammad, We store our offsets in Cassandra,  and use that for tracking. This solved a few issues for us,  as it provides a good persistence mechanism even when you're reading from multiple clusters. Bryan Jeffrey Get Outlook for Android On Tue, Feb 14, 2017 at 7:03 PM

streaming-kafka-0-8-integration (direct approach) and monitoring

2017-02-14 Thread Mohammad Kargar
As explained here , direct approach of integration between spark streaming and kafka does not update offsets in Zookeeper, hence Zookeeper-based Kafka monitoring tools will not show progress (details). We followed the