I will.
Thanks anyway
Mohammad
On Feb 14, 2017 7:24 PM, "Cody Koeninger" <c...@koeninger.org> wrote:
> Not sure what to tell you at that point - maybe compare what is
> present in ZK to a known working group.
>
> On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar <mkar...@phemi.com>
> wrote:
> > Yes offset nodes are in zk and I can get the values.
> >
> > On Feb 14, 2017 6:54 PM, "Cody Koeninger" <c...@koeninger.org> wrote:
> >>
> >> Are there offset nodes in ZK?
> >>
> >> for a consumer group named mygroup, a topic named test, and partition
> >> 0 you should be able to connect to ZK and do e.g.
> >>
> >> get /consumers/mygroup/offsets/test/0
> >>
> >> If those don't exist, those are the ZK nodes you would need to make
> >> sure get created / updated from your spark job.
> >>
> >>
> >>
> >> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar <mkar...@phemi.com>
> >> wrote:
> >> > I'm running 0.10 version and
> >> >
> >> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >> >
> >> > lists the group.
> >> >
> >> > On Feb 14, 2017 6:34 PM, "Cody Koeninger" <c...@koeninger.org> wrote:
> >> >>
> >> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
> >> >> versions of the kafka server. Which version of the server are you
> >> >> running?
> >> >>
> >> >> At any rate, the low-level consumer used by the
> >> >> spark-streaming-kafka-0-8 integration isn't going to create a
> consumer
> >> >> group or otherwise interact with them. If
> >> >>
> >> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >> >>
> >> >> doesn't show the group you're expecting, have you used a consumer
> that
> >> >> would have created it? Or did you explicitly try to create the
> >> >> related ZK nodes yourself?
> >> >>
> >> >>
> >> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar <mkar...@phemi.com>
> >> >> wrote:
> >> >> > Tried
> >> >> >
> >> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
> >> >> > --describe
> >> >> > --group mygroup
> >> >> >
> >> >> > and got
> >> >> >
> >> >> > No topic available for consumer group provided!
> >> >> >
> >> >> > Let me know if you still need the code (have to make changes before
> >> >> > sharing).
> >> >> >
> >> >> > Thanks
> >> >> >
> >> >> >
> >> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger <
> c...@koeninger.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> Can you explain what wasn't successful and/or show code?
> >> >> >>
> >> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar <
> mkar...@phemi.com>
> >> >> >> wrote:
> >> >> >> > As explained here, direct approach of integration between spark
> >> >> >> > streaming
> >> >> >> > and kafka does not update offsets in Zookeeper, hence
> >> >> >> > Zookeeper-based
> >> >> >> > Kafka
> >> >> >> > monitoring tools will not show progress (details). We followed
> >> >> >> > the
> >> >> >> > recommended workaround to update the zookeeper with the latest
> >> >> >> > offset
> >> >> >> > after
> >> >> >> > each batch, but no success. Wondering if there's any end-to-end
> >> >> >> > example
> >> >> >> > we
> >> >> >> > can use?
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Mohammad
> >> >> >> >
> >> >> >
> >> >> >
>
