Yes offset nodes are in zk and I can get the values.
On Feb 14, 2017 6:54 PM, "Cody Koeninger" <c...@koeninger.org> wrote:
> Are there offset nodes in ZK?
>
> for a consumer group named mygroup, a topic named test, and partition
> 0 you should be able to connect to ZK and do e.g.
>
> get /consumers/mygroup/offsets/test/0
>
> If those don't exist, those are the ZK nodes you would need to make
> sure get created / updated from your spark job.
>
>
>
> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar <mkar...@phemi.com>
> wrote:
> > I'm running 0.10 version and
> >
> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >
> > lists the group.
> >
> > On Feb 14, 2017 6:34 PM, "Cody Koeninger" <c...@koeninger.org> wrote:
> >>
> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
> >> versions of the kafka server. Which version of the server are you
> >> running?
> >>
> >> At any rate, the low-level consumer used by the
> >> spark-streaming-kafka-0-8 integration isn't going to create a consumer
> >> group or otherwise interact with them. If
> >>
> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >>
> >> doesn't show the group you're expecting, have you used a consumer that
> >> would have created it? Or did you explicitly try to create the
> >> related ZK nodes yourself?
> >>
> >>
> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar <mkar...@phemi.com>
> >> wrote:
> >> > Tried
> >> >
> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
> >> > --describe
> >> > --group mygroup
> >> >
> >> > and got
> >> >
> >> > No topic available for consumer group provided!
> >> >
> >> > Let me know if you still need the code (have to make changes before
> >> > sharing).
> >> >
> >> > Thanks
> >> >
> >> >
> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger <c...@koeninger.org>
> >> > wrote:
> >> >>
> >> >> Can you explain what wasn't successful and/or show code?
> >> >>
> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar <mkar...@phemi.com>
> >> >> wrote:
> >> >> > As explained here, direct approach of integration between spark
> >> >> > streaming
> >> >> > and kafka does not update offsets in Zookeeper, hence
> Zookeeper-based
> >> >> > Kafka
> >> >> > monitoring tools will not show progress (details). We followed the
> >> >> > recommended workaround to update the zookeeper with the latest
> offset
> >> >> > after
> >> >> > each batch, but no success. Wondering if there's any end-to-end
> >> >> > example
> >> >> > we
> >> >> > can use?
> >> >> >
> >> >> > Thanks,
> >> >> > Mohammad
> >> >> >
> >> >
> >> >
>
