Hi Chris,
In 0.8.2, the simple consumer Java API supports committing/fetching
offsets that are stored in ZooKeeper. You don't need to issue any
ConsumerMetadataRequest for this. Unfortunately, the API currently
does not support fetching offsets that are stored in Kafka.
Thanks,
Joel
On Mon, Feb 16, 2015 at 05:02:08PM -0500, Christopher Piggott wrote:
> Hi,
>
> I am still using 0.8.1.1 because of the CPU use concerns.
>
> I'm confused about why the SimpleConsumer has:
>
> OffsetCommitResponse commitOffsets(OffsetCommitRequest request)
>
> and
>
> OffsetFetchResponse fetchOffsets(OffsetFetchRequest request)
>
> but no way that I can see to issue a ConsumerMetadataRequest, which is
> what I think when restarting my consumers so that they can begin
> working where they last left off (in the event that they were stopped
> for a while then restarted some time later, and new messages had come
> in).
>
> The fetchOffsets() works on time, usually it looks like you send it
> Earliest or Latest (beginning or end of what's currently in the
> stream).
>
> I realize the documentation says this:
>
>
> > *Downsides of using SimpleConsumer*The SimpleConsumer does require a
> > significant amount of work not needed in the Consumer Groups:
> >
> > 1. You must keep track of the offsets in your application to know where
> > you left off consuming.
> >
> > But that's not really quite true ... not as long as commitOffsets() has
> > been provided. It seems the SimpleConsumer provides you with a solution to
> > only one half of the problem of offset management.
>
> Using some zookeeper python scripts I wrote I can see that the
> commitOffsets() is doing its job and writing to
>
>
> /consumers/myGroupId/offsets/myTopic/0
>
>
> That has this value:
>
> ('32757408', ZnodeStat(czxid=2211679, mzxid=14779964, ctime=1423777630972,
> > mtime=1424122117397, version=12568262, cversion=0, aversion=0,
> > ephemeralOwner=0, dataLength=8, numChildren=0, pzxid=2211679))
>
>
> Now the question is just how to retrieve that - do I really have to
> have my client connect to ZK directly? If that's the case, future
> upgrades would break (e.g. 0.8.2 having its own storage for commit
> watermarks).
>
>
> What was the intent here, and what's the advice on how to proceed
> being that 0.8.2 is in an iffy state right now?
>
>
> --Chris
