Setting the consumer's offset

[Sean Grayson]

Hello,

Hopefully I'm sending this question to the right place. I'm currently
trying to set up a consumer that will allow me to specify the offset,
partition, and consumer group ID all at the same time. This obviously
causes a dilemma since neither the low-level or high-level consumer APIs
seem to support all three. So I have a couple questions:

1) Am I correct in my understanding that there is a 1-to-1 relationship
between brokers and partitions? Are multiple consumers per partition
supported? If not, I imagine I could just start up a simple consumer per
partition and that would give me what I want. If so, then do simple
consumers actually support consumer group IDs without my knowledge?

2) Is it possible to specify a custom offset for the high-level consumer
API other than from the tail (autooffset.reset = "smallest") or the head
(autooffset.reset = "largest")? I know that the offset for each consumer,
topic, and partition relationship is stored in a zookeeper ephemeral node.
Would connecting to zookeeper and changing all these offsets to the desired
values give me what I want? I have tried this with just one consumer using
the high-level api with one partition and one zookeeper host and it seemed
to work alright. But I'm wondering if changing these offsets would work
correctly and efficiently with multiple partitions and consumers with the
same ID, as well as multiple zookeeper hosts.

Thanks,
Sean

Re: Setting the consumer's offset

[Jun Rao]

Sean,

A broker can have multiple topics, each with multiple partitions. Each
partition can be consumed by multiple consumers.

Our high level consumer API doesn't allow you to specify a starting offset.
SimpleConsumer does. If you use SimpleConsumer, you are responsible for
managing the consumption of all partitions. Consumer group is only used in
the high level consumer.

Thanks,

Jun

On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson  wrote:

> Hello,
>
> Hopefully I'm sending this question to the right place. I'm currently
> trying to set up a consumer that will allow me to specify the offset,
> partition, and consumer group ID all at the same time. This obviously
> causes a dilemma since neither the low-level or high-level consumer APIs
> seem to support all three. So I have a couple questions:
>
> 1) Am I correct in my understanding that there is a 1-to-1 relationship
> between brokers and partitions? Are multiple consumers per partition
> supported? If not, I imagine I could just start up a simple consumer per
> partition and that would give me what I want. If so, then do simple
> consumers actually support consumer group IDs without my knowledge?
>
> 2) Is it possible to specify a custom offset for the high-level consumer
> API other than from the tail (autooffset.reset = "smallest") or the head
> (autooffset.reset = "largest")? I know that the offset for each consumer,
> topic, and partition relationship is stored in a zookeeper ephemeral node.
> Would connecting to zookeeper and changing all these offsets to the desired
> values give me what I want? I have tried this with just one consumer using
> the high-level api with one partition and one zookeeper host and it seemed
> to work alright. But I'm wondering if changing these offsets would work
> correctly and efficiently with multiple partitions and consumers with the
> same ID, as well as multiple zookeeper hosts.
>
> Thanks,
> Sean
>

Re: Setting the consumer's offset

[James Englert]

I wrote some code
herethat
resets the offset.  You just need to be sure to set this as the
consumer(s) is starting up.   I'm not certain if it is the "right way" to
do things.

http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E


On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao  wrote:

> Sean,
>
> A broker can have multiple topics, each with multiple partitions. Each
> partition can be consumed by multiple consumers.
>
> Our high level consumer API doesn't allow you to specify a starting offset.
> SimpleConsumer does. If you use SimpleConsumer, you are responsible for
> managing the consumption of all partitions. Consumer group is only used in
> the high level consumer.
>
> Thanks,
>
> Jun
>
> On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson  wrote:
>
> > Hello,
> >
> > Hopefully I'm sending this question to the right place. I'm currently
> > trying to set up a consumer that will allow me to specify the offset,
> > partition, and consumer group ID all at the same time. This obviously
> > causes a dilemma since neither the low-level or high-level consumer APIs
> > seem to support all three. So I have a couple questions:
> >
> > 1) Am I correct in my understanding that there is a 1-to-1 relationship
> > between brokers and partitions? Are multiple consumers per partition
> > supported? If not, I imagine I could just start up a simple consumer per
> > partition and that would give me what I want. If so, then do simple
> > consumers actually support consumer group IDs without my knowledge?
> >
> > 2) Is it possible to specify a custom offset for the high-level consumer
> > API other than from the tail (autooffset.reset = "smallest") or the head
> > (autooffset.reset = "largest")? I know that the offset for each consumer,
> > topic, and partition relationship is stored in a zookeeper ephemeral
> node.
> > Would connecting to zookeeper and changing all these offsets to the
> desired
> > values give me what I want? I have tried this with just one consumer
> using
> > the high-level api with one partition and one zookeeper host and it
> seemed
> > to work alright. But I'm wondering if changing these offsets would work
> > correctly and efficiently with multiple partitions and consumers with the
> > same ID, as well as multiple zookeeper hosts.
> >
> > Thanks,
> > Sean
> >
>



-- 
Jim Englert
Gilt Groupe
2 Park Ave South, 5th Floor
New York, NY 10011
M: 847-707-2942
Please accept my invitation to join Gilt:
http://www.giltgroupe.com/invite/jenglert

Re: Setting the consumer's offset

[Jun Rao]

Currently, we have tools like ImportOffset that can update the starting
offset of a consumer in ZK. However, one has to stop all consumers first.

We do plan to allow the high level consumer to specify a starting offset in
the future when we revisit the consumer design. Some of the details are
described in
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Thanks,

Jun

On Tue, Apr 2, 2013 at 6:56 AM, James Englert  wrote:

> I wrote some code
> here<
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E
> >that
> resets the offset.  You just need to be sure to set this as the
> consumer(s) is starting up.   I'm not certain if it is the "right way" to
> do things.
>
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E
>
>
> On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao  wrote:
>
> > Sean,
> >
> > A broker can have multiple topics, each with multiple partitions. Each
> > partition can be consumed by multiple consumers.
> >
> > Our high level consumer API doesn't allow you to specify a starting
> offset.
> > SimpleConsumer does. If you use SimpleConsumer, you are responsible for
> > managing the consumption of all partitions. Consumer group is only used
> in
> > the high level consumer.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson  wrote:
> >
> > > Hello,
> > >
> > > Hopefully I'm sending this question to the right place. I'm currently
> > > trying to set up a consumer that will allow me to specify the offset,
> > > partition, and consumer group ID all at the same time. This obviously
> > > causes a dilemma since neither the low-level or high-level consumer
> APIs
> > > seem to support all three. So I have a couple questions:
> > >
> > > 1) Am I correct in my understanding that there is a 1-to-1 relationship
> > > between brokers and partitions? Are multiple consumers per partition
> > > supported? If not, I imagine I could just start up a simple consumer
> per
> > > partition and that would give me what I want. If so, then do simple
> > > consumers actually support consumer group IDs without my knowledge?
> > >
> > > 2) Is it possible to specify a custom offset for the high-level
> consumer
> > > API other than from the tail (autooffset.reset = "smallest") or the
> head
> > > (autooffset.reset = "largest")? I know that the offset for each
> consumer,
> > > topic, and partition relationship is stored in a zookeeper ephemeral
> > node.
> > > Would connecting to zookeeper and changing all these offsets to the
> > desired
> > > values give me what I want? I have tried this with just one consumer
> > using
> > > the high-level api with one partition and one zookeeper host and it
> > seemed
> > > to work alright. But I'm wondering if changing these offsets would work
> > > correctly and efficiently with multiple partitions and consumers with
> the
> > > same ID, as well as multiple zookeeper hosts.
> > >
> > > Thanks,
> > > Sean
> > >
> >
>
>
>
> --
> Jim Englert
> Gilt Groupe
> 2 Park Ave South, 5th Floor
> New York, NY 10011
> M: 847-707-2942
> Please accept my invitation to join Gilt:
> http://www.giltgroupe.com/invite/jenglert
>

Re: Setting the consumer's offset

[Yonghui Zhao]

Hi,

How about this feature? thanks





*We do plan to allow the high level consumer to specify a
starting offset inthe future when we revisit the consumer design. Some of
the details aredescribed
inhttps://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
*

2013-04-02 22:45 GMT+08:00 Jun Rao :

> Currently, we have tools like ImportOffset that can update the starting
> offset of a consumer in ZK. However, one has to stop all consumers first.
>
> We do plan to allow the high level consumer to specify a starting offset in
> the future when we revisit the consumer design. Some of the details are
> described in
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>
> Thanks,
>
> Jun
>
> On Tue, Apr 2, 2013 at 6:56 AM, James Englert  wrote:
>
> > I wrote some code
> > here<
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E
> > >that
> > resets the offset.  You just need to be sure to set this as the
> > consumer(s) is starting up.   I'm not certain if it is the "right way" to
> > do things.
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E
> >
> >
> > On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao  wrote:
> >
> > > Sean,
> > >
> > > A broker can have multiple topics, each with multiple partitions. Each
> > > partition can be consumed by multiple consumers.
> > >
> > > Our high level consumer API doesn't allow you to specify a starting
> > offset.
> > > SimpleConsumer does. If you use SimpleConsumer, you are responsible for
> > > managing the consumption of all partitions. Consumer group is only used
> > in
> > > the high level consumer.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson  wrote:
> > >
> > > > Hello,
> > > >
> > > > Hopefully I'm sending this question to the right place. I'm currently
> > > > trying to set up a consumer that will allow me to specify the offset,
> > > > partition, and consumer group ID all at the same time. This obviously
> > > > causes a dilemma since neither the low-level or high-level consumer
> > APIs
> > > > seem to support all three. So I have a couple questions:
> > > >
> > > > 1) Am I correct in my understanding that there is a 1-to-1
> relationship
> > > > between brokers and partitions? Are multiple consumers per partition
> > > > supported? If not, I imagine I could just start up a simple consumer
> > per
> > > > partition and that would give me what I want. If so, then do simple
> > > > consumers actually support consumer group IDs without my knowledge?
> > > >
> > > > 2) Is it possible to specify a custom offset for the high-level
> > consumer
> > > > API other than from the tail (autooffset.reset = "smallest") or the
> > head
> > > > (autooffset.reset = "largest")? I know that the offset for each
> > consumer,
> > > > topic, and partition relationship is stored in a zookeeper ephemeral
> > > node.
> > > > Would connecting to zookeeper and changing all these offsets to the
> > > desired
> > > > values give me what I want? I have tried this with just one consumer
> > > using
> > > > the high-level api with one partition and one zookeeper host and it
> > > seemed
> > > > to work alright. But I'm wondering if changing these offsets would
> work
> > > > correctly and efficiently with multiple partitions and consumers with
> > the
> > > > same ID, as well as multiple zookeeper hosts.
> > > >
> > > > Thanks,
> > > > Sean
> > > >
> > >
> >
> >
> >
> > --
> > Jim Englert
> > Gilt Groupe
> > 2 Park Ave South, 5th Floor
> > New York, NY 10011
> > M: 847-707-2942
> > Please accept my invitation to join Gilt:
> > http://www.giltgroupe.com/invite/jenglert
> >
>