Hi, How about this feature? thanks
*We do plan to allow the high level consumer to specify a starting offset inthe future when we revisit the consumer design. Some of the details aredescribed inhttps://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design <https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design>* 2013-04-02 22:45 GMT+08:00 Jun Rao <jun...@gmail.com>: > Currently, we have tools like ImportOffset that can update the starting > offset of a consumer in ZK. However, one has to stop all consumers first. > > We do plan to allow the high level consumer to specify a starting offset in > the future when we revisit the consumer design. Some of the details are > described in > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design > > Thanks, > > Jun > > On Tue, Apr 2, 2013 at 6:56 AM, James Englert <jengl...@gilt.com> wrote: > > > I wrote some code > > here< > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E > > >that > > resets the offset. You just need to be sure to set this as the > > consumer(s) is starting up. I'm not certain if it is the "right way" to > > do things. > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E > > > > > > On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > Sean, > > > > > > A broker can have multiple topics, each with multiple partitions. Each > > > partition can be consumed by multiple consumers. > > > > > > Our high level consumer API doesn't allow you to specify a starting > > offset. > > > SimpleConsumer does. If you use SimpleConsumer, you are responsible for > > > managing the consumption of all partitions. Consumer group is only used > > in > > > the high level consumer. > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson <s...@burstly.com> wrote: > > > > > > > Hello, > > > > > > > > Hopefully I'm sending this question to the right place. I'm currently > > > > trying to set up a consumer that will allow me to specify the offset, > > > > partition, and consumer group ID all at the same time. This obviously > > > > causes a dilemma since neither the low-level or high-level consumer > > APIs > > > > seem to support all three. So I have a couple questions: > > > > > > > > 1) Am I correct in my understanding that there is a 1-to-1 > relationship > > > > between brokers and partitions? Are multiple consumers per partition > > > > supported? If not, I imagine I could just start up a simple consumer > > per > > > > partition and that would give me what I want. If so, then do simple > > > > consumers actually support consumer group IDs without my knowledge? > > > > > > > > 2) Is it possible to specify a custom offset for the high-level > > consumer > > > > API other than from the tail (autooffset.reset = "smallest") or the > > head > > > > (autooffset.reset = "largest")? I know that the offset for each > > consumer, > > > > topic, and partition relationship is stored in a zookeeper ephemeral > > > node. > > > > Would connecting to zookeeper and changing all these offsets to the > > > desired > > > > values give me what I want? I have tried this with just one consumer > > > using > > > > the high-level api with one partition and one zookeeper host and it > > > seemed > > > > to work alright. But I'm wondering if changing these offsets would > work > > > > correctly and efficiently with multiple partitions and consumers with > > the > > > > same ID, as well as multiple zookeeper hosts. > > > > > > > > Thanks, > > > > Sean > > > > > > > > > > > > > > > -- > > Jim Englert > > Gilt Groupe > > 2 Park Ave South, 5th Floor > > New York, NY 10011 > > M: 847-707-2942 > > Please accept my invitation to join Gilt: > > http://www.giltgroupe.com/invite/jenglert > > >
