That page is packed full of super design! Many of these features we would find useful, I think. One thing I found myself not knowing is what a consumer rebalance actually is.
Is a rebalance when the thread that is consuming a particular partition dies, this is detected and the partition is reassigned to a new thread, thus violating manual partitioning? Or is a rebalance when a broker dies and new leader partitions are elected? The simple consumer must get told of a leader election and this is nothing to do with rebalance, is it so? Thanks, rob > -----Original Message----- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Friday, May 17, 2013 7:32 AM > To: users@kafka.apache.org > Subject: RE: only-once consumer groups > > We spent some time thinking about consolidating the high level and low level > consumer APIs. It will be great if you can read the wiki and provide feedback > - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re- > Design > > Thanks, > Neha > On May 16, 2013 10:29 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > We want to ensure only-once message processing, but we also want the > > benefit of rebalancing. commitOffsets updates all partitions from out > > of a connector instance. We want to commit the offset for just the > > partition that delivered a message to the iterator, even if several > > fetchers are feeding a thread. Perhaps the message metadata contains > > the partition a msg came from; could you not just update the offset > > for that partition if the property "only.once=true" is sent to the > > consumer connector? > > > > Thanks, > > rob > > > > > -----Original Message----- > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > Sent: Thursday, May 16, 2013 10:03 PM > > > To: users@kafka.apache.org > > > Subject: Re: only-once consumer groups > > > > > > Can you describe your requirements in a little more detail? > > > > > > Thanks, > > > Neha > > > On May 16, 2013 6:11 AM, "Withers, Robert" > <robert.with...@dish.com> > > > wrote: > > > > > > > is it technically feasible to use an only-once simple consumer > > > > within a consumer group? > > > > > > > > thanks, > > > > rob > > > >
