Sorry for being unclear Neha. I meant that I had forgotten that the introduction of replicas is happening in 0.8 and I was confusing the two.
Thanks, rob > -----Original Message----- > From: Rob Withers [mailto:reefed...@gmail.com] > Sent: Monday, May 20, 2013 8:21 PM > To: 'users@kafka.apache.org' > Subject: RE: only-once consumer groups > > Right, Neha, consumer groups are from 7, while replicas are in 8. Does this > mean the simple consumer in 8 can recognize a leader change? > > Thanks, > rob > > > -----Original Message----- > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > Sent: Monday, May 20, 2013 8:00 PM > > To: users@kafka.apache.org > > Subject: RE: only-once consumer groups > > > > Rob, > > > > A consumer rebalances whenever a consumer process dies or a new > > consumer process joins the group. The details of the algorithm can be > > found here http://kafka.apache.org/07/design.html > > > > Thanks, > > Neha > > On May 20, 2013 6:45 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > > > That page is packed full of super design! Many of these features we > > > would find useful, I think. One thing I found myself not knowing is > > > what a consumer rebalance actually is. > > > > > > Is a rebalance when the thread that is consuming a particular > > > partition dies, this is detected and the partition is reassigned to > > > a new thread, thus violating manual partitioning? > > > > > > Or is a rebalance when a broker dies and new leader partitions are > elected? > > > The simple consumer must get told of a leader election and this is > > > nothing to do with rebalance, is it so? > > > > > > Thanks, > > > rob > > > > > > > > > > -----Original Message----- > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > > Sent: Friday, May 17, 2013 7:32 AM > > > > To: users@kafka.apache.org > > > > Subject: RE: only-once consumer groups > > > > > > > > We spent some time thinking about consolidating the high level and > > > > low > > > level > > > > consumer APIs. It will be great if you can read the wiki and > > > > provide > > > feedback > > > > - > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re > > > > - > > > > Design > > > > > > > > Thanks, > > > > Neha > > > > On May 16, 2013 10:29 PM, "Rob Withers" <reefed...@gmail.com> > wrote: > > > > > > > > > We want to ensure only-once message processing, but we also want > > > > > the benefit of rebalancing. commitOffsets updates all > > > > > partitions from out of a connector instance. We want to commit > > > > > the offset for just the partition that delivered a message to > > > > > the iterator, even if several fetchers are feeding a thread. > > > > > Perhaps the message metadata contains the partition a msg came > > > > > from; could you not just update the offset for that partition if > > > > > the property "only.once=true" is sent to the consumer connector? > > > > > > > > > > Thanks, > > > > > rob > > > > > > > > > > > -----Original Message----- > > > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > > > > Sent: Thursday, May 16, 2013 10:03 PM > > > > > > To: users@kafka.apache.org > > > > > > Subject: Re: only-once consumer groups > > > > > > > > > > > > Can you describe your requirements in a little more detail? > > > > > > > > > > > > Thanks, > > > > > > Neha > > > > > > On May 16, 2013 6:11 AM, "Withers, Robert" > > > > <robert.with...@dish.com> > > > > > > wrote: > > > > > > > > > > > > > is it technically feasible to use an only-once simple > > > > > > > consumer within a consumer group? > > > > > > > > > > > > > > thanks, > > > > > > > rob > > > > > > > > > > > > > > > >
