We already have a JIRA for auto rebalance. I would be working on this soon.
KAFKA-930 <https://issues.apache.org/jira/browse/KAFKA-930> On 10/11/13 5:39 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >Hello Siyuan, > >For the automatic leader re-election, yes we are considering to make it >work. Could you file a JIRA for this issue? > >For the high-level consumer's rebalancing logic, you can find it at > >https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredictthere >sultsoftheconsumerrebabalance%3F > >Guozhang > > >On Fri, Oct 11, 2013 at 11:06 AM, hsy...@gmail.com <hsy...@gmail.com> >wrote: > >> Hi Jun, >> >> Thanks for your reply, but in a real cluster, one broker could serve >> different topics and different partitions, the simple consumer only has >> knowledge of brokers that are available but it has no knowledge to >>decide >> which broker is best to pick up to consume messages. If you don't >>choose >> carefully, multiple simple consumer might end up with reading from same >> node which is definitely not good for performance. >> Interesting thing is I find out there is >> command kafka-preferred-replica-election.sh which will try to equally >> distribute the leadership among different brokers, this is good that I >>can >> always let my simple consumer reads from leader broker(even it fails, >>the >> replica will pick up as leader which is fine). But why don't kafka >>cluster >> run this command automatically when there is a broker change(up/down) in >> the cluster so that the leadership can always be equally distributed >>among >> different brokers ASAP? I think it's very good for simple consumer to >> decide which broker is good to read from. >> >> Another question is I'm also curious how high-level consumer is >>balanced. I >> assume each high-level consumer know other consumers(int the same group) >> which broker they read message from and it can try to avoid those >>brokers >> and to pick up a free one? Is there a document for the balancing rule >> among high-level consumer. Does it always guarantee that after several >> leadership change/temporary broker fail, it can always equally >>distribute >> the read among the brokers. Basically I think it's nice to have a API to >> let dev know which consumer reads from which broker otherwise I don't >>know >> anything behind the high-level consumer >> >> Thanks! >> >> Best, >> Siyuan >> > > > >-- >-- Guozhang
