Well may be yes and may no.... I think that in design we must point out that co-ordinator also must analize connection graph and must map consumers to broker with minimal network overhead...
-----Исходное сообщение----- От: Guozhang Wang [mailto:wangg...@gmail.com] Отправлено: Thursday, May 29, 2014 9:02 PM Кому: users@kafka.apache.org Тема: Re: how to control assign policy for consumers In the new consumer design the partitioning algorithm can be customized by the user: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Would that fit your use case? Guozhang On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > May be right place to add such optimization within > https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ? > > -----Исходное сообщение----- > От: Guozhang Wang [mailto:wangg...@gmail.com] > Отправлено: Wednesday, May 28, 2014 8:57 PM > Кому: users@kafka.apache.org > Тема: Re: how to control assign policy for consumers > > With high-level consumers you cannot control the assignment of > partitions to consumers, and if partitions are located across all > brokers each consumer is likely to connect to each of the brokers. > > One (little hacky) thing you can do is to manually assign the > partitions to brokers such that each broker i of topic t is in broker > i % 3, the for consumer fetching partition i of topic t, it will only > connect to broker i % 3. By default the assignment is not aligned > hence you will likely to have TCP between each pair of consumer/broker. > > Guozhang > > > On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > How can I tell consumer to connect to one broker ...For example: > > - I have 3 topic and 3 broker...also each topic have 6 partitions... > > - I than start 6 consumers to listen this 3 topics...And in such > > case we can found from 6 (each consumer connect only to one broker) > > to > > 18 (each consumer connect to every broker) TCP connection .... How I > > can minimize network connections? > > > > With best regards, Aleksey Ryabkov > > > > > > > -- > -- Guozhang > -- -- Guozhang
