Re: how to control assign policy for consumers

[Guozhang Wang]

In the new consumer design the partitioning algorithm can be customized by
the user:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

Would that fit your use case?

Guozhang


On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич 
a.ryab...@ntc-vulkan.ru wrote:

 May be right place to add such optimization within
 https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?

 -Исходное сообщение-
 От: Guozhang Wang [mailto:wangg...@gmail.com]
 Отправлено: Wednesday, May 28, 2014 8:57 PM
 Кому: users@kafka.apache.org
 Тема: Re: how to control assign policy for consumers

 With high-level consumers you cannot control the assignment of partitions
 to consumers, and if partitions are located across all brokers each
 consumer is likely to connect to each of the brokers.

 One (little hacky) thing you can do is to manually assign the partitions
 to brokers such that each broker i of topic t is in broker i % 3, the for
 consumer fetching partition i of topic t, it will only connect to broker i
 % 3. By default the assignment is not aligned hence you will likely to have
 TCP between each pair of consumer/broker.

 Guozhang


 On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич 
 a.ryab...@ntc-vulkan.ru wrote:

  How can I tell consumer to connect  to one broker  ...For example:
   -  I have 3 topic and 3 broker...also each topic have 6 partitions...
- I than start 6 consumers to listen this 3 topics...And in such
  case we can found from 6 (each consumer connect only to one broker) to
  18 (each consumer connect to every broker) TCP connection  How I
  can minimize network connections?
 
  With best regards, Aleksey Ryabkov
 
 


 --
 -- Guozhang




-- 
-- Guozhang

Re: how to control assign policy for consumers

[Guozhang Wang]

With high-level consumers you cannot control the assignment of partitions
to consumers, and if partitions are located across all brokers each
consumer is likely to connect to each of the brokers.

One (little hacky) thing you can do is to manually assign the partitions to
brokers such that each broker i of topic t is in broker i % 3, the for
consumer fetching partition i of topic t, it will only connect to broker i
% 3. By default the assignment is not aligned hence you will likely to have
TCP between each pair of consumer/broker.

Guozhang


On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич 
a.ryab...@ntc-vulkan.ru wrote:

 How can I tell consumer to connect  to one broker  ...For example:
  -  I have 3 topic and 3 broker...also each topic have 6 partitions...
   - I than start 6 consumers to listen this 3 topics...And in such case we
 can found from 6 (each consumer connect only to one broker) to 18 (each
 consumer connect to every broker) TCP connection  How I can minimize
 network connections?

 With best regards, Aleksey Ryabkov




-- 
-- Guozhang