[
https://issues.apache.org/jira/browse/KAFKA-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134741#comment-13134741
]
Indebir Singh Pall commented on KAFKA-168:
------------------------------------------
Putting in the discussion from the thread which started this JIRA for more
context.
@Inder
Use-Case
------------
1. I have a topic T spread across two brokers (B1, B2)running on different
machines, each having 2 partitions configured for T. Totally 4 partitions (1-0,
1-1, 2-0, 2-1)
2. Consumer C1 is part of group g1 and is consuming from from B1, B2 for T
3. Add a new consumer C2 part of g1
This is triggering a re balance across C1 & C2 and eventually C1 gets 1-0, 1-1
and C2 gets 2-0, 2-1.
P.S. - B1, C1 are sharing the same machine, same is the case with B2,C2
Behavior
---------
both consumers are getting partitions which are hosted on the same boxes. Is
this a coincidence or an optimization w.r.t locality of data and will always be
applied.
@Jun
During rebalance, we simply sort all partitions and consumers by name and
give each consumer an even range of partitions. Since partitions on the same
broker sort together, they tend to be given out to the same consumer, as in
this case.
@Inder
Sorted partitions grouped together might go to same consumer but it may not be
going to the consumer where its local instead of NIO, if consumers are hosted
on brokers.
For example :
In this case my partitions were like 1-0, 1-1 hosted on a machine named
proudarmburn and 2-0, 2-1, later on trainingarmburn so we got the localization
as sorting followed the same behavior, however if my second machine name would
had been leadergather then after sorting it would had gotten partitions
starting with 1 series which are remote to it.
Since we know the brokers hosting the partitions and consumers which are
subscribed if we add affinity to broker while re balancing it could save quite
a bit of nio.
Of course I am not sure if its a recommended configuration to be able to have
consumers share hardware with brokers. What do you guys think
> Support locality in consumer partition assignment algorithm
> -----------------------------------------------------------
>
> Key: KAFKA-168
> URL: https://issues.apache.org/jira/browse/KAFKA-168
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Reporter: Jay Kreps
>
> There are some use-cases where it makes sense to co-locate brokers and
> consumer processes. In this case it would be nice to optimize the assignment
> of partitions to consumers so that the consumer preferentially consumes from
> the broker with which it is co-located.
> If we are going to do KAFKA-167, moving the assignment to the broker, it
> would make sense to do that first so we only have to change the logic in one
> place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
