[
https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13715900#comment-13715900
]
Guozhang Wang commented on KAFKA-984:
-------------------------------------
Following Sriram's comments, I am trying to propose a simpler solution for a
special case of "the event handler reports only adding ONE new topic".
1. Add a new function handleAddNewTopicForFilteredStreams, which updates
topicThreadIdAndQueues and topicStreamsMap and directly calls
syncedPartialRebalance(topic : String) of ZKRebalancerListener.
2. syncedPartialRebalance will use the same rebalanceLock as syncedRebalance.
2.1 For the added topic, first check if its ownership has not be claimed in ZK
yet. If it is already claimed (i.e., ZK has the znode for this topic in
owners), release the partitions first.
2.2 Read the number of consumers of the group from ZK (we only do this once for
all topics), and read the number of partitions of this topic, assign the
partitions to consumers using the same deterministic algorithm.
2.2 Try writing to the ZK for all added topics. If succeed, update fetchers
(start new threads) and returns, otherwise also returns since it means a
syncedRebalance will happen next.
Considerata: to make things simple when a topic change and consumer/broker
change happens at the same time:
1 At the beginning of syncedPartialRebalance checks if isWatcherTriggered is
true, if yes returns directly.
2 During syncedPartialRebalance if the new topic has already been claimed
(i.e., there are some znode in owners already), calls the syncedRebalance
directly.
The correctness of this is based on:
1. If syncedRebalance is triggered after the syncedPartialRebalance, it will
see the new topic.
2. If syncedRebalance is triggered before the syncedPartialRebalance and does
not see this new topic, a later syncedRebalance can still successfully do the
rebalance.
> Avoid a full rebalance in cases when a new topic is discovered but
> container/broker set stay the same
> -----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-984
> URL: https://issues.apache.org/jira/browse/KAFKA-984
> Project: Kafka
> Issue Type: Bug
> Reporter: Guozhang Wang
> Assignee: Guozhang Wang
> Fix For: 0.8
>
>
> Currently a full rebalance will be triggered on high level consumers even
> when just a new topic is added to ZK. Better avoid this behavior but only
> rebalance on this newly added topic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
