[
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659039#comment-14659039
]
Jun Rao commented on KAFKA-2406:
--------------------------------
[~becket_qin], the ZK watcher is a one-time watcher. So, when a watcher is
triggered, you have to do another read to register the watcher. There could be
multiple changes in between, but only one more watcher will be triggered.
However, the brokers may not be writing to ZK at exactly the same time. So, the
more brokers you have, likely the more times the watcher will be fired. I think
if we make the batch window configurable, one can adjust this value based on
the cluster size if needed.
Writes are more expensive than reads in ZK. By batching, we can significantly
reduce the number of ZK writes for isr changes. The controller still needs to
read the changed isr from ZK for all affected partitions though. Hopefully
there's not the main bottleneck.
> ISR propagation should be throttled to avoid overwhelming controller.
> ---------------------------------------------------------------------
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
> Issue Type: Bug
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller
> to broker traffic. This might significantly increase time of controlled
> shutdown or cluster startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
