[ https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658781#comment-14658781 ]
Jiangjie Qin commented on KAFKA-2406: ------------------------------------- [~junrao], I'm still doing the test, here is what I see: 1. When controller was on old version, the controlled shutdown during rolling bounce seems to be normal. 2. After controller is running on new version, the controlled shutdown of brokers become very slow. 3. There are many zk paths still left in /isr_change_notification after the cluster bounce. 4. The first broker shuts down a little bit slower than before, but after that the subsequent shutdown takes super long - which is expected. Because of [1] I was thinking maybe throttling UpdateMetadataRequest would be the minimum solution for now, but I have the same concern as you do on the number of zk writes and watcher fires. [3] probably is an indication of zk watcher cannot catch up with the change reported by brokers. Having broker side to batch the changes makes sense. I am thinking about doing the following: 1. Broker only update ISR change in a batch periodically by writing partitions data to /isr_change_notification/brokerId_IsrChangeEpoch path. so zkWrite is bounded to #broker/update_interval. 2. Instead of using zk watcher, controller simply periodically query zookeeper and propagate ISR changes. > ISR propagation should be throttled to avoid overwhelming controller. > --------------------------------------------------------------------- > > Key: KAFKA-2406 > URL: https://issues.apache.org/jira/browse/KAFKA-2406 > Project: Kafka > Issue Type: Bug > Reporter: Jiangjie Qin > Assignee: Jiangjie Qin > Priority: Blocker > > This is a follow up patch for KAFKA-1367. > We need to throttle the ISR propagation rate to avoid flooding in controller > to broker traffic. This might significantly increase time of controlled > shutdown or cluster startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)