[
https://issues.apache.org/jira/browse/KAFKA-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355829#comment-16355829
]
Kyle Ambroff-Kao commented on KAFKA-6469:
-----------------------------------------
[~junrao] The number of children of isr_change_notification isn't the problem.
It's the size of the children of isr_change_notification that is the problem.
In theory with a large enough cluster (I would guess around 400 or 500 with a
large number of partitions per broker) you'd run in to problems with the number
of children.
The specific situation I'm referring to is the broker attempting to write a
child of isr_change_notification which exceeds 1MB. I just submitted a PR that
addresses this.
> ISR change notification queue can prevent controller from making progress
> -------------------------------------------------------------------------
>
> Key: KAFKA-6469
> URL: https://issues.apache.org/jira/browse/KAFKA-6469
> Project: Kafka
> Issue Type: Bug
> Reporter: Kyle Ambroff-Kao
> Assignee: Kyle Ambroff-Kao
> Priority: Major
>
> When the writes /isr_change_notification in ZooKeeper (which is effectively a
> queue of ISR change events for the controller) happen at a rate high enough
> that the node with a watch can't dequeue them, the trouble starts.
> The watcher kafka.controller.IsrChangeNotificationListener is fired in the
> controller when a new entry is written to /isr_change_notification, and the
> zkclient library sends a GetChildrenRequest to zookeeper to fetch all child
> znodes.
> We've failures in one of our test clusters as the partition count started to
> climb north of 60k per broker. We had brokers writing child nodes under
> /isr_change_notification that were larger than the jute.maxbuffer size in
> ZooKeeper (1MB), causing the ZooKeeper server to drop the controller's
> session, effectively bricking the cluster.
> This can be partially mitigated by chunking ISR notifications to increase the
> maximum number of partitions a broker can host.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
