[
https://issues.apache.org/jira/browse/KAFKA-20989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Youssef Ismail updated KAFKA-20989:
-----------------------------------
Description:
During ZooKeeper-to-KRaft migration, the KRaft controller creates persistent
sequential notifications for [config
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkConfigMigrationClient.scala#L352-L357]
and [ACL
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkAclMigrationClient.scala#L41-L44].
These nodes are normally deleted after 15 minutes by
[{{ZkNodeChangeNotificationListener}}|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala#L125-L140],
which is started by [ZooKeeper-mode
brokers|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/KafkaServer.scala#L631-L633].
Once all brokers enter KRaft mode in dual-write phase no cleanup listener
remains, while the [migration controller continues
dual-writing|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/ControllerServer.scala#L284-L314].
Consequently, {{{}/config/changes{}}}, {{{}/kafka-acl-changes{}}}, and
{{/kafka-acl-extended-changes}} keep growing without deletion.
Normally if the migration happens quickly this shouldn't be an issue. However
we want to let our clusters bake for some time in this phase before finalizing
the migration which causes the backlog.
This can increase ZooKeeper load and eventually make {{getChildren}} exceed
{{{}jute.maxbuffer{}}}, potentially impairing rollback.
I think the simplest solution is to have the migration controller run an
equivalent cleanup while migration mode is enabled.
was:
During ZooKeeper-to-KRaft migration, the KRaft controller creates persistent
sequential notifications for [config
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkConfigMigrationClient.scala#L352-L357]
and [ACL
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkAclMigrationClient.scala#L41-L44].
These nodes are normally deleted after 15 minutes by
[{{ZkNodeChangeNotificationListener}}|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala#L125-L140],
which is started by [ZooKeeper-mode
brokers|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/KafkaServer.scala#L631-L633].
Once all brokers enter KRaft mode in dual-write phase no cleanup listener
remains, while the [migration controller continues
dual-writing|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/ControllerServer.scala#L284-L314].
Consequently, {{{}/config/changes{}}}, {{{}/kafka-acl-changes{}}}, and
{{/kafka-acl-extended-changes}} keep growing without deletion.
Normally if the migration happens quickly this shouldn't be an issue However
we want to let our clusters bake for some time in this phase before finalizing
the migration which causes the backlog.
This can increase ZooKeeper load and eventually make {{getChildren}} exceed
{{{}jute.maxbuffer{}}}, potentially impairing rollback.
I think the simplest solution is to have the migration controller run an
equivalent cleanup while migration mode is enabled.
> KRaft dual-write does not clean up ZooKeeper config and ACL notification
> znodes
> -------------------------------------------------------------------------------
>
> Key: KAFKA-20989
> URL: https://issues.apache.org/jira/browse/KAFKA-20989
> Project: Kafka
> Issue Type: Bug
> Components: kraft
> Affects Versions: 3.9.1
> Reporter: Youssef Ismail
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> During ZooKeeper-to-KRaft migration, the KRaft controller creates persistent
> sequential notifications for [config
> changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkConfigMigrationClient.scala#L352-L357]
> and [ACL
> changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkAclMigrationClient.scala#L41-L44].
> These nodes are normally deleted after 15 minutes by
> [{{ZkNodeChangeNotificationListener}}|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala#L125-L140],
> which is started by [ZooKeeper-mode
> brokers|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/KafkaServer.scala#L631-L633].
> Once all brokers enter KRaft mode in dual-write phase no cleanup listener
> remains, while the [migration controller continues
> dual-writing|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/ControllerServer.scala#L284-L314].
> Consequently, {{{}/config/changes{}}}, {{{}/kafka-acl-changes{}}}, and
> {{/kafka-acl-extended-changes}} keep growing without deletion.
> Normally if the migration happens quickly this shouldn't be an issue. However
> we want to let our clusters bake for some time in this phase before
> finalizing the migration which causes the backlog.
> This can increase ZooKeeper load and eventually make {{getChildren}} exceed
> {{{}jute.maxbuffer{}}}, potentially impairing rollback.
> I think the simplest solution is to have the migration controller run an
> equivalent cleanup while migration mode is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
