[
https://issues.apache.org/jira/browse/KAFKA-17966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17897799#comment-17897799
]
Luke Chen commented on KAFKA-17966:
-----------------------------------
[~jsancio] [~cmccabe] , any thought about it?
> Controller replacement does not support scaling up before scaling down
> ----------------------------------------------------------------------
>
> Key: KAFKA-17966
> URL: https://issues.apache.org/jira/browse/KAFKA-17966
> Project: Kafka
> Issue Type: Bug
> Components: kraft
> Affects Versions: 3.9.0
> Reporter: Federico Valeri
> Priority: Major
>
> In KRaft, complex quorum changes are implemented as a series of
> single-controller changes. In this case, it is preferable to add controllers
> before removing controllers. For example, to replace a controller in a
> three-controller cluster, adding one controller and then removing the other
> allows the system to handle one controller failure at all times throughout
> the whole process. This is currently not possible, as it leads to
> DuplicateVoterException, so you are forced to do a scale down, followed by a
> scale up.
> Example:
> The operator can replace a failed disk with a new one. The replaced disk
> needs to be formatted with a new directory ID.
> {code}
> $ CLUSTER_ID="$(bin/kafka-cluster.sh cluster-id --bootstrap-server
> localhost:9092 | awk -F': ' '{print $2}')"
> $ bin/kafka-storage.sh format \
> --config /opt/kafka/server2/config/server.properties \
> --cluster-id "$CLUSTER_ID" \
> --no-initial-controllers \
> --ignore-formatted
> Formatting metadata directory /opt/kafka/server2/metadata with
> metadata.version 3.9-IV0.
> {code}
> After restarting the controller, the quorum will have two nodes with ID two:
> the original incarnation with a failed disk and an ever growing lag and
> follower status, plus a new one with a different directory ID and observer
> status.
> {code}
> $ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:8000 describe
> --re --hu
> NodeId DirectoryId LogEndOffset Lag
> LastFetchTimestamp LastCaughtUpTimestamp Status
> 0 pbvuBlaTTwKRxS5NLJwRFQ 535 0 6 ms ago
> 6 ms ago Leader
> 1 QjRpFtVDTtCa8OLXiSbmmA 535 0 283 ms ago
> 283 ms ago Follower
> 2 slcsM5ZAR0SMIF_u__MAeg 407 128 63307 ms ago
> 63802 ms ago Follower
> 2 wrqMDI1WDsqaooVSOtlgYw 535 0 281 ms ago
> 281 ms ago Observer
> 8 aXLz3ixjqzXhCYqKHRD4WQ 535 0 284 ms ago
> 284 ms ago Observer
> 7 KCriHQZm3TlxvEVNgyWKJw 535 0 284 ms ago
> 284 ms ago Observer
> 9 v5nnIwK8r0XqjyqlIPW-aw 535 0 284 ms ago
> 284 ms ago Observer
> {code}
> Once the new controller is in sync with the leader, we try to do a scale up.
> {code}
> $ bin/kafka-metadata-quorum.sh \
> --bootstrap-controller localhost:8000 \
> --command-config /opt/kafka/server2/config/server.properties \
> add-controller
> org.apache.kafka.common.errors.DuplicateVoterException: The voter id for
> ReplicaKey(id=2, directoryId=Optional[u7e_mCmg0VAIz0zuAOcraA]) is already
> part of the set of voters [ReplicaKey(id=0,
> directoryId=Optional[PbEthh6mR8iVNizvUTUVFw]), ReplicaKey(id=1,
> directoryId=Optional[kIpbbU79QaCIIiOLOyCjJg]), ReplicaKey(id=2,
> directoryId=Optional[2ab0gajpS5aUf5d-2Jw02w])].
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.DuplicateVoterException: The voter id for
> ReplicaKey(id=2, directoryId=Optional[u7e_mCmg0VAIz0zuAOcraA]) is already
> part of the set of voters [ReplicaKey(id=0,
> directoryId=Optional[PbEthh6mR8iVNizvUTUVFw]), ReplicaKey(id=1,
> directoryId=Optional[kIpbbU79QaCIIiOLOyCjJg]), ReplicaKey(id=2,
> directoryId=Optional[2ab0gajpS5aUf5d-2Jw02w])].
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
> Caused by: org.apache.kafka.common.errors.DuplicateVoterException: The voter
> id for ReplicaKey(id=2, directoryId=Optional[u7e_mCmg0VAIz0zuAOcraA]) is
> already part of the set of voters [ReplicaKey(id=0,
> directoryId=Optional[PbEthh6mR8iVNizvUTUVFw]), ReplicaKey(id=1,
> directoryId=Optional[kIpbbU79QaCIIiOLOyCjJg]), ReplicaKey(id=2,
> directoryId=Optional[2ab0gajpS5aUf5d-2Jw02w])].
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
