[
https://issues.apache.org/jira/browse/KAFKA-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Chen updated KAFKA-17866:
------------------------------
Description:
When upgrading old controllers to v3.9.0, we didn't update the kraft.version to
1 and the directory ID, which causes it blocks the following controller
addition/removal.
Steps to reproduce:
1. format 1 controller node in MV 3.8:
{code:java}
bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c
config/kraft/controller1.properties --release-version 3.8
{code}
2. start up the controller
3. update the metadata version to 3.9:
{code:java}
bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata
3.9
or
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature
kraft.version=1
{code}
4. format one additional controller to be expanded
{code:java}
bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config
config/kraft/controller2.properties --no-initial-controllers
{code}
5. Try to add this node
{code:java}
bin/kafka-metadata-quorum.sh --command-config
config/kraft/controller2.properties --bootstrap-controller localhost:9093
add-controller
{code}
It'll fail with:
{code:java}
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
support adding voter because the kraft.version feature is 0
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
support adding voter because the kraft.version feature is 0
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at
org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
at
org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
at
org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
at
org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster
doesn't support adding voter because the kraft.version feature is 0
{code}
Under this situation, I can't find any workaround to update kraft.version even
if rebooting the node.
Besides, we also didn't generate directory id for the upgraded nodes. Without
the directory id, even if we have correct kraft.version, the controller removal
still cannot succeed.
{code:java}
./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe
--replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp
LastCaughtUpTimestamp Status
1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123
1729752652123 Leader
4 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651778
1729752651778 Follower
5 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651782
1729752651782 Follower{code}
The directoryID `AAAAAAAAAAAAAAAAAAAAAA` in all 3 nodes means zero UUID (new
Uuid(0L, 0L)), which is not correct.
was:
When upgrading old controllers to v3.9.0, we didn't update the kraft.version to
1 and the directory ID, which causes it blocks the following controller
addition/removal.
Steps to reproduce:
1. format 1 controller node in MV 3.8:
{code:java}
bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c
config/kraft/controller1.properties --release-version 3.8
{code}
2. start up the controller
3. update the metadata version to 3.9:
{code:java}
bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata
3.9
or
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature
kraft.version=1
{code}
4. format one additional controller to be expanded
{code:java}
bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config
config/kraft/controller2.properties --no-initial-controllers
{code}
5. Try to add this node
{code:java}
bin/kafka-metadata-quorum.sh --command-config
config/kraft/controller2.properties --bootstrap-controller localhost:9093
add-controller
{code}
It'll fail with:
{code:java}
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
support adding voter because the kraft.version feature is 0
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
support adding voter because the kraft.version feature is 0
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at
org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
at
org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
at
org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
at
org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster
doesn't support adding voter because the kraft.version feature is 0
{code}
Under this situation, I can't find any workaround to update kraft.version even
if rebooting the node.
Besides, we also don't generate directory id for the upgraded nodes. Without
the directory id, even if we have correct kraft.version, the controller removal
still cannot succeed.
{code:java}
./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe
--replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp
LastCaughtUpTimestamp Status
1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123
1729752652123 Leader
4 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651778
1729752651778 Follower
5 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651782
1729752651782 Follower{code}
The directoryID `AAAAAAAAAAAAAAAAAAAAAA` means zero UUID (new Uuid(0L, 0L)),
which is not correct.
> kraft.version/dirID is not updated after upgrading to MV 3.9
> ------------------------------------------------------------
>
> Key: KAFKA-17866
> URL: https://issues.apache.org/jira/browse/KAFKA-17866
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.9.0
> Reporter: Luke Chen
> Priority: Blocker
> Fix For: 3.9.0
>
>
> When upgrading old controllers to v3.9.0, we didn't update the kraft.version
> to 1 and the directory ID, which causes it blocks the following controller
> addition/removal.
>
> Steps to reproduce:
> 1. format 1 controller node in MV 3.8:
> {code:java}
> bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c
> config/kraft/controller1.properties --release-version 3.8
> {code}
> 2. start up the controller
> 3. update the metadata version to 3.9:
> {code:java}
> bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade
> --metadata 3.9
> or
> bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature
> kraft.version=1
> {code}
> 4. format one additional controller to be expanded
> {code:java}
> bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config
> config/kraft/controller2.properties --no-initial-controllers
> {code}
> 5. Try to add this node
> {code:java}
> bin/kafka-metadata-quorum.sh --command-config
> config/kraft/controller2.properties --bootstrap-controller localhost:9093
> add-controller
> {code}
> It'll fail with:
> {code:java}
> org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
> support adding voter because the kraft.version feature is 0
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't
> support adding voter because the kraft.version feature is 0
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
> at
> org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> Cluster doesn't support adding voter because the kraft.version feature is 0
> {code}
>
> Under this situation, I can't find any workaround to update kraft.version
> even if rebooting the node.
> Besides, we also didn't generate directory id for the upgraded nodes. Without
> the directory id, even if we have correct kraft.version, the controller
> removal still cannot succeed.
> {code:java}
> ./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe
> --replication
> NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp
> LastCaughtUpTimestamp Status
> 1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123
> 1729752652123 Leader
> 4 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651778
> 1729752651778 Follower
> 5 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752651782
> 1729752651782 Follower{code}
>
> The directoryID `AAAAAAAAAAAAAAAAAAAAAA` in all 3 nodes means zero UUID (new
> Uuid(0L, 0L)), which is not correct.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
