[
https://issues.apache.org/jira/browse/KAFKA-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093270#comment-18093270
]
Alex edited comment on KAFKA-20295 at 7/2/26 4:15 PM:
------------------------------------------------------
I ran into the exact same problem during the upgrade from 4.1.0 to 4.3.1
release version output:
{code:java}
kafka-features.sh --bootstrap-controller $BOOTSTRAP_SERVER:$CONTROLLER_PORT
--command-config $COMMAND_CONFIG_PATH upgrade --release-version 4.3
Could not upgrade eligible.leader.replicas.version to 1. The update failed for
all features since the following feature had an error: Invalid update version
30 for feature metadata.version. Controller 2 only supports versions 7-27
Could not upgrade group.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade kraft.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade metadata.version to 30. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade share.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade streams.version to 1. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade transaction.version to 2. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
7 out of 7 operation(s) failed.
{code}
even if the supported version printed 4.3
{code:java}
kafka-features.sh --bootstrap-controller $BOOTSTRAP_SERVER:$CONTROLLER_PORT
--command-config $COMMAND_CONFIG_PATH describe
Feature: eligible.leader.replicas.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: group.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: kraft.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: metadata.version SupportedMinVersion: 3.3-IV3
SupportedMaxVersion: 4.3-IV0 FinalizedVersionLevel: 4.1-IV1
Epoch: 51118268
Feature: share.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 0
Epoch: 51118268
Feature: streams.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 0
Epoch: 51118268
Feature: transaction.version SupportedMinVersion: 0
SupportedMaxVersion: 2 FinalizedVersionLevel: 2
Epoch: 51118268
{code}
my cluster is heathy and the replication report showed:
{code:java}
kafka-metadata-quorum.sh --bootstrap-controller $BOOTSTRAP_SERVER:9090
--command-config $COMMAND_CONFIG_PATH describe --replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp
LastCaughtUpTimestamp Status
5 4np8eilV3Zq7g2y7YQowGg 51137519 0 1783008857682
1783008857682 Leader
6 nTgrHxgtqNYTRd38sVmGzA 51137519 0 1783008857206
1783008857206 Follower
7 SocQOxBZRxSENVYlTsU1fw 51137519 0 1783008857211
1783008857211 Follower
8 _Uuav4ZhRDCMuOoBtAF0TA 51137519 0 1783008857206
1783008857206 Follower
9 Ex2zKbzjS9e0OG4NaWCuVg 51137519 0 1783008857211
1783008857211 Follower
1 Ishmrc3Y-1_erf9-TSqo-w 51137519 0 1783008857206
1783008857206 Observer
3 WsoWm7C2A-q9K74953n2QQ 51137519 0 1783008857206
1783008857206 Observer
2 WDJ4ikQGFf_J0-mEL_h0yg 51137519 0 1783008857206
1783008857206 Observer
4 q8a8qQVA501OVa7fb-4wFg 51137519 0 1783008857206
1783008857206 Observer
{code}
[~rsommer] I didn't archive to read my snapshot to see if something looks odd
(like a stale registered controller) could you share your full command ?
if someone have any clue or hint it would be appreciated !
was (Author: JIRAUSER311782):
I ran into the exact same problem during the upgrade from 4.1.0 to 4.3.1
release version output:
{code:java}
kafka-features.sh --bootstrap-controller $BOOTSTRAP_SERVER:$CONTROLLER_PORT
--command-config $COMMAND_CONFIG_PATH upgrade --release-version 4.3
Could not upgrade eligible.leader.replicas.version to 1. The update failed for
all features since the following feature had an error: Invalid update version
30 for feature metadata.version. Controller 2 only supports versions 7-27
Could not upgrade group.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade kraft.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade metadata.version to 30. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade share.version to 1. The update failed for all features since
the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade streams.version to 1. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
Could not upgrade transaction.version to 2. The update failed for all features
since the following feature had an error: Invalid update version 30 for feature
metadata.version. Controller 2 only supports versions 7-27
7 out of 7 operation(s) failed.
{code}
even if the supported version printed 4.3
{code:java}
kafka-features.sh --bootstrap-controller $BOOTSTRAP_SERVER:$CONTROLLER_PORT
--command-config $COMMAND_CONFIG_PATH describe
Feature: eligible.leader.replicas.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: group.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: kraft.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 1
Epoch: 51118268
Feature: metadata.version SupportedMinVersion: 3.3-IV3
SupportedMaxVersion: 4.3-IV0 FinalizedVersionLevel: 4.1-IV1
Epoch: 51118268
Feature: share.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 0
Epoch: 51118268
Feature: streams.version SupportedMinVersion: 0
SupportedMaxVersion: 1 FinalizedVersionLevel: 0
Epoch: 51118268
Feature: transaction.version SupportedMinVersion: 0
SupportedMaxVersion: 2 FinalizedVersionLevel: 2
Epoch: 51118268
{code}
[~rsommer] I didn't archive to read my snapshot to see if something looks odd
(like a stale registered controller) could you share your full command ?
if someone have any clue or hint it would be appreciated !
> Removed controllers still in metadata, blocking finalizing upgrade to 4.2.0
> ---------------------------------------------------------------------------
>
> Key: KAFKA-20295
> URL: https://issues.apache.org/jira/browse/KAFKA-20295
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Environment: Kafka 4.2.0 (Scala 2.13) running on Debian Trixie 13.3
> Reporter: Roland Sommer
> Priority: Major
>
> While upgrading our kafka clusters to new operating systems I switched to
> dynamic voter configuration and removed controller instances with
> {{/opt/kafka/bin/kafka-metadata-quorum.sh}} and the {{remove-controller}}
> subcommand. Inspecting the cluster with {{describe}} only shows the actual
> running nodes.
> Now during the update to 4.2.0, the final metadata upgrade step complains
> about
> {code:java}
> Could not upgrade eligible.leader.replicas.version to 1. The update failed
> for all features since the following feature had an error: Invalid update
> version 29 for feature metadata.version. Controller 351 only supports
> versions 7-27{code}
> with 351 being an ID of an already removed controller. Inspecting a snapshot
> with {{/opt/kafka/bin/kafka-metadata-shell.sh}} indeed shows all controller
> ids of already removed controllers:
> {code:java}
> >> ls image/cluster/controllers/
> 158 206 351 584 611 686 {code}
> while other tools only show the expected nodes:
> {code:java}
> ~$ /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller
> localhost:9093 describe --replication --human-readable
> NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp
> Status
> 158 2gsvOvnT7urpZcA_-LUy5w 196823524 0 7 ms ago 8 ms ago Leader
> 611 27Ii-xdAZ7ReQBLsvvJb0A 196823524 0 348 ms ago 348 ms ago Follower
> 206 Q7X9o3XbKxk_3tz4T8torg 196823524 0 348 ms ago 348 ms ago Follower
> 226 7n6aedUEuytkqhBnbe7ESw 196823524 0 348 ms ago 348 ms ago Observer
> 181 tZ17VQ8cYpf7R-LyAQWf2w 196823524 0 349 ms ago 349 ms ago Observer
> 299 P4qXt3K0G5Qg_7w_UdvaNA 196823524 0 348 ms ago 348 ms ago Observer
> 290 bA0pqZFsUa45lRTB6bS4bg 196823524 0 348 ms ago 348 ms ago Observer
> 293 Av_12222lURKVYVt-aNKOQ 196823524 0 348 ms ago 348 ms ago Observer
> 485 glENIgkIng1MYDF8HxxoDQ 196823524 0 349 ms ago 350 ms ago Observer {code}
> Grepping through {{bin/kafka-dump-log.sh --cluster-metadata-decoder}} only
> shows the expected three {{REGISTER_CONTROLLER_RECORD}} entries.
> Is there any clear path for removing those stale nodes?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
