[
https://issues.apache.org/jira/browse/KAFKA-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093388#comment-18093388
]
Alex commented on KAFKA-20295:
------------------------------
ok my folder _kraft-combined-logs/__cluster_metadata-0_ have a tons of
checkpoint file and only one log file looking like:
(removed few of the checkpoint files for lisibility here)
{code:java}
-rw-r--r-- 1 kafka kafka 35109427 Jul 3 11:21 00000000000050772989.log
-rw-r--r-- 1 kafka kafka 10485760 Jul 3 11:21 00000000000050772989.index
-rw-r--r-- 1 kafka kafka 10485756 Jul 3 11:21 00000000000050772989.timeindex
-r-------- 1 kafka kafka 23017 Jul 3 10:36
00000000000051255604-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 3 09:36
00000000000051248222-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 22:36
00000000000051169028-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 21:36
00000000000051161829-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 20:36
00000000000051154630-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 19:36
00000000000051147431-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 18:36
00000000000051140232-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 17:36
00000000000051132852-0000000065.checkpoint
-r-------- 1 kafka kafka 23017 Jul 2 16:36
00000000000051125470-0000000065.checkpoint
-rw-r--r-- 1 kafka kafka 76 Jul 2 15:37 leader-epoch-checkpoint
-rw-r--r-- 1 kafka kafka 105 Jul 2 15:37 quorum-state
-rw-r--r-- 1 kafka kafka 10 Jul 2 15:36 00000000000051118259.snapshot
-r-------- 1 kafka kafka 22702 Jul 2 13:16
00000000000051099868-0000000058.checkpoint
-r-------- 1 kafka kafka 22702 Jun 30 20:16
00000000000050804709-0000000058.checkpoint
-r-------- 1 kafka kafka 22702 Jun 30 19:16
00000000000050797510-0000000058.checkpoint
-r-------- 1 kafka kafka 22702 Jun 30 18:16
00000000000050790311-0000000058.checkpoint
-r-------- 1 kafka kafka 22702 Jun 30 17:16
00000000000050783112-0000000058.checkpoint
-r-------- 1 kafka kafka 22702 Jun 30 16:16
00000000000050775913-0000000058.checkpoint
-rw-r--r-- 1 kafka kafka 43 Sep 9 2025 partition.metadata{code}
if I run on any checkpoint file (oldest or newest) I get:
{code:java}
kafka-metadata-shell.sh -s 00000000000051099868-0000000058.checkpoint
>> ls /image/cluster/controllers
1 2 3 4 5 6 7 8 9
>> ls /image/cluster/brokers
1 2 3 4
cat /image/cluster/controllers/2
ControllerRegistration(id=2, incarnationId=S2taUwWuRm-tpMNeH5yJPA,
zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLLER-9090',
securityProtocol=SASL_SSL, host='MY_HOST_NAME', port=9090)],
supportedFeatures={eligible.leader.replicas.version: 0-1, group.version: 0-1,
kraft.version: 0-1, metadata.version: 7-27, share.version: 0-1,
transaction.version: 0-2})
{code}
So this look like an old setup we had with broker,controller roles in version
4.1.0
but if I read the only .log file (00000000000050772989.log) I get:
{code:java}
[ Kafka Metadata Shell ]
>> ls /image/cluster/controllers
5 6 7 8 9
>> ls /image/cluster/brokers/
1 2 3 4
>> cat /image/cluster/controllers/5
ControllerRegistration(id=5, incarnationId=EsmhMrNQRrKIfagKbUY19w,
zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLLER-9090',
securityProtocol=SASL_SSL, host='MY_HOST_NAME', port=9090)],
supportedFeatures={eligible.leader.replicas.version: 0-1, group.version: 0-1,
kraft.version: 0-1, metadata.version: 7-30, share.version: 0-1,
streams.version: 0-1, transaction.version: 0-2})
{code}
Which look like my actual setup with 4 broker only and 5 controllers that do
return the right version 30 which is equals to the 4.3.1.
My brokers are reported as observers in the cluster is that correct ? (I expect
so but wanna be sure)
Is there a way to clear all the checkpoint or force the quorum to restart new ?
> note: host of course host is not MY_HOST_NAME but a node FQDN
> Removed controllers still in metadata, blocking finalizing upgrade to 4.2.0
> ---------------------------------------------------------------------------
>
> Key: KAFKA-20295
> URL: https://issues.apache.org/jira/browse/KAFKA-20295
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Environment: Kafka 4.2.0 (Scala 2.13) running on Debian Trixie 13.3
> Reporter: Roland Sommer
> Priority: Major
>
> While upgrading our kafka clusters to new operating systems I switched to
> dynamic voter configuration and removed controller instances with
> {{/opt/kafka/bin/kafka-metadata-quorum.sh}} and the {{remove-controller}}
> subcommand. Inspecting the cluster with {{describe}} only shows the actual
> running nodes.
> Now during the update to 4.2.0, the final metadata upgrade step complains
> about
> {code:java}
> Could not upgrade eligible.leader.replicas.version to 1. The update failed
> for all features since the following feature had an error: Invalid update
> version 29 for feature metadata.version. Controller 351 only supports
> versions 7-27{code}
> with 351 being an ID of an already removed controller. Inspecting a snapshot
> with {{/opt/kafka/bin/kafka-metadata-shell.sh}} indeed shows all controller
> ids of already removed controllers:
> {code:java}
> >> ls image/cluster/controllers/
> 158 206 351 584 611 686 {code}
> while other tools only show the expected nodes:
> {code:java}
> ~$ /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller
> localhost:9093 describe --replication --human-readable
> NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp
> Status
> 158 2gsvOvnT7urpZcA_-LUy5w 196823524 0 7 ms ago 8 ms ago Leader
> 611 27Ii-xdAZ7ReQBLsvvJb0A 196823524 0 348 ms ago 348 ms ago Follower
> 206 Q7X9o3XbKxk_3tz4T8torg 196823524 0 348 ms ago 348 ms ago Follower
> 226 7n6aedUEuytkqhBnbe7ESw 196823524 0 348 ms ago 348 ms ago Observer
> 181 tZ17VQ8cYpf7R-LyAQWf2w 196823524 0 349 ms ago 349 ms ago Observer
> 299 P4qXt3K0G5Qg_7w_UdvaNA 196823524 0 348 ms ago 348 ms ago Observer
> 290 bA0pqZFsUa45lRTB6bS4bg 196823524 0 348 ms ago 348 ms ago Observer
> 293 Av_12222lURKVYVt-aNKOQ 196823524 0 348 ms ago 348 ms ago Observer
> 485 glENIgkIng1MYDF8HxxoDQ 196823524 0 349 ms ago 350 ms ago Observer {code}
> Grepping through {{bin/kafka-dump-log.sh --cluster-metadata-decoder}} only
> shows the expected three {{REGISTER_CONTROLLER_RECORD}} entries.
> Is there any clear path for removing those stale nodes?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
