Anton Agestam created KAFKA-20762:
-------------------------------------
Summary: Node replacements are not supported during KRaft migration
Key: KAFKA-20762
URL: https://issues.apache.org/jira/browse/KAFKA-20762
Project: Kafka
Issue Type: Bug
Affects Versions: 3.9.2
Reporter: Anton Agestam
Hi,
We are Aiven, a Kafka-as-a-service provider. We run a large fleet of managed
Kafka services.
In a recent production incident where we had to use KRaft migration backup
tooling, we discovered that node replacements are not supported during the
migration sequence. As far as we have been able to find, this incompatibility
is not stated anywhere in documentation, and is causing a regression in ability
to recover clusters to a healthy state.
The problematic sequence is as follows:
# Cluster is put into KRaft migration, migrates all the way to [dual-write
phase|https://kafka.apache.org/37/operations/kraft/#migration-phases]. We
linger here to allow discovery of any issues that may occur.
# Loss of a single service node occurs, automation provisions a node
replacement.
# During provisioning of the replacement, we issue `kafka-storage.sh format`
as per [the
documentation.|https://kafka.apache.org/41/operations/kraft/#provisioning-nodes]
# The new node joins the cluster and starts serving workloads.
# Later, we discover an issue and an operator decides to trigger rollback of
the migration. When configuring brokers back into migration mode, the newly
provisioned broker goes into a crash loop with the below error log.
{code:java}
Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,647] ERROR Exiting Kafka due
to fatal exception during startup. (kafka.Kafka$)
java.lang.RuntimeException: Found unexpected
version in /srv/kafka/meta.properties. ZK-based brokers that are not migrating
only support version 0 (which is implicit when the `version` field is missing).
at
org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:489)
at
kafka.server.KafkaServer.startup(KafkaServer.scala:258)
at kafka.Kafka$.main(Kafka.scala:112)
at kafka.Kafka.main(Kafka.scala)
Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,650] INFO shutting down
(kafka.server.KafkaServer) {code}
---
The issue happens because kafka-storage.sh format always outputs
meta.properties with version=1. Once the rollback sequence is entered, this is
rejected. The provisioning tooling does not provide a way to format the local
storage in a way such that the broker spun up on top of it will support rolling
back an ongoing migration. And documentation does not mention this
incompatibility.
As these things are documented as orthogonal concepts, where the
meta.properties version is an opaque implementation detail, we think it's fair
to expect that there should be compatibility here. We'd like to propose as
solution to provide a flag for the format tool that will make it create a
meta.properties with version=0, and that supports rolling back. The issue could
also be solved by deciding that this incompatibility is as designed, and
documenting the status quo, but we think that would be unfortunate as it
imposes increased risk and cumbersome operations during an already complex
operational mode.
As there is already support for formatting storage in a way that supports
rollback (the broker does this itself during the migration), it seems it should
not be hard to implement the fix in the format command.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
