Roland Sommer created KAFKA-15330:
-------------------------------------
Summary: Migration from ZK to KRaft works with 3.4 but fails from
3.5 upwards
Key: KAFKA-15330
URL: https://issues.apache.org/jira/browse/KAFKA-15330
Project: Kafka
Issue Type: Bug
Affects Versions: 3.5.1, 3.5.0
Environment: Debian Bookworm/12.1
kafka 3.4 and 3.5 / scala 2.13
OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
Reporter: Roland Sommer
We recently did some migration testing from our old ZK-based kafka clusters to
KRaft while still being on kafka 3.4. The migration tests succeeded at first
try. In the meantime we updated to kafka 3.5/3.5.1 and now we wanted to
continue our migration work, which ran into unexpected problems.
On the controller we get messages like:
{code:java}
Aug 10 06:49:33 kafkactl01 kafka-server-start.sh[48572]: [2023-08-10
06:49:33,072] INFO [KRaftMigrationDriver id=495] Still waiting for all
controller nodes ready to begin the migration. due to: Missing apiVersion from
nodes: [514, 760]
(org.apache.kafka.metadata.migration.KRaftMigrationDriver){code}
On the broker side, we see:
{code:java}
06:52:56,109] INFO [BrokerLifecycleManager id=6 isZkBroker=true] Unable to
register the broker because the RPC got timed out before it could be sent.
(kafka.server.BrokerLifecycleManager){code}
If we reinstall the same development cluster with kafka 3.4, using the exact
same steps provided by your migration documentation (only difference is using
{{inter.broker.protocol.version=3.4}} instead of
{{{}inter.broker.protocol.version=3.5{}}}), everything works as expected.
Updating to kafka 3.5/3.5.1 yields the same problems.
Testing is done on a three-node kafka cluster with a three-node zookeeper
ensemble and a three-node controller setup.
Besides our default configuration containing the active zookeeper hosts etc.,
this is what was added on the brokers:
{code:java}
# Migration
advertised.listeners=PLAINTEXT://kafka03:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
zookeeper.metadata.migration.enable=true
controller.quorum.voters=495@kafkactl01:9093,760@kafkactl02:9093,514@kafkactl03:9093
controller.listener.names=CONTROLLER
{code}
The main controller config looks like this:
{code:java}
process.roles=controller
node.id=495
controller.quorum.voters=495@kafkactl01:9093,760@kafkactl02:9093,514@kafkactl03:9093
listeners=CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
zookeeper.metadata.migration.enable=true
{code}
Both configs contain the identical {{zookeeper.connect}} settings, everything
is setup automatically so it should be identical on every run and we can
reliably reproduce migration success on kafka 3.4 and migration failure using
the same setup with kafka 3.5.
There are other issues mentioning problems with ApiVersions like KAFKA-15230 -
not quite sure if this is a duplicate of the underlying problem there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
