Hi Dethmix, > The interesting thing is that this doesn't happen if I rollback inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is able to start and read all the data from the __consumer_offsets partitions with metadata version 3. The question is that expected behavior?
I think you should understand that even if it works well so far, we can't guarantee there will be no unexpected issue happening later. I think the document clearly said that downgrading is not supported, please try not to do that. Thanks. Luke On Wed, Aug 14, 2024 at 3:36 AM dethmix dethmix <deth...@gmail.com> wrote: > Hi all, > > I do understand that everybody is excited about the Kafka 3.8 release, but > this question is still actual :) > Any updates. > > Thanks, > > On Tue, Jul 16, 2024 at 2:54 PM dethmix dethmix <deth...@gmail.com> wrote: > > > Hi all, > > > > I am working on upgrading the Kafka cluster from the version 2.0.1 to > > 3.5.2. The main concern in that update is passing version 2.1 which has > the > > following note in the upgrade document: > > ``` > > Note that 2.1.x contains a change to the internal schema used to store > > consumer offsets. Once the upgrade is complete, it will not be possible > to > > downgrade to previous versions. > > ``` > > In the rolling upgrade section the documentation says the following: > > ``` > > Restart the brokers one by one for the new protocol version to take > > effect. Once the brokers begin using the latest protocol version, it will > > no longer be possible to downgrade the cluster to an older version. > > ``` > > So, as soon as inter.broker.protocol.version is set to 2.1+ you should > not > > be able to rollback to the old version. > > I did reproduce the issue with the rollback with the following steps: > > 1. Start cluster with Kafka 2.0.1 binaries and > > inter.broker.protocol.version=2.0. Produce some messages in the test > topic > > 2. Restart cluster with Kafka 3.5.2 binaries > > and inter.broker.protocol.version=3.5. Consume messages. > > 3. Restart cluster once again with Kafka 2.0.1 binaries and > > inter.broker.protocol.version=2.0. During start you will get error like > > this: > > ``` > > ERROR [GroupMetadataManager brokerId=0] Error loading offsets from > > __consumer_offsets-43 (kafka.coordinator.group.GroupMetadataManager) > > org.apache.kafka.common.KafkaException: Unknown group metadata version 3 > > ``` > > > > The interesting thing is that this doesn't happen if I rollback > > inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster > is > > able to start and read all the data from the __consumer_offsets > partitions > > with metadata version 3. > > The question is that expected behavior? Below are the steps that I have > > used: > > 1. Start cluster with Kafka 2.0.1 binaries and > > inter.broker.protocol.version=2.0. Produce some messages in the test > topic > > 2. Restart cluster with Kafka 3.5.2 binaries > > and inter.broker.protocol.version=3.5. Consume messages. > > 3. Restart cluster once again with Kafka 3.5.2 binaries and > > inter.broker.protocol.version=2.0. Cluster starts properly and is able to > > read partitions from the __consumer_offsets topic. > > > > I did double check Kafka startup logs and make sure that it uses > > configured inter.broker.protocol.version versions. > > if this is okay, then I guess that can be used as a rollback in case of > > problems after switching to the inter.broker.protocol.version 3.X > > > > Thanks in advance, > > Dmytro > > >
