[jira] [Updated] (KAFKA-16583) Update from 3.4.0 to 3.7.0 image write failed in Kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen updated KAFKA-16583: -- Fix Version/s: 3.8.0 3.7.1 > Update from 3.4.0 to 3.7.0 image write failed in Kraft mode > --- > > Key: KAFKA-16583 > URL: https://issues.apache.org/jira/browse/KAFKA-16583 > Project: Kafka > Issue Type: Bug > Components: kraft >Affects Versions: 3.7.0 >Reporter: HanXu >Assignee: HanXu >Priority: Blocker > Fix For: 3.8.0, 3.7.1 > > Original Estimate: 6h > Remaining Estimate: 6h > > How to reproduce: > 1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode; > 2. Create a topic with 1 partition; > 3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 > partition to Broker B; > 4. Upgrade Broker B to 3.7.0; > The Broker B will keep log the following error: > {code:java} > [2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled > error initializing new publishers > (org.apache.kafka.server.fault.LoggingFaultHandler) > org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been > lost because the following could not be represented in metadata version > 3.4-IV0: the directory assignment state of one or more replicas > at > org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) > at > org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) > at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) > at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) > at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) > at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) > at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) > at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) > at java.base/java.lang.Thread.run(Thread.java:840) > {code} > Bug: > - When reassigning partition, PartitionRegistration#merge will set the new > replicas with UNASSIGNED directory; > - But in metadata version 3.4.0 PartitionRegistration#toRecord only allows > MIGRATING directory; > {code:java} > if (options.metadataVersion().isDirectoryAssignmentSupported()) { > record.setDirectories(Uuid.toList(directories)); > } else { > for (Uuid directory : directories) { > if (!DirectoryId.MIGRATING.equals(directory)) { > options.handleLoss("the directory assignment state of one > or more replicas"); > break; > } > } > } > {code} > Solution: > - PartitionRegistration#toRecord allows both MIGRATING and UNASSIGNED -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16583) Update from 3.4.0 to 3.7.0 image write failed in Kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen updated KAFKA-16583: -- Priority: Blocker (was: Major) > Update from 3.4.0 to 3.7.0 image write failed in Kraft mode > --- > > Key: KAFKA-16583 > URL: https://issues.apache.org/jira/browse/KAFKA-16583 > Project: Kafka > Issue Type: Bug > Components: kraft >Affects Versions: 3.7.0 >Reporter: HanXu >Assignee: HanXu >Priority: Blocker > Original Estimate: 6h > Remaining Estimate: 6h > > How to reproduce: > 1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode; > 2. Create a topic with 1 partition; > 3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 > partition to Broker B; > 4. Upgrade Broker B to 3.7.0; > The Broker B will keep log the following error: > {code:java} > [2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled > error initializing new publishers > (org.apache.kafka.server.fault.LoggingFaultHandler) > org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been > lost because the following could not be represented in metadata version > 3.4-IV0: the directory assignment state of one or more replicas > at > org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) > at > org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) > at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) > at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) > at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) > at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) > at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) > at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) > at java.base/java.lang.Thread.run(Thread.java:840) > {code} > Bug: > - When reassigning partition, PartitionRegistration#merge will set the new > replicas with UNASSIGNED directory; > - But in metadata version 3.4.0 PartitionRegistration#toRecord only allows > MIGRATING directory; > {code:java} > if (options.metadataVersion().isDirectoryAssignmentSupported()) { > record.setDirectories(Uuid.toList(directories)); > } else { > for (Uuid directory : directories) { > if (!DirectoryId.MIGRATING.equals(directory)) { > options.handleLoss("the directory assignment state of one > or more replicas"); > break; > } > } > } > {code} > Solution: > - PartitionRegistration#toRecord allows both MIGRATING and UNASSIGNED -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16583) Update from 3.4.0 to 3.7.0 image write failed in Kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HanXu updated KAFKA-16583: -- Description: How to reproduce: 1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode; 2. Create a topic with 1 partition; 3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 partition to Broker B; 4. Upgrade Broker B to 3.7.0; The Broker B will keep log the following error: {code:java} [2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.4-IV0: the directory assignment state of one or more replicas at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) at java.base/java.lang.Thread.run(Thread.java:840) {code} Bug: - When reassigning partition, PartitionRegistration#merge will set the new replicas with UNASSIGNED directory; - But in metadata version 3.4.0 PartitionRegistration#toRecord only allows MIGRATING directory; {code:java} if (options.metadataVersion().isDirectoryAssignmentSupported()) { record.setDirectories(Uuid.toList(directories)); } else { for (Uuid directory : directories) { if (!DirectoryId.MIGRATING.equals(directory)) { options.handleLoss("the directory assignment state of one or more replicas"); break; } } } {code} Solution: - PartitionRegistration#toRecord allows both MIGRATING and UNASSIGNED was: How to reproduce: 1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode; 2. Create a topic with 1 partition; 3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 partition to Broker B; 4. Upgrade Broker B to 3.7.0; The Broker B will keep log the following error: {code:java} [2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.4-IV0: the directory assignment state of one or more replicas at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) at java.base/java.lang.Thread.run(Thread.java:840) {code} Bug: - When reassigning partition, PartitionRegistration#merge will set the new replicas with UNASSIGNED directory; - But in metadata version 3.4.0 PartitionRegistration#toRecord only allows MIGRATING directory; {code:java} if (options.metadataVersion().isDirectoryAssignmentSupported()) { record.setDirectories(Uuid.toList(directories)); } else { for (Uuid directory : directories) { if (!DirectoryId.MIGRATING.equals(directory)) { options.handleLoss("the directory assignment state of one or more replicas"); break;