[jira] [Created] (KAFKA-16892) TopicsDelta#localChanges get empty deletes when load from snapshot

[HanXu (Jira)]

HanXu created KAFKA-16892:
-

 Summary: TopicsDelta#localChanges get empty deletes when load from 
snapshot
 Key: KAFKA-16892
 URL: https://issues.apache.org/jira/browse/KAFKA-16892
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.7.0, 3.6.0, 3.5.0, 3.4.0
Reporter: HanXu
Assignee: HanXu


MetadataLoader#loadSnapshot get empty TopicsDelta.localChanges(brokerId).deletes

https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicsDelta.java#L71C1-L76C6


Bug:
1. 当从 Snapshot 中生成 MetadataDelta 时，TopicsDelta 会首先 #replay(TopicRecord)；
2. 然而 TopicsDelta 直接创建了一个无 Partitions 的 TopicDelta，而不是使用之前的 TopicImage；
3. 因为 TopicDelta.image 没有任何 Partitions，导致 TopicDelta#localChanges 生成 deletes 
列表时，返回了 empty deletes；

https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicsDelta.java#L71C1-L76C6

https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicDelta.java#L142-L148



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16583) Update from 3.4.0 to 3.7.0 image write failed in Kraft mode

[HanXu (Jira)]

HanXu created KAFKA-16583:
-

 Summary: Update from 3.4.0 to 3.7.0 image write failed in Kraft 
mode
 Key: KAFKA-16583
 URL: https://issues.apache.org/jira/browse/KAFKA-16583
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.7.0
Reporter: HanXu
Assignee: HanXu


How to reproduce:
1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode;
2. Create a topic with 1 partition;
3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 
partition to Broker B;
4. Upgrade Broker B to 3.7.0;

The Broker B will keep log the following error:
{code:java}
[2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled 
error initializing new publishers 
(org.apache.kafka.server.fault.LoggingFaultHandler)
org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been 
lost because the following could not be represented in metadata version 
3.4-IV0: the directory assignment state of one or more replicas
at 
org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94)
at 
org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391)
at org.apache.kafka.image.TopicImage.write(TopicImage.java:71)
at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84)
at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155)
at 
org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295)
at 
org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
at java.base/java.lang.Thread.run(Thread.java:840)
{code}

Bug:
- When reassigning partition, PartitionRegistration#merge will set the new 
replicas with UNASSIGNED directory;
- But in metadata version 3.4.0 PartitionRegistration#toRecord only allows 
MIGRATING directory；
{code:java}
if (options.metadataVersion().isDirectoryAssignmentSupported()) {
record.setDirectories(Uuid.toList(directories));
} else {
for (Uuid directory : directories) {
if (!DirectoryId.MIGRATING.equals(directory)) {
options.handleLoss("the directory assignment state of one 
or more replicas");
break;
}
}
}
{code}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)