[ https://issues.apache.org/jira/browse/KAFKA-16969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855219#comment-17855219 ]
Igor Soarez edited comment on KAFKA-16969 at 6/15/24 10:46 AM: --------------------------------------------------------------- This needs to target also 3.7.1 and 3.8.0, since KAFKA-16606 was backported there too. 😞  [~showuon] the error message says: Multiple log directories (aka JBOD) are not supported in the current MetadataVersion 3.6-IV2. Need 3.7-IV2 or higher A few questions: # Where is the value `3.6-IV2` coming from? # Did you use that value as the metadata version to format storage? # Did you run the features command to upgrade metadata.version before configuring JBOD? # Can you share the config and/or reproducing instructions?  [EDIT] I see now this is from the system tests. I'll look in there for these answers. was (Author: soarez): This needs to target also 3.7.1 and 3.8.0, since KAFKA-16606 was backported there too. 😞  [~showuon] the error message says: Multiple log directories (aka JBOD) are not supported in the current MetadataVersion 3.6-IV2. Need 3.7-IV2 or higher A few questions: # Where is the value `3.6-IV2` coming from? # Did you use that value as the metadata version to format storage? # Did you run the features command to upgrade metadata.version before configuring JBOD? # Can you share the config and/or reproducing instructions? > KRaft unable to upgrade to v3.7.1 and later when multiple log dir is set > ------------------------------------------------------------------------ > > Key: KAFKA-16969 > URL: https://issues.apache.org/jira/browse/KAFKA-16969 > Project: Kafka > Issue Type: Bug > Affects Versions: 3.8.0, 3.7.1 > Reporter: Luke Chen > Priority: Blocker > > After KAFKA-16606, we added validation metadata version for JBOD support. > This validation works well in isolated KRaft mode (i.e. separate > controller/broker node). But when in combined mode, this validation will let > the node fail to startup. The log will be like this: > {code:java} > [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the broker > metadata publishers to be installed (kafka.server.BrokerServer) > [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Finished waiting for the > broker metadata publishers to be installed (kafka.server.BrokerServer) > [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the controller > to acknowledge that we are caught up (kafka.server.BrokerServer) > [2024-06-15 16:00:45,621] INFO [MetadataLoader id=1] InitializeNewPublishers: > initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4 > (org.apache.kafka.image.loader.MetadataLoader) > [2024-06-15 16:00:45,621] ERROR Encountered metadata publishing fault: Broker > configuration does not support the cluster MetadataVersion > (org.apache.kafka.server.fault.LoggingFaultHandler) > java.lang.IllegalArgumentException: requirement failed: Multiple log > directories (aka JBOD) are not supported in the current MetadataVersion > 3.6-IV2. Need 3.7-IV2 or higher >   at scala.Predef$.require(Predef.scala:337) >   at > kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545) >   at > kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62) >   at > kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55) >   at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309) >   at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) >   at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) >   at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) >   at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) >   at java.base/java.lang.Thread.run(Thread.java:1623) > [2024-06-15 16:00:45,622] ERROR Encountered fatal fault: Unhandled error > initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4 > (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) > org.apache.kafka.server.fault.FaultHandlerException: Broker configuration > does not support the cluster MetadataVersion >   at scala.Predef$.require(Predef.scala:337) >   at > kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545) >   at > kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62) >   at > kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55) >   at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309) >   at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) >   at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) >   at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) >   at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) >   at java.base/java.lang.Thread.run(Thread.java:1623) > Caused by: java.lang.IllegalArgumentException: requirement failed: Multiple > log directories (aka JBOD) are not supported in the current MetadataVersion > 3.6-IV2. Need 3.7-IV2 or higher >   ... 10 more{code} >  > This will block combined node setting multiple log dirs upgrade to v3.7.1 or > later. -- This message was sent by Atlassian Jira (v8.20.10#820010)