[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r572510600
##
File path: core/src/main/scala/kafka/log/Log.scala
##
@@ -256,7 +256,8 @@ class Log(@volatile private var _dir: File,
val topicPartition: TopicPartition,
val producerStateManager: ProducerStateManager,
logDirFailureChannel: LogDirFailureChannel,
- private val hadCleanShutdown: Boolean = true) extends Logging with
KafkaMetricsGroup {
+ private val hadCleanShutdown: Boolean = true,
+ val keepPartitionMetadataFile: Boolean = true) extends Logging with
KafkaMetricsGroup {
Review comment:
Yes, looks good to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r572474952
##
File path: core/src/main/scala/kafka/log/Log.scala
##
@@ -256,7 +256,8 @@ class Log(@volatile private var _dir: File,
val topicPartition: TopicPartition,
val producerStateManager: ProducerStateManager,
logDirFailureChannel: LogDirFailureChannel,
- private val hadCleanShutdown: Boolean = true) extends Logging with
KafkaMetricsGroup {
+ private val hadCleanShutdown: Boolean = true,
+ val keepPartitionMetadataFile: Boolean = true) extends Logging with
KafkaMetricsGroup {
Review comment:
Could we add the new param to the javadoc? In the javadoc, it would be
useful to explain a bit how this helps with re-upgrade.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r572450889
##
File path: core/src/test/scala/unit/kafka/log/LogManagerTest.scala
##
@@ -217,6 +217,7 @@ class LogManagerTest {
}
assertTrue(log.numberOfSegments > 1, "There should be more than one
segment now.")
log.updateHighWatermark(log.logEndOffset)
+log.partitionMetadataFile.get.write(Uuid.randomUuid())
Review comment:
That's what I am thinking. It's kind of weird to add the new file just
to get the existing math work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r572450889
##
File path: core/src/test/scala/unit/kafka/log/LogManagerTest.scala
##
@@ -217,6 +217,7 @@ class LogManagerTest {
}
assertTrue(log.numberOfSegments > 1, "There should be more than one
segment now.")
log.updateHighWatermark(log.logEndOffset)
+log.partitionMetadataFile.get.write(Uuid.randomUuid())
Review comment:
That's what I am thinking. It's kind of weird to add the new file just
to get the existing math works.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r57205
##
File path: core/src/test/scala/unit/kafka/log/LogManagerTest.scala
##
@@ -217,6 +217,7 @@ class LogManagerTest {
}
assertTrue(log.numberOfSegments > 1, "There should be more than one
segment now.")
log.updateHighWatermark(log.logEndOffset)
+log.partitionMetadataFile.get.write(Uuid.randomUuid())
Review comment:
Got it. Since the partitionMetadataFile is now created on demand,
perhaps we could just change the math on the expected number of files?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10041: MINOR: Prevent creating partition.metadata until ID can be written
junrao commented on a change in pull request #10041:
URL: https://github.com/apache/kafka/pull/10041#discussion_r572399914
##
File path: core/src/main/scala/kafka/log/Log.scala
##
@@ -341,10 +342,15 @@ class Log(@volatile private var _dir: File,
producerStateManager.removeStraySnapshots(segments.values().asScala.map(_.baseOffset).toSeq)
loadProducerState(logEndOffset, reloadFromCleanShutdown = hadCleanShutdown)
-// Recover topic ID if present
+// Delete partition metadata file if the version does not support topic
IDs.
+// Recover topic ID if present and topic IDs are supported
partitionMetadataFile.foreach { file =>
- if (!file.isEmpty())
-topicId = file.read().topicId
+ if (file.exists()) {
+if (!usesTopicId)
+ file.delete()
Review comment:
Does partitionMetadataFile need to be of Some? It seems that we can just
always instantiate the object.
##
File path: core/src/main/scala/kafka/log/Log.scala
##
@@ -256,7 +256,8 @@ class Log(@volatile private var _dir: File,
val topicPartition: TopicPartition,
val producerStateManager: ProducerStateManager,
logDirFailureChannel: LogDirFailureChannel,
- private val hadCleanShutdown: Boolean = true) extends Logging with
KafkaMetricsGroup {
+ private val hadCleanShutdown: Boolean = true,
+ val usesTopicId: Boolean = true) extends Logging with
KafkaMetricsGroup {
Review comment:
Could we add the new param to javadoc? Also, will
keepPartitionMetdataFile be better than usesTopicId?
##
File path: core/src/main/scala/kafka/server/PartitionMetadataFile.scala
##
@@ -91,11 +91,10 @@ class PartitionMetadataFile(val file: File,
private val lock = new Object()
private val logDir = file.getParentFile.getParent
-
- try Files.createFile(file.toPath) // create the file if it doesn't exist
- catch { case _: FileAlreadyExistsException => }
-
def write(topicId: Uuid): Unit = {
+try Files.createFile(file.toPath) // create the file if it doesn't exist
Review comment:
Do we need to create the file first? It seems that later on we always
rename the temp file to this one.
##
File path: core/src/test/scala/unit/kafka/log/LogManagerTest.scala
##
@@ -217,6 +217,7 @@ class LogManagerTest {
}
assertTrue(log.numberOfSegments > 1, "There should be more than one
segment now.")
log.updateHighWatermark(log.logEndOffset)
+log.partitionMetadataFile.get.write(Uuid.randomUuid())
Review comment:
Is this needed? It seems Log never reads UUID? Ditto below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
