jmalizia-branch commented on a change in pull request #8936:
URL: https://github.com/apache/kafka/pull/8936#discussion_r449181257
##########
File path: core/src/test/scala/unit/kafka/log/TimeIndexTest.scala
##########
@@ -148,5 +148,39 @@ class TimeIndexTest {
idx.close()
}
-}
+ /**
+ * In certain cases, index files fail to have their pre-allocated 0 bytes
trimmed from the tail
+ * when a new segment is rolled. This causes a silent failure at the next
startup where all retention
+ * windows are breached purging out data whether or not the window was
really breached.
+ * KAFKA-10207
+ */
+ @Test
+ def testLoadingUntrimmedIndex(): Unit = {
+ // A larger index size must be specified or the starting offset will round
down
+ // preventing this issue from being reproduced. Configs default to 10mb.
+ val max1MbEntryCount = 100000
+ // Create a file that will exist on disk and be removed when we are done
+ val file = nonExistantTempFile()
+ file.deleteOnExit()
+ // create an index that can have up to 100000 entries, about 1mb
+ var idx2 = new TimeIndex(file, baseOffset = 0, max1MbEntryCount * 12)
+ // Append less than the maximum number of entries, leaving 0 bytes padding
the end
+ for (i <- 1 until max1MbEntryCount)
+ idx2.maybeAppend(i, i)
+
+ idx2.flush()
+ // jvm 1.8.0_191 fails to always flush shrinking resize to zfs disk
Review comment:
I believe it is known and since resolved, but I don't think I have
identified the exact issue just yet. The closest lead I've been able to find is
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8202261 which appears
to have been resolved around 8u202
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
