Mickael Maison created KAFKA-20979:
--------------------------------------
Summary: Potential data loss caused by time index truncation
Key: KAFKA-20979
URL: https://issues.apache.org/jira/browse/KAFKA-20979
Project: Kafka
Issue Type: Bug
Reporter: Mickael Maison
Assignee: Mickael Maison
On broker startup, I see these lines for many partitions:
{noformat}
[LocalLog partition=TOPIC-0, dir=LOG_DIR/TOPIC-0] Rolled new log segment at
offset 83461 in 0 ms.
[UnifiedLog partition=TOPIC-0, dir=LOG_DIR] Incremented log start offset to
83461 due to segment deletion
[UnifiedLog partition=TOPIC-0, dir=LOG_DIR] Deleting segment
LogSegment(baseOffset=83454, size=4483, lastModifiedTime=1786538669095,
largestRecordTimestamp=0) due to log retention time 604800000ms breach based on
the largest record timestamp in the segment
{noformat}
It's deleting many segments due to the retention limits. This has happened
multiple times in the past few weeks. Using DumpLogSegments (on followers and
by using file.delete.delay.ms to delay the actual deletion) I've been able to
confirm that the retention time was not breached and that all records have
valid timestamps (I first suspected some records with bad timestamps).
In all cases it happened on a clean shutdown.
Being unable to explain why a broker would incorrectly compute
largestRecordTimestamp=0 I looked at the code and I think this behavior could
be caused by invalid time index truncation.
During clean shutdown, LogManager calls flush(true) then close() on each log.
The flush() fsyncs the time index file at its pre-allocated size (10MB by
default, zero-filled). The close() truncates the file to its actual size in
AbstractIndex.resize() which calls RandomAccessFile.setLength(), but never
fsyncs the truncation.
If the broker restarts before the OS flushes the file metadata, the time index
could revert to its fsynced pre-allocated state. The zero-filled tail is then
read as valid entries with timestamp=0. Then when time-based retention
triggers, it deletes the segment, incorrectly deleting valid, recent records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
