sidharthdaga commented on PR #22585:
URL: https://github.com/apache/kafka/pull/22585#issuecomment-4930182210
Measured the cost of calling timeIndex().sanityCheck() during segment
loading by timing
full log initialization (UnifiedLog.create()) on a directory of pre-existing
segments, with
and without the check. The segments are minimal (empty log files,
single-entry trimmed time
indexes), so the delta between runs isolates the cost of materializing the
lazy time index.
Calling timeIndex() on a segment for the first time triggers LazyIndex to
construct the
TimeIndex, which mmaps the file and reads the last entry from the mapped
region. sanityCheck()
itself only compares cached values. The entire cost is in materializing the
lazy index.
The only variable between runs is the presence of the
timeIndex().sanityCheck() call. Each number in the table is the average of 5
runs. Results (MacBook Pro M3, SSD):
Segments Without check With check Delta
1,000 761 ms 960 ms +199 ms
5,000 3,417 ms 4,570 ms +1,153 ms
10,000 6,469 ms 8,860 ms +2,391 ms
The cost scales linearly at ~0.24 ms per segment on this hardware. The
offset index is not
materialized (stays lazy), so this is half the I/O that KAFKA-7283
originally removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
