cypherean opened a new issue, #19576:
URL: https://github.com/apache/pinot/issues/19576
## Summary
We observed IllegalStateException: Unknown type N /
IndexOutOfBoundsException thrown by RoaringBitmap's RangeBitmap when queries
touch certain segments' range indexes. Forensic byte-diffing of affected
segments (comparing corrupted local replica vs. a clean replica vs. deep store)
found a consistent signature:
- A contiguous 128KiB-aligned storage block in the .range_index file.
- The first 4KB/16KB sub-page of that block is correct; the remaining
sub-pages are exact byte-for-byte copies of the first sub-page.
- Deep store always has the clean bytes -- corruption is local-disk-only.
- Corruption is non-deterministic given identical input.
- Both committers (segment builders) and followers (segment downloaders) are
independently affected.
## Root cause theory
The full write path (BitSlicedRangeIndexCreator, MmapFileWriter,
RoaringBitmap's Appender/container serializers) was read end-to-end with no
application-level bug found. The duplication pattern (exact copy of the
preceding 128KiB block) is most consistent with a storage/virtualization-layer
fault below the page cache, but this couldn't be conclusively confirmed.
## The confirmed Pinot-side gap
Independent of root cause: nothing in the segment write path ever confirms
bytes reached physical storage before a checksum is computed over them, and
nothing re-verifies bytes at load time.
1. Every segment index writer closed its output via a bare
FileChannel.close()/unmap, relying on the OS's own lazy writeback schedule. A
write that never reached disk (disk-full, storage fault) succeeded silently.
2. hasSameCRC() only compares two already-recorded checksum values; it never
re-hashes live on-disk bytes, and doesn't run at all on a plain committer's own
commit transition.
3. Because RoaringBitmap's
RangeBitmap.SingleEvaluation.evaluateHorizontalSliceRange() only fully
evaluates a bit-plane container when a query predicate happens to route into
the "evaluate" branch, corruption can sit silent and undetected indefinitely,
dependent on query predicate values rather than whether the segment is queried
at all.
Related prior issue: #11368 ("Pinot Does not FSync and Relies on FD Close")
raised the same issue against BaseJsonIndexCreator.java specifically, and was
closed with a fix there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
