cypherean opened a new pull request, #19577: URL: https://github.com/apache/pinot/pull/19577
## Summary This PR addresses the silent segment-corruption window described in #19576 : Pinot could finish a segment write or download path after bytes had only reached the OS page cache. A crash or storage-sync failure in that window could leave a truncated or stale on-disk segment without an actionable write failure. It makes completion explicit and failure-aware: - Adds shared force-and-close utilities for FileChannels and memory-mapped buffers. - Updates segment index writers—including var-length, chunked, bitmap, mmap, and CLP forward indexes—to force mmap-backed writes separately before closing their channels. - Ensures retired bitmap mappings are forced during resize, including on platforms where explicit unmapping is unavailable. - Makes PinotDataBuffer.close() flush before release, still releases resources if flushing fails, and aggregates failures rather than leaking later buffers. - Makes native mmap flushing fail loudly when msync(MS_SYNC) reports an error. - Adds untarDurably() for segment installation paths. It fsyncs each extracted segment file before returning; streamed HTTP/PinotFS fetches and server/predownload extraction use it. - Keeps untarOneFile() non-durable because it is used for temporary, recomputable metadata extraction on LLC/control paths. - Adds focused tests for channel/mmap failure aggregation, mmap flush errors, bitmap close ordering, and durable extraction. The intended guarantee is durable file contents before Pinot regards a segment write/extraction as complete. It deliberately does not claim crash-atomic directory replacement or rename durability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
