KKcorps opened a new pull request, #19706: URL: https://github.com/apache/pinot/pull/19706
The segment data CRC covers only `.fwd` and `.dict` files. Pinot stores a null as the column's default value in the forward index, so a segment with a null and one with a real default value get the same data CRC. `BaseTableDataManager.hasSameCRC` passes on a data CRC match even when the full CRC differs, so a replica whose local build differs from the committed copy only in nulls keeps its copy at commit, restart and reload. Partial upsert tables can build such copies, because a merge treats a null differently from a stored value. This adds `.bitmap.nullvalue` to the data CRC file list. Segments with no nulls write no null vector file, so their data CRC does not change. Compatibility: existing segments keep the CRC stored in `creation.meta`. During a rolling upgrade, a segment with nulls built on an old and a new server gets two data CRCs, so the check falls back to the full CRC. That costs a one-time download only where the full CRC also differs, for example with a Lucene text index. Tested with a new `CrcUtilsTest` case that builds two segments differing only in one null. It fails without the fix (both CRCs 221905033) and passes with it. All 6 `CrcUtilsTest` tests pass, and spotless, checkstyle and license checks pass on `pinot-segment-local` and `pinot-core`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HrB4cnRFUk7PiheVNbyJzC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
