KKcorps opened a new pull request, #19728: URL: https://github.com/apache/pinot/pull/19728
Comparing validDocIds snapshot files across replicas to catch divergence gives false alarms today. A snapshot pass doesn't block segment adds, replaces or removes, and SRT, FIT or compaction uploads land on each replica at a different time. This is a follow-up to #19637, but it doesn't depend on it. After each snapshot pass, the server now records a CRC of the sealed segment names, their count, and whether the pass was a consistent cut. A pass is consistent only if no segment operation was in progress when it started or started before it finished, every changed segment got its file written, and no earlier consuming segment was still waiting to be sealed. Two counters around the four segment operations track this without taking a lock. `validDocIdsMetadata` returns it as `snapshotPass` for the SNAPSHOT types, so a caller compares two replicas only when both are consistent and have the same names CRC. One limit: on `ALLOW_ALWAYS` tables a replica often starts the next consuming segment before the previous one is sealed, so those passes report not consistent and get compared less often. Compatibility is a new response field only, and `ValidDocIdsMetadataInfo` already ignores unknown fields. Tested with three new `BasePartitionUpsertMetadataManagerTest` cases (matching replicas, an add that overlaps the pass, a failed snapshot write plus an unsealed consuming segment) and a `TablesResourceTest` assertion. The overlap case fails when the in-progress check is removed. Spotless, checkstyle and license checks pass on both modules. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HrB4cnRFUk7PiheVNbyJzC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
