KKcorps commented on PR #20204: URL: https://github.com/apache/pinot/pull/20204#issuecomment-6765159150
closing this, we're consciously leaving the gap. here's what we learned. the bug: an older release saves the TTL watermark uncapped, even when the snapshot pass skipped a segment because its lock was busy. if the server restarts before the next pass, that segment looks out of TTL, gets skipped, and keeps its stale bitmap, so duplicates stay visible on that replica. we tried two things. distrusting an untagged watermark fixes it, but every metadataTTL table then loads every segment past TTL in full on the first restart after upgrade, and those keys only expire at the next consuming commit. tagging the file and counting skips against an untagged one is just noise, because every old file is untagged. there's no cheap detector either, since the snapshot pass only rewrites changed segments, so stale and unchanged snapshots look the same on disk. we stopped because it's rare, it already exists in released versions, and #19832 caps the watermark so new cases can't happen. if it comes back, what's left is an escape hatch config, deleting the watermark file only on affected servers, or a one time background key check after load. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
