KKcorps opened a new pull request, #20204: URL: https://github.com/apache/pinot/pull/20204
#19832 caps the TTL watermark it persists, so a segment with a stale validDocIds snapshot stays within TTL and gets added in full on restart. But a watermark file written before that change is not capped. After an upgrade, a segment with a stale snapshot can still look out of TTL against it, get skipped, and keep duplicate rows visible on that replica. This PR writes a version marker after the watermark when it is capped. On load, a watermark file without the marker still sets the in-memory watermark for expiring keys, but no segment is skipped as out of TTL against it. Only the first restart after the upgrade adds every segment in full. The next snapshot writes a marked file, and later restarts skip out-of-TTL segments as before. Old readers only read the leading double, so a rollback still works. Tested with a new case in `ConcurrentMapPartitionUpsertMetadataManagerTest` plus the existing upsert TTL and snapshot restart tests. The paired StarTree PR adds a regression test that loads a real watermark and snapshot written by the old writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
