tarun11Mavani opened a new pull request, #18733: URL: https://github.com/apache/pinot/pull/18733
### Problem When `upsertMetadataTTL` is configured, expired metadata entries are only cleaned up at segment commit time (which can be hours apart for low-throughput tables). Records arriving after TTL expiry are still compared against stale metadata — a record with a lower comparison value gets rejected as out-of-order even though the existing entry is expired and should be irrelevant. ### Fix Add an inline TTL expiry check in `doAddRecord()`. When the existing record's comparison value is below the TTL threshold (`largestSeenComparisonValue - metadataTTL`), skip the comparison entirely: - The new record always wins regardless of its comparison value - The old doc is invalidated via `replaceDocId` (no duplicate queryable rows) - The map entry is replaced with a fresh `RecordLocation` Also guard the partial upsert merge in `doUpdateRecord()` — skip reading the previous row when it's expired, preventing stale data from being merged into the new record. ### Changes - `ConcurrentMapPartitionUpsertMetadataManager.doAddRecord()` — TTL check before comparison - `ConcurrentMapPartitionUpsertMetadataManager.doUpdateRecord()` — TTL guard on partial upsert merge - Test: expired metadata is skipped, lower comparison value accepted, old doc invalidated ### Not changed - `ConcurrentMapPartitionUpsertMetadataManagerForConsistentDeletes` — `metadataTTL` and `enableDeletedKeysCompactionConsistency` are mutually exclusive, so the check can never fire there - `removeExpiredPrimaryKeys()` — lazy cleanup at segment commit continues as-is -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
