dlscjf151 opened a new issue, #20172: URL: https://github.com/apache/pinot/issues/20172
## Reported problem When a REALTIME dedup table is rebalanced, or its replication factor is increased, a newly assigned server can have no persisted dedup TTL watermark for the assigned partition. During its initial segment loading, historical primary keys accumulate in the in-memory dedup map and the server can run out of heap before recovery completes. This report is scoped to **dedup**. Upsert recovery has additional validDocIds/snapshot requirements and is deliberately out of scope. ## Source analysis The analysis below concerns the default dedup implementation with `metadataTTL > 0`, at Apache Pinot master commit `0a53497fddfc61220feabc5b8f62516f4183525f`: - `BasePartitionDedupMetadataManager` reads its partition watermark from a file in the server-local table index directory. `TTL_WATERMARK_NOT_SET` is `0`; when the file is absent, `WatermarkUtils.loadWatermark()` returns that default. - `skipSegmentOutOfTTL()` compares a segment's maximum dedup-column value against the watermark currently known to this manager. If older segments are processed before a sufficiently recent watermark is available, those segments can pass the TTL check. - `addSegment()` / `preloadSegment()` then iterate the segment's records and populate `_primaryKeyToSegmentAndTimeMap`. The watermark advances after the segment's metadata is registered. - Expired keys are removed by a separate `removeExpiredPrimaryKeys()` operation. A watermark advancing during recovery does not, by itself, evict all historical keys already inserted into the map. This makes cold bootstrap different from restarting a server that still has a usable local watermark. The transient metadata footprint can depend on historical segment load order and cleanup timing, rather than only the intended dedup window. Segments that span both old and recent data also need consideration, since segment-level pruning alone does not exclude every expired row. Source references: - [Dedup watermark initialization](https://github.com/apache/pinot/blob/0a53497fddfc61220feabc5b8f62516f4183525f/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java#L76-L96) - [Segment registration](https://github.com/apache/pinot/blob/0a53497fddfc61220feabc5b8f62516f4183525f/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java#L194-L218) - [TTL check and record iteration](https://github.com/apache/pinot/blob/0a53497fddfc61220feabc5b8f62516f4183525f/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java#L245-L280) - [In-memory key registration](https://github.com/apache/pinot/blob/0a53497fddfc61220feabc5b8f62516f4183525f/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/ConcurrentMapPartitionDedupMetadataManager.java#L32-L85) ## Reproduction scenario to validate 1. Configure a REALTIME dedup table with positive `metadataTTL`, retaining historical segments over many dedup windows. 2. Ingest enough distinct keys for loading the historical key set to exceed a new server's heap, while the active dedup window itself fits in memory. 3. Rebalance a partition onto a fresh server, or increase replication so the partition is assigned to a server without its local dedup watermark file. 4. Observe the initial dedup primary-key count and heap while historical segments are loaded. Older-first and concurrent loading orders should be included. The server memory failure above is an operational observation. The source analysis identifies a matching recovery path; a deterministic full-cluster OOM reproduction on the cited master commit has **not** been run as part of this report. The steps above are a validation scenario, not a claimed completed test. ## Expected behavior A fresh server should be able to recover dedup metadata without first materializing the retained historical primary-key set. With a fixed dedup window that fits in memory, increasing the amount of expired history should not require a correspondingly larger dedup map during recovery. Historical segments must remain available for queries: `metadataTTL` limits dedup metadata, not table data retention. Duplicate handling and ordered stream replay must remain correct. ## Possible direction Establish a safe, partition-specific recovery watermark before historical PK registration, then avoid retaining expired dedup metadata during recovery. The boundary must correspond to the actual configured `dedupTimeColumn` and its units; using wall-clock time, a different partition's maximum, or an unrelated table-time `endTimeMs` would not be safe. This must cover both preload and normal ONLINE loading, including a newly assigned partition on a table manager that is already running. The initialization barrier must preserve replay ordering and must not reintroduce the watermark/cleanup race fixed by #19512. This is a proposed direction, not an implemented or validated fix. ## Related work - #10571: general dedup memory footprint. - #14137: dedup TTL watermark persistence/restoration, which still depends on a local watermark file. - #14187: faster dedup preloading for restart/rebalance. - #17254: periodic dedup metadata cleanup; its [discussion](https://github.com/apache/pinot/pull/17254#issuecomment-3577189528) also raises watermark loss during server replacement/bootstrap, in an upsert comparison. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
