shuturmurgh opened a new issue, #20230: URL: https://github.com/apache/pinot/issues/20230
## Problem A writer whose lineage entry has been reverted can still upload segments listed in that entry's `segmentsTo`. This happens when a stalled or zombie attempt still has an upload in flight, or retries one. The upload path doesn't consult segment lineage. 1. Writer A starts `E_A` (`IN_PROGRESS`) and begins uploading `a0`. 2. `E_A` is reverted while A is stalled. The revert deletes the `a0` segments that are already present. 3. On the next retention run, none of `E_A`'s `segmentsTo` are left in the table, so retention removes `E_A`. That happens within `controller.retention.frequencyPeriod`, 6 h by default. 4. A's late upload of `a0` lands. No lineage entry hides it anymore, so it's served as a normal segment. The result is duplicate or partial data. If the upload lands before step 3, it stays hidden and the next retention run deletes it. The hazard is only an upload that lands after the entry is gone. The same applies to stale `IN_PROGRESS` entries, which retention removes after `lineageEntryCleanupRetentionPeriod` (1 day by default). Entries get reverted while their writer is still alive in several ways: - **`startReplaceSegments` with `forceCleanup=true`.** It reverts an overlapping `IN_PROGRESS` entry. `forceCleanup=true` is hardcoded in `BaseMultipleSegmentsConversionExecutor` and used by `MaterializedViewTask`. - **An explicit revert by an operator or orchestrator.** - **Takeover in the partition-key lock proposal.** ## Why existing safeguards miss it - **Uploads aren't checked against lineage.** A segment is visible unless lineage hides it. - **The revert's cleanup and retention only delete segments that exist when they run.** - **Rejecting names found in a `REVERTED` entry wouldn't help.** That check stops working as soon as retention removes the entry. ## Proposal Admit lineage-managed uploads only through an active entry. This is opt-in. - **The uploader passes the lineage entry ID with each segment upload**, as a new header or query parameter. - **The controller accepts the upload only if that entry exists, is `IN_PROGRESS`, and lists the segment in `segmentsTo`.** Otherwise it rejects the upload with 409. - **Apply the check to every upload path:** new and refresh uploads, single and batch. In practice that means `ZKOperator`, before it splits into new-segment and existing-segment handling. - **Optionally, make it atomic.** Create the segment ZK metadata in the same ZK multi-op that checks the lineage ZNode version read during the check. Older controllers ignore the new parameter, so writers that rely on the check must first verify that the controller supports it. **Alternative / defense in depth:** in retention, keep `REVERTED` entries for a configurable TTL instead of removing them as soon as their segments are gone. Stale `IN_PROGRESS` entries would be converted to `REVERTED` and kept the same way. Late uploads during the TTL stay hidden and get reaped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
