Jackie-Jiang opened a new pull request, #19069: URL: https://github.com/apache/pinot/pull/19069
## Summary Follow-up to #19054. That PR reduces the *cost* of the version-checked `IdealState` update losing the compare-and-set during a tier-relocation rebalance of a strict realtime table (upsert/dedup); this PR reduces the cost of *recovering* from a lost compare-and-set. Each batch of segment moves is applied with a version-checked `IdealState` update. On a continuously-ingesting table, consuming-segment commits bump the `IdealState` version between the read and the write, so the update loses the compare-and-set. Previously every lost compare-and-set fell back to the top of the convergence loop — another ExternalView-convergence wait (~hundreds of ms) plus a target recompute — so the rebalance could make very little progress while ingestion continued. ## Change When the rebalance moves **only tier segments** (the base placements of the partitions are unchanged — the same condition #19054 uses to skip the target recompute), a concurrent write that does not touch the segments this batch moves cannot invalidate the batch. So on a `ZkBadVersionException`: 1. Re-read the `IdealState`. 2. If the concurrent change is **disjoint** from the segments this batch moves, rebase the batch onto the latest `IdealState` (reapply the batch's moves, preserving the concurrent changes) and retry the compare-and-set in place — no ExternalView wait, no `rebalanceTable`. 3. On overlap, re-read failure, or exhausting a bounded number of attempts (`MAX_IDEAL_STATE_UPDATE_REBASE_ATTEMPTS`), fall back to the convergence loop as before. On a successful rebase, the target assignment is refreshed from the adopted `IdealState` so the `currentAssignment.equals(targetAssignment)` convergence check stays well-defined when a rebase pulls in concurrently added (e.g. new consuming) or removed (e.g. retention) segments. Net effect: disjoint version churn is absorbed in place, so a tier relocation makes progress against a steady stream of consuming-segment commits instead of repeatedly restarting. ## Notes - The rebase is gated on the same "moves only tier segments" check as #19054, so when the rebalance moves base placements it conservatively keeps the existing fall-back behavior. - Reviewers: the observer stats handling on a successful rebase (`onRollback` is skipped since the batch did land) is the subtlest part and the best target for a unit test that injects a version bump between the read and the compare-and-set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
