xiangfu0 commented on code in PR #20097:
URL: https://github.com/apache/pinot/pull/20097#discussion_r4417920241
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -554,6 +557,9 @@ private RebalanceResult doRebalance(TableConfig
tableConfig, RebalanceConfig reb
// NOTE: Monitor the segments to be moved from both the previous round and
this round to ensure the moved segments
// in the previous round are also converged.
List<String> oldSegmentsToMove = segmentsToMove;
+ // Force commit each partition's consuming segments at most once during
this rebalance. A force commit creates
+ // a consuming successor in the same partition, which must not be
committed again on retries or later steps.
+ IntSet forceCommittedPartitions = new IntOpenHashSet();
Review Comment:
This also changes behavior for multi-step moves, not just retries. The
default `minAvailableReplicas=1` makes a strict-replica-group move take 2+
steps. Master force-commits the successor again at each step: the new test's
`(1, 0, 0)` case gets 2 commits on base and 1 here. With this change, steps
after the first move the successor as CONSUMING. Its new replica then
re-consumes everything since the first step's commit, which can be a lot when
that step waited a long time for completed segments to download. That may be
the right trade-off, but it changes what `forceCommit` does by default, so a
rebalance owner should confirm it. If the intent is only to fix retries, the
previous head's clear-on-successful-move approach kept per-step commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
