[
https://issues.apache.org/jira/browse/KUDU-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103895#comment-18103895
]
ASF subversion and git services commented on KUDU-3789:
-------------------------------------------------------
Commit bea38fae5e1a94f4a83acd90a398a50f4aff46f9 in kudu's branch
refs/heads/master from Gabriella Lotz
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=bea38fae5 ]
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Note that stopping
the replica rebalancer to remove that movement makes things worse: on
a static cluster the leader rebalancer gets stuck moving the same
leaders back and forth and never balances, so the replica rebalancer
is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout.
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Reviewed-on: http://gerrit.cloudera.org:8080/24591
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
> Investigate and fix flaky tests
> -------------------------------
>
> Key: KUDU-3789
> URL: https://issues.apache.org/jira/browse/KUDU-3789
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> # In auto_rebalancer-test.cc
> # In auto_leader_rebalancer-test.cc
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
