Hello Marton Greber, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24591
to look at the new patch set (#4).
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Note that stopping
the replica rebalancer to remove that movement makes things worse: on
a static cluster the leader rebalancer gets stuck moving the same
leaders back and forth and never balances, so the replica rebalancer
is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout.
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
---
M src/kudu/master/auto_leader_rebalancer-test.cc
1 file changed, 158 insertions(+), 167 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/91/24591/4
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Gabriella Lotz <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
