Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test ...................................................................... KUDU-3789 Deflake auto_leader_rebalancer-test Under TSAN this test suite was very flaky (about 488 of 1000 runs failed at one baseline). The causes were all test-side timing assumptions on a cluster whose leadership keeps moving, not product bugs. Fixed retry loops (run N rounds, then assert balanced) raced the cluster's ongoing leadership moves. Replaced them with a RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over a TSAN-aware time budget via AssertEventually. AddTserver and RestartTserver used 59 tablets. That many raft groups move leadership around through natural elections and keep the cluster from ever reaching an exactly balanced snapshot. Note that stopping the replica rebalancer to remove that movement makes things worse: on a static cluster the leader rebalancer gets stuck moving the same leaders back and forth and never balances, so the replica rebalancer is left running. Placement-sensitive tests (MultiTableLeaderBalance, FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by extra leader elections when TSAN CPU starvation exceeded the default leader failure timeout. Relaxed the ignored soft-deleted table check in FilterSoftDeletedTableTest to "not fully balanced" rather than requiring every leader count to differ from 3, which natural elections could trip. Rewrote TestMaintenanceMode to arrange the imbalance directly with MakeLeaderDistribution and assert the rebalancer never places a leader on the maintenance-mode tserver, instead of restarting the tserver and racing on election timing. Test-only. No production behavior change. Change-Id: I00224fddfead0fb462e793f085d935b597792aaa Reviewed-on: http://gerrit.cloudera.org:8080/24591 Tested-by: Alexey Serbin <[email protected]> Reviewed-by: Alexey Serbin <[email protected]> --- M src/kudu/master/auto_leader_rebalancer-test.cc 1 file changed, 158 insertions(+), 167 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa Gerrit-Change-Number: 24591 Gerrit-PatchSet: 5 Gerrit-Owner: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]>
