[
https://issues.apache.org/jira/browse/HBASE-30093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109384#comment-18109384
]
mazhengxuan commented on HBASE-30093:
-------------------------------------
I took a look at the current code and I think the main question is what
maxRunningTime is supposed to cover.
At the moment, balanceTable sets the deadline before initCosts, the initial
computeCost, needsBalance, and the other setup work. On a busy machine, this
setup can use most of the available time, leaving little or no time for the
stochastic walk.
The proposed fix is small: keep startTime for metrics, but set stopRequestedAt
immediately before the stochastic loop. This would make maxRunningTime the time
budget for the search itself.
However, this does change the behavior. The total time spent in balanceTable
could become initialization time plus maxRunningTime. Do we consider
maxRunningTime a limit for the whole balanceTable call, or only for the
stochastic walk?
I also noticed that the original test now sets the runtime to 10 seconds
explicitly, so the old 250 ms failure is less likely on current master. The
timing issue in the balancer itself still exists, though.
If maxRunningTime is meant for the stochastic walk, I can prepare a small patch
that moves the deadline and adds a deterministic test with a controllable
EnvironmentEdge. The test can simulate slow initialization without relying on
sleep or CI load.
There is one related issue: NULL_ACTION paths continue before reaching the
deadline check at the bottom of the loop. I would prefer to handle that
separately unless we want to make the time limit strict as part of this issue.
> LoadBalancer related tests timed out
> ------------------------------------
>
> Key: HBASE-30093
> URL: https://issues.apache.org/jira/browse/HBASE-30093
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, test
> Reporter: Duo Zhang
> Priority: Major
>
> Sonnet 4.5(4.6?) summary
> TestStochasticLoadBalancerRegionReplicaSameHosts
> Root cause
> StochasticLoadBalancer.balanceTable() set
> cluster.setStopRequestedAt(startTime + maxRunningTime) at the very beginning
> of the method, before build ing costs (initCosts), the first computeCost,
> needsBalance, and other setup.
> So the entire maxRunningTime budget (e.g. 250 ms in
> StochasticBalancerTestBase) was counting initialization + search, not just
> the stochastic walk.
> On a slow or busy CI host, setup alone could take ~250 ms or more. Then when
> the walk started, isStopRequested() was already true (or became true af ter a
> single rejected move). The loop could exit with step still 0, no accepted
> moves, and balanceCluster returning null even though region repl icas were
> still colocated on the same host—so
> TestStochasticLoadBalancerRegionReplicaSameHosts failed. Locally, setup is
> usually only tens of ms, so the bug rarely shows up.
> Fix
> Set setStopRequestedAt only after initialization is done, immediately before
> the stochastic for loop, using a new timestamp (searchStartTime +
> maxRunningTime). That way maxRunningTime applies to the search phase only,
> which matches the intended meaning of the limit and avoids “no search time
> left” on loaded agents.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
