[
https://issues.apache.org/jira/browse/HBASE-30547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-30547:
-----------------------------------
Labels: pull-request-available (was: )
> Master active time can go backwards after a failover to a master whose clock
> is behind
> --------------------------------------------------------------------------------------
>
> Key: HBASE-30547
> URL: https://issues.apache.org/jira/browse/HBASE-30547
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 4.0.0-alpha-1, 3.0.0-beta-2, 2.6.1, 2.5.11
> Reporter: Boyi Dai
> Priority: Major
> Labels: pull-request-available
>
> Since HBASE-28690, remote procedures send the active time of the master that
> started them, and {{MasterRpcServices.throwOnOldMaster}} rejects a report
> whose initiating active time is greater than the receiving master's own, with
> {{MasterNotRunningException("Another master is active")}}. The active time is
> taken from the wall clock in {{HMaster.finishActiveMasterInitialization}}. If
> the next active master's clock is behind the previous one's by more than the
> time between the two masters becoming active, its active time is lower than
> the previous master's, and it rejects reports for procedures the previous
> master started.
> For a region open this does not recover on its own:
> # The active master M0 dispatches an {{OpenRegionProcedure}} with its active
> time T0, and the region server opens the region and reports OPENED with T0.
> # M0 loses leadership before handling the report, and M1 becomes active with
> an active time T1 < T0 because its clock is behind.
> # {{HRegionServer.reportRegionStateTransition}} builds the request once and
> retries it with T0, and M1 rejects every attempt.
> {{MasterNotRunningException}} is not one of the exceptions it backs off on,
> so it retries immediately.
> # M1 dispatches the open again with the same procedure id, and the region
> server ignores it with "already submitted, just ignore it", since the first
> handler is still reporting.
> The region stays in OPENING for as long as M1 is active. With the clocks
> skewed this way, the same comparison also lets a master that is no longer
> active accept reports for procedures started by the new master (T1 < T0),
> which is what HBASE-28690 meant to prevent. Region close and the other remote
> procedures carry the same active time, so their reports are rejected the same
> way.
> This was found through code analysis, and I reproduced the region open case
> with a test that holds the OPENED report on the active master, fails over to
> a master whose clock is behind, and releases the report. The new master
> logged {{Report for procId: 9 and initiatingMasterAT 1794786260006 received
> on master with activeTime 1794786200028}} about 50,000 to 110,000 times in
> the 60 seconds the test waits, since the region server retries without
> backing off, and the procedure never finished.
> *Expected:* the active time of a new active master is greater than that of
> every earlier one, whatever the clocks say.
> *Proposed fix:* store the active time in the master local region. A master
> becoming active opens the master local region, which fences off the previous
> master, then takes {{max(now, last stored + 1)}} as its active time and
> stores it, before any remote procedure is dispatched. A warning is logged
> when the stored time wins, since it means the masters' clocks disagree. The
> fix only covers failovers between masters that have it; branch-2.5 has
> HBASE-28690 but no master state store, so a backport there would need another
> place to keep the value.
> I have a patch with tests and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)