SpookyJumpyBeans opened a new pull request, #8898: URL: https://github.com/apache/hbase/pull/8898
https://issues.apache.org/jira/browse/HBASE-30547 Since HBASE-28690 a master rejects a report whose initiating master active time is greater than its own. The active time comes from the wall clock, so after a failover to a master whose clock is behind, the new master can have a lower active time than the old one and rejects every report for a procedure the old master started. For a region open, the region server keeps retrying OPENED with the old active time and ignores the new master's re-dispatch of the same procedure, so the region stays in OPENING while the new master is active. ### Changes - New `MasterActiveTimeStore`, a `MasterStateStore` holding the last active master's active time in the master local region. - `HMaster.finishActiveMasterInitialization` still reads the clock as before, then, right after opening the master local region, sets the active time to `max(now, last stored + 1)` and stores it: ```java masterRegion = MasterRegionFactory.create(this); rsListStorage = new MasterRegionServerList(masterRegion, this); // The active time is sent with remote procedures and must be greater than that of every earlier // active master, see MasterActiveTimeStore. Opening the master local region has fenced off the // earlier active master, and no remote procedure has been sent yet. this.masterActiveTime = new MasterActiveTimeStore(masterRegion).advance(masterActiveTime); ``` Opening the master local region renames the previous master's WAL directory and recovers the lease on its WAL files, so the previous master can no longer write to it. The procedure executor is created after this point, so no remote procedure carries the earlier value, and reports are only checked once `serviceStarted` is set, also later. A warning is logged when the stored time is used, since it means the masters' clocks disagree. - The "completed initialization" log line measures from the clock read instead of the active time, which can now be ahead of the clock. - `masterActiveTime` is now `volatile`, since RPC handlers read it and it is now written twice during initialization. - The clock read is moved into a protected `getCurrentTimeForActiveTime()`, so a test can simulate a master whose clock is behind. ### Limits - If a past active master's clock was far ahead, later active times follow the stored value (plus one) until the clocks catch up, so the active time shown in the UI and the `masterActiveTime` metric can be ahead of the wall clock in that time. The comparisons stay consistent; without this change such a master would have caused this bug for the masters after it. - Only failovers between masters with this change are covered. The first master started on this version has no stored value and uses its clock. An older master in a rolling upgrade does not store its active time. - branch-2.5 has HBASE-28690 but no master state store, so a backport there would need another place for the value. branch-2.6, branch-2, branch-3 have `MasterStateStore`. Other approaches I looked at: the master start code comes from the same clock; the zxid of the active master znode is monotonic but on a different scale, which would break comparisons with older masters during an upgrade; and the initiating active time is not persisted with the procedures, so it can't be recovered from the procedure store. ### Tests - `TestOpenRegionReportAfterFailoverToMasterWithClockBehind`: holds the OPENED report on the active master, fails over to a master whose clock is behind (through an override of `getCurrentTimeForActiveTime()`), and releases the report. Without this change the new master keeps rejecting the report ("initiatingMasterAT 1794786260006 received on master with activeTime 1794786200028", thousands of times per run) and the open never finishes. With this change the open finishes on the new master. - `TestMasterActiveTimeStore`: unit tests for `advance`, including a malformed stored value. I ran these on Linux (WSL) with JDK 21: the new test passed 5 of 5 runs with this change and fails without it. These also pass with it: `TestMasterActiveTimeStore`, `TestOpenRegionProcedureHang`, `TestReportRegionStateTransitionRetry`, `TestServerRemoteProcedure`, `TestMasterFailover`, `TestMasterMetrics`, `TestMasterMetricsWrapper`, `TestMasterNoCluster`, `TestMasterRegionInitialize`, `TestSnapshotCleanupStateStore`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
