[
https://issues.apache.org/jira/browse/HDDS-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16896:
------------------------------
Attachment: TestBugMC2BootstrappedScmSafeMode.java
> Bootstrapped SCM builds its container safe mode rules from its empty DB and
> leaves safe mode before datanodes report its containers
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16896
> URL: https://issues.apache.org/jira/browse/HDDS-16896
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-reload-container-safemode-rules-after-catch-up.patch,
> TestBugMC2BootstrappedScmSafeMode.java
>
>
> h3. Mechanism
> {{RatisContainerSafeModeRule}} and {{ECContainerSafeModeRule}} read the
> CLOSED and QUASI_CLOSED containers with keys from the container DB once, in
> the SCM constructor ({{AbstractContainerSafeModeRule.initializeRule}}). That
> is before {{StorageContainerManager.start}} replays the Ratis log or installs
> a snapshot. Afterwards the periodic refresh ({{reinitializeRule}}) only
> removes DELETED containers, by design since HDDS-15238, and
> {{SCMSafeModeManager.refreshAndValidate}} does nothing when the periodic
> refresh is enabled (the default).
> On a newly bootstrapped SCM the DB is still empty when the rules are built,
> so both rules see 0 of 0 containers and {{getCurrentContainerThreshold}}
> returns 1. Both are marked validated on the next safe mode refresh (every
> 5s), usually before the state machine has caught up. Nothing reloads them, so
> even when catch-up is faster, they are validated at 0 of 0. A validated rule
> is never revalidated, so as soon as the datanode, pipeline and state machine
> rules pass, the SCM leaves safe mode, however few of the containers in the
> installed snapshot the datanodes have reported.
> h3. Trigger
> # An HA cluster with CLOSED containers that have keys.
> # A new SCM is bootstrapped and catches up from the leader (in every run, by
> installing a checkpoint of the leader DB, with the default Ratis settings).
> # Only some datanodes report to the new SCM, as during a rolling update of
> {{ozone.scm.nodes}} on the datanodes, or when datanodes register unevenly.
> # The new SCM leaves safe mode once the minimum number of datanodes and the
> pipeline rules are satisfied.
> h3. Impact
> * The new SCM leaves safe mode with only part of its closed containers
> reported. In the reproduction this was 6 of 10 containers, with a cutoff of
> 99%.
> * If it becomes leader before the remaining datanodes report to it
> (leadership transfer, or failover during the rollout), ReplicationManager
> runs on that partial view after {{hdds.scm.wait.time.after.safemode.exit}} (5
> minutes by default, 3 seconds in the mini cluster). In the reproduction it
> classified the 4 unreported single replica containers as MISSING. MISSING is
> only a classification: no command is sent for it.
> * Containers with at least one reported replica are handled as on any SCM
> that leaves safe mode before all datanodes have reported. What this defect
> adds is that containers with no reported replica no longer keep the SCM in
> safe mode, so the container check of safe mode does nothing on a bootstrapped
> SCM. I found no path to data loss.
> * The effect ends when the remaining datanodes register with the new SCM.
> h3. Reproduction
> PASS on the unmodified source. [^TestBugMC2BootstrappedScmSafeMode.java]
> ({{hadoop-ozone/integration-test}}) runs a mini cluster with 3 SCMs, 1 OM and
> 5 datanodes, and uses the client, {{closeContainer}}, {{bootstrapSCM}},
> {{transferLeadership}} and the datanode reconfiguration handler, as an
> operator adding an SCM would. It writes RATIS/ONE keys, closes their
> containers, bootstraps a fourth SCM with the default Ratis settings and
> points only the three datanodes of one RATIS/THREE pipeline at it. Before any
> datanode registers, the new SCM reports {{RatisContainerSafeModeRule
> validate=true: ... Containers(0 / 0)}} although it holds 10 closed containers
> with keys. After the three datanodes register, it leaves safe mode within
> about 1s with 6 of 10 containers reported. After leadership is transferred to
> it, ReplicationManager runs and reports MISSING=4. A passing test means the
> defect is present.
> With the patch the same test fails at its first assertion. The new SCM logs
> "Initialized RATIS Containers threshold count to 10" once it has caught up,
> stays in safe mode at {{Containers(6 / 10)}}, and ReplicationManager does not
> run after the leadership transfer.
> Not run: the restart variant (a restarted SCM whose replayed log closes
> containers). Containers closed while that SCM was down are replayed with the
> key count it had persisted, which is usually 0 for a container that was open
> when it stopped, so the rule mostly skips them anyway.
> h3. Suggested fix
> Rebuilding the rules after every snapshot install is not safe. After the
> datanode server has started, a rebuild would refill the list from the DB
> without resetting the reported count. Containers that registered datanodes
> have already reported would be back in the list and could be counted a second
> time. Containers closed since registration would raise the total, but
> datanodes report containers to these rules only when they register, so SCM
> could stay in safe mode indefinitely (the concern behind HDDS-15238).
> Clearing an already validated rule from {{validatedRules}} at readiness would
> race with the refresh thread. The patch instead keeps the rules from
> validating until the data they count is current.
> h3. Patch
> [^MC-2-reload-container-safemode-rules-after-catch-up.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 6818707698c.
> When the SCM has a Ratis state machine, {{SafeModeRuleFactory}} passes it to
> the two container rules. Until the state machine is ready,
> {{AbstractContainerSafeModeRule.validate}} returns false. On its first call
> after that, it reads the containers again with {{initializeRule}}. Datanode
> reports only reach the rules after the state machine is ready, because the
> datanode server is started in
> {{SCMStateMachine.tryStartDNServerAndRefreshSafeMode}}, and each report goes
> through {{validate}} before it is processed. So the reload always runs before
> any container is counted, and no report is lost. Rules without a state
> machine ({{SCMHAManagerStub}}, unit tests) behave as before. An SCM whose DB
> is still empty after catch-up still validates the rules at once.
> The fix relies on the datanode server starting only after the state machine
> is ready, which came with HDDS-14989 (2.3.0). Released versions (2.2.0 and
> 2.2.1) build the rules the same way and start the datanode server in
> {{StorageContainerManager.start}} unconditionally, so they are also affected
> (argued from the code, not run). They need HDDS-14989 together with this fix:
> without it, reports processed before readiness stay counted, because
> {{initializeRule}} does not reset the reported count or the EC replica map,
> so the reload could over count.
> Covered by {{testContainerRulesCountContainersLoadedBeforeStateMachineReady}}
> in the existing {{TestSCMSafeModeManager}}. The rules are built from an empty
> DB, 10 closed containers are then added and the state machine becomes ready,
> and a datanode reports half of them. Without the change SCM leaves safe mode
> at that point. With it, SCM stays in safe mode until the remaining containers
> are reported, and with 0 containers it leaves safe mode on the first
> registration as before. With the patch, the safe mode unit suites (76 tests)
> and the mini cluster suites {{TestSCMFollowerCatchupWithContainerReport}},
> {{TestDatanodeSCMNodesReconfiguration}}, {{TestSafeModeCheckSubcommandHA}},
> {{TestSCMInstallSnapshot}}, {{TestStorageContainerManagerHA}},
> {{TestSCMSnapshot}} and {{TestFailoverWithSCMHA}} pass, and checkstyle is
> clean.
> Not changed: the same cause makes {{OneReplicaPipelineSafeModeRule}} pass
> with a threshold of 0 on a bootstrapped SCM. It is built from the empty
> pipeline DB, validated on the first refresh, and its refresh only rebuilds it
> while it is not validated, so the threshold is never corrected. What is lost
> is the check that 90% of the open RATIS/THREE pipelines have a reported
> datanode before exit. Its effect is limited, because
> {{HealthyPipelineSafeModeRule}} is rebuilt on every refresh until it passes
> and still waits for at least one healthy pipeline (with the default
> {{hdds.scm.safemode.min.datanode}} of 3). It is left for a follow up.
> Found by TLA+ model checking and code review of the SCM safe mode rules under
> HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
