Siyao Meng created HDDS-16903:
---------------------------------
Summary: HealthyPipelineSafeModeRule counts DORMANT and CLOSED
pipelines, so SCM can leave safe mode with no healthy OPEN RATIS/THREE pipeline
Key: HDDS-16903
URL: https://issues.apache.org/jira/browse/HDDS-16903
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch,
TestBugMC4NonOpenPipelineCountsAsHealthy.java
h3. Mechanism
While SCM is in safe mode, {{PipelineReportHandler.processPipelineReport}}
fires {{OPEN_PIPELINE}} for every reported pipeline that is healthy (all
members reported, leader known), whatever its state in SCM.
{{HealthyPipelineSafeModeRule.process}}, the only subscriber, checks the
replication config, whether the pipeline was already counted, the member count
and the member health, but not the state. A DORMANT or CLOSED RATIS/THREE
pipeline whose datanodes still host the Ratis group is therefore counted,
although the threshold in {{initializeRule}} is computed from OPEN pipelines
only. The count never goes down, and once every rule has passed SCM does not
return to safe mode.
The other checks in the same area count OPEN pipelines only:
{{OneReplicaPipelineSafeModeRule.process}} checks {{isOpen()}}, and so does the
report free path {{validateHealthyPipelineSafeModeRuleUsingPipelineManager}}
(HDDS-13590). HDDS-4399 ("Safe mode rule for pipelines should only consider
open pipelines") moved the {{fireEvent}} out of the ALLOCATED branch so that
pipelines reloaded as OPEN are counted, and the state check was lost with that
move.
h3. Trigger
Datanodes keep a pipeline's Ratis group, and keep reporting it, until SCM sends
a close command. For a pipeline that was deactivated, or closed by a pipeline
action, an admin close or {{StaleNodeHandler}}, SCM sends it only when the
pipeline scrubber deletes the CLOSED pipeline, and the scrubber runs only
outside safe mode. Two ways to get there:
* DORMANT: an admin runs {{ozone admin pipeline deactivate}}. Only the SCM
state changes, and the state is persisted, so the pipeline counts at every
later SCM restart for as long as its datanodes are up.
* CLOSED: during safe mode a member asks SCM to close the pipeline (pipeline
action, for example after a Ratis failure), an admin runs {{ozone admin
pipeline close}}, or a member goes stale for a while (for example a long GC
pause) and {{StaleNodeHandler}} closes its pipelines before it recovers. The
members keep reporting the pipeline for the rest of safe mode. A pipeline
closed shortly before an SCM restart is loaded as CLOSED and behaves the same
(argued, not run).
SCM leaves safe mode on such a pipeline once the other rules pass, for example
after a restart where an OPEN pipeline has only one live member: that one
member satisfies {{OneReplicaPipelineSafeModeRule}}, but the pipeline cannot
become healthy.
h3. Impact
SCM leaves safe mode although no OPEN RATIS/THREE pipeline is healthy. That is
the situation this rule exists to prevent (HDDS-1178). Block allocation is then
allowed, and new RATIS/THREE writes can only proceed once a new pipeline is
created and opened. Where the datanodes have room for a new pipeline, this
mostly makes the exit earlier. Where they have none (for example, the slots are
held by the DORMANT pipeline), writes keep failing until an operator acts
(argued from the code, not run). No data is lost, and the container rules still
gate the exit. The state check is missing since 1.1.0 (HDDS-4399).
h3. Reproduction
[^TestBugMC4NonOpenPipelineCountsAsHealthy.java] ({{hadoop-hdds/server-scm}})
uses the real {{PipelineManagerImpl}}, {{PipelineReportHandler}},
{{PipelineActionHandler}} and {{SCMSafeModeManager}} with all its rules, driven
only by datanode registration, pipeline report and pipeline action events.
Pipeline P1 is OPEN with one live member. Pipeline P2 is on three other live
datanodes and is deactivated before the start (first test), or closed by a
member's close action during safe mode (second test). Before P2's members
report it, only {{HealthyPipelineSafeModeRule}} is unmet. After their reports,
SCM leaves safe mode with P1 as the only OPEN RATIS/THREE pipeline and P1 not
healthy. Both tests pass on unmodified source, which means the defect is
present. With the patch, both fail because SCM stays in safe mode. The
reproduction does not model datanodes opening pipelines that the background
pipeline creator allocates.
h3. Patch
[^MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
a5aeb764d80.
{{HealthyPipelineSafeModeRule.process}} reads the pipeline's current state from
{{PipelineManager}} and skips it unless it is OPEN, as
{{OneReplicaPipelineSafeModeRule}} does. The check is in the rule rather than
in {{PipelineReportHandler}} for two reasons. The {{Pipeline}} object the
handler fires is the copy it read before calling {{openPipeline}}, so it still
says ALLOCATED for a pipeline that the same report has just opened. And the
event is processed asynchronously. {{PipelineReportHandler}} is unchanged.
One behaviour change: if the only fully reported RATIS/THREE pipelines are
DORMANT or CLOSED and no OPEN one can become healthy, SCM now stays in safe
mode until one does or an operator runs {{ozone admin safemode exit}}, instead
of leaving it without a writable pipeline. With the default
{{hdds.scm.safemode.pipeline.creation=true}} a CLOSED pipeline only delays the
exit while three healthy datanodes remain: CLOSED pipelines do not count
against the datanode pipeline limit, so the background creator builds a new
one. The case that changes is DORMANT pipelines taking every free pipeline
slot, for example an admin who deactivates all pipelines and then restarts SCM.
SCM then stays in safe mode, so {{ReplicationManager}} does not run either,
until the admin activates a pipeline or runs {{ozone admin safemode exit}}.
Covered by {{testPipelineNotOpenIsIgnored}} in the existing
{{TestHealthyPipelineSafeModeRule}}: with one OPEN, one DORMANT and one CLOSED
pipeline, all fully reported, the rule must not count the DORMANT or the CLOSED
pipeline, and must pass on the OPEN one. The events carry stale copies, as the
handler's may: an OPEN copy of the DORMANT and of the CLOSED pipeline, and the
ALLOCATED copy of the OPEN pipeline, so a check on the event's own copy fails
the test as well. Without the change it fails ({{expected: 0 but was: 1}}).
With it, the {{org.apache.hadoop.hdds.scm.safemode}} unit tests together with
{{TestPipelineManagerImpl}} and {{TestPipelineActionHandler}} (108 tests) pass,
as do {{TestSCMSafeModeWithPipelineRules}} and
{{TestSafeModeCheckSubcommandHA}} on a mini cluster (8 tests), and checkstyle
is clean.
Found by TLA+ model checking and code review of the SCM safe mode rules under
HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. This issue also
covers the same defect found by code review of the SCM pipeline lifecycle under
HDDS-16538, which approached it from {{PipelineReportHandler}} and the DORMANT
state. Checked against HDDS issues and apache/ozone pull requests for
duplicates before filing. Related but not duplicates: HDDS-4399 (introduced the
state independent event), HDDS-13590 (OPEN filter for the report free path
only), HDDS-11904 (datanode health check in the same method) and HDDS-11844
(open, proposes not waiting for all pipelines). The attached patch is a
proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
