Siyao Meng created HDDS-16904:
---------------------------------
Summary: HealthyPipelineSafeModeRule keeps pipelines closed during
safe mode in its threshold, so a running SCM can stay in safe mode
Key: HDDS-16904
URL: https://issues.apache.org/jira/browse/HDDS-16904
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-drop-closed-pipelines-from-healthy-pipeline-threshold.patch,
TestBugCR4ClosedPipelinesKeepSafeModeThreshold.java
h3. Mechanism
{{HealthyPipelineSafeModeRule.initializeRule}} runs at startup and on every
refresh while the rule is unmet. It adds the IDs of the OPEN RATIS/THREE
pipelines to {{unProcessedPipelineSet}} and sets the threshold to
{{max(minHealthyPipelines, ceil(pct * set size))}}. IDs leave the set only when
the pipeline is counted as healthy, or in {{cleanup()}} after the rule has
passed. A pipeline that was OPEN when SCM started and is closed during safe
mode, before it was reported healthy, therefore stays in the threshold until
the rule passes or SCM restarts. The same holds for a pipeline added by an
earlier refresh, and in HA for a pipeline loaded from the DB as OPEN that the
Ratis log replay then closes (argued from the code). A closed pipeline cannot
become healthy, so the threshold can stay above what the remaining OPEN
pipelines can reach.
HDDS-5263 (1.2.0) added this refresh so that the pipeline thresholds follow the
pipelines that are OPEN now, for example after the Ratis log replay closes
pipelines that the DB still listed as OPEN. HDDS-9478 (2.0.0) introduced the
set to show sample pipelines in the status output, and the set undid that for
this rule. {{OneReplicaPipelineSafeModeRule}} and the report free path of the
same rule still compute the threshold from the current number of OPEN pipelines.
h3. Trigger
* Either {{hdds.scm.safemode.pipeline.creation=false}}, or fewer than three
healthy datanodes left, so that the background pipeline creator cannot build a
RATIS/THREE pipeline (for example a three datanode cluster with
{{hdds.scm.safemode.min.datanode}} of 1 or 2 and one datanode down; Ozone's
docker compose clusters default to 1). Otherwise the creator builds new
pipelines on the surviving datanodes during safe mode (closed pipelines do not
count against the datanode pipeline limit), which rescues the rule (argued, not
run).
* RATIS/THREE pipelines that are still OPEN when SCM starts, and whose members
are then partly down. The surviving members ask SCM to close them (pipeline
action) after their Ratis timeouts, which is after the one shot pipeline
creation run at the end of the precheck. That run created nothing, because
fewer than three healthy datanodes were left or the datanodes were at their
pipeline limit.
* Either no pipeline survives and {{hdds.scm.safemode.min.datanode}} is 2 or
less (with the default of 3 the rule requires one healthy pipeline anyway), or
fewer than 10% of the pipelines that were OPEN at startup survive and at least
one does. The three datanode cluster above is an example of the first case.
With {{hdds.scm.safemode.pipeline.creation=false}}, so is losing one of two
racks with rack aware placement: every pipeline loses a member, and every
container keeps a replica, so the container rules pass.
h3. Impact
SCM stays in safe mode while every other rule passes, and a fresh SCM on the
same state (a restart) would leave it. Block allocation is refused, and
{{ReplicationManager}} does not start, so the containers left under-replicated
by the lost datanodes are not repaired (argued from the code, not run). It ends
with a restart, with the lost datanodes returning or a new datanode registering
(both trigger a one shot pipeline creation run), or with {{ozone admin safemode
exit}}. No data is lost. Present since 2.0.0 (HDDS-9478).
h3. Reproduction
[^TestBugCR4ClosedPipelinesKeepSafeModeThreshold.java]
({{hadoop-hdds/server-scm}}) uses the real {{PipelineManagerImpl}} with its
background pipeline creator, {{PipelineReportHandler}},
{{PipelineActionHandler}} and {{SCMSafeModeManager}} with all its rules. Three
datanodes hold two OPEN RATIS/THREE pipelines, which fills their pipeline
limit, with one closed container with keys per pipeline. One datanode is down
at startup. The other two register with their replicas, so the container rules
pass, and the one shot creation run after the precheck creates nothing. Then
they send close actions for both pipelines, and the rules are refreshed.
Configuration: {{hdds.scm.safemode.min.datanode=1}}, with
{{hdds.scm.safemode.pipeline.creation}} left at its default and the SCM leader
ready, so the creator runs in safe mode. Output on unmodified source: 0 OPEN
pipelines, a new RATIS/THREE pipeline cannot be created, threshold 1,
{{HealthyPipelineSafeModeRule}} the only rule not met, SCM in safe mode. A new
safe mode manager on the same state computes threshold 0 and leaves safe mode.
The test passes on unmodified source, which means the defect is present. With
the patch, the refresh lowers the threshold to 0 and SCM leaves safe mode, so
the test fails.
h3. Patch
[^CR-4-drop-closed-pipelines-from-healthy-pipeline-threshold.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
a5aeb764d80. It and the patch on HDDS-16903 add the same two imports to
{{TestHealthyPipelineSafeModeRule}}, so whichever lands second needs a trivial
rebase.
{{initializeRule}} clears {{unProcessedPipelineSet}} before adding the
pipelines that are OPEN now, so the threshold is again based on the current
OPEN RATIS/THREE count, as it was from HDDS-5263 until HDDS-9478, and the
refresh thread lowers it on the running SCM. If no pipeline closes or is
deactivated during safe mode, the threshold is the same as before: the old set
already re-added the counted OPEN pipelines on every refresh.
Covered by {{testClosedPipelinesAreDroppedFromThresholdOnRefresh}} in the
existing {{TestHealthyPipelineSafeModeRule}}, with default thresholds: 11 OPEN
pipelines (threshold 2), 10 closed before they are reported, then
{{SCMSafeModeManager.refreshAndValidate}}. The rule must then require 1, and
pass on the remaining healthy pipeline. Without the change it fails
({{expected: 1 but was: 2}}). With it, the
{{org.apache.hadoop.hdds.scm.safemode}} unit tests together with
{{TestPipelineManagerImpl}} and {{TestPipelineActionHandler}} (108 tests) pass,
as do {{TestSCMSafeModeWithPipelineRules}} and
{{TestSafeModeCheckSubcommandHA}} on a mini cluster (8 tests), and checkstyle
is clean.
Found by code review of the SCM safe mode rules, as part of the TLA+
verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16903 (the same rule counts DORMANT and CLOSED pipelines, a
separate defect in {{process}}), HDDS-9478 (introduced the set), HDDS-5263
(added the refresh so that pipeline thresholds follow the current OPEN
pipelines, for example after Ratis log replay; HDDS-9478 undid that for this
rule) and HDDS-11844 (open, proposes not waiting for all pipelines). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
