[
https://issues.apache.org/jira/browse/HDDS-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16904:
------------------------------
Attachment: TestBugCR4ClosedPipelinesKeepSafeModeThreshold.java
> HealthyPipelineSafeModeRule keeps pipelines closed during safe mode in its
> threshold, so a running SCM can stay in safe mode
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16904
> URL: https://issues.apache.org/jira/browse/HDDS-16904
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-4-drop-closed-pipelines-from-healthy-pipeline-threshold.patch,
> TestBugCR4ClosedPipelinesKeepSafeModeThreshold.java
>
>
> h3. Mechanism
> {{HealthyPipelineSafeModeRule.initializeRule}} runs at startup and on every
> refresh while the rule is unmet. It adds the IDs of the OPEN RATIS/THREE
> pipelines to {{unProcessedPipelineSet}} and sets the threshold to
> {{max(minHealthyPipelines, ceil(pct * set size))}}. IDs leave the set only
> when the pipeline is counted as healthy, or in {{cleanup()}} after the rule
> has passed. A pipeline that was OPEN when SCM started and is closed during
> safe mode, before it was reported healthy, therefore stays in the threshold
> until the rule passes or SCM restarts. The same holds for a pipeline added by
> an earlier refresh, and in HA for a pipeline loaded from the DB as OPEN that
> the Ratis log replay then closes (argued from the code). A closed pipeline
> cannot become healthy, so the threshold can stay above what the remaining
> OPEN pipelines can reach.
> HDDS-5263 (1.2.0) added this refresh so that the pipeline thresholds follow
> the pipelines that are OPEN now, for example after the Ratis log replay
> closes pipelines that the DB still listed as OPEN. HDDS-9478 (2.0.0)
> introduced the set to show sample pipelines in the status output, and the set
> undid that for this rule. {{OneReplicaPipelineSafeModeRule}} and the report
> free path of the same rule still compute the threshold from the current
> number of OPEN pipelines.
> h3. Trigger
> * Either {{hdds.scm.safemode.pipeline.creation=false}}, or fewer than three
> healthy datanodes left, so that the background pipeline creator cannot build
> a RATIS/THREE pipeline (for example a three datanode cluster with
> {{hdds.scm.safemode.min.datanode}} of 1 or 2 and one datanode down; Ozone's
> docker compose clusters default to 1). Otherwise the creator builds new
> pipelines on the surviving datanodes during safe mode (closed pipelines do
> not count against the datanode pipeline limit), which rescues the rule
> (argued, not run).
> * RATIS/THREE pipelines that are still OPEN when SCM starts, and whose
> members are then partly down. The surviving members ask SCM to close them
> (pipeline action) after their Ratis timeouts, which is after the one shot
> pipeline creation run at the end of the precheck. That run created nothing,
> because fewer than three healthy datanodes were left or the datanodes were at
> their pipeline limit.
> * Either no pipeline survives and {{hdds.scm.safemode.min.datanode}} is 2 or
> less (with the default of 3 the rule requires one healthy pipeline anyway),
> or fewer than 10% of the pipelines that were OPEN at startup survive and at
> least one does. The three datanode cluster above is an example of the first
> case. With {{hdds.scm.safemode.pipeline.creation=false}}, so is losing one of
> two racks with rack aware placement: every pipeline loses a member, and every
> container keeps a replica, so the container rules pass.
> h3. Impact
> SCM stays in safe mode while every other rule passes, and a fresh SCM on the
> same state (a restart) would leave it. Block allocation is refused, and
> {{ReplicationManager}} does not start, so the containers left
> under-replicated by the lost datanodes are not repaired (argued from the
> code, not run). It ends with a restart, with the lost datanodes returning or
> a new datanode registering (both trigger a one shot pipeline creation run),
> or with {{ozone admin safemode exit}}. No data is lost. Present since 2.0.0
> (HDDS-9478).
> h3. Reproduction
> [^TestBugCR4ClosedPipelinesKeepSafeModeThreshold.java]
> ({{hadoop-hdds/server-scm}}) uses the real {{PipelineManagerImpl}} with its
> background pipeline creator, {{PipelineReportHandler}},
> {{PipelineActionHandler}} and {{SCMSafeModeManager}} with all its rules.
> Three datanodes hold two OPEN RATIS/THREE pipelines, which fills their
> pipeline limit, with one closed container with keys per pipeline. One
> datanode is down at startup. The other two register with their replicas, so
> the container rules pass, and the one shot creation run after the precheck
> creates nothing. Then they send close actions for both pipelines, and the
> rules are refreshed. Configuration: {{hdds.scm.safemode.min.datanode=1}},
> with {{hdds.scm.safemode.pipeline.creation}} left at its default and the SCM
> leader ready, so the creator runs in safe mode. Output on unmodified source:
> 0 OPEN pipelines, a new RATIS/THREE pipeline cannot be created, threshold 1,
> {{HealthyPipelineSafeModeRule}} the only rule not met, SCM in safe mode. A
> new safe mode manager on the same state computes threshold 0 and leaves safe
> mode. The test passes on unmodified source, which means the defect is
> present. With the patch, the refresh lowers the threshold to 0 and SCM leaves
> safe mode, so the test fails.
> h3. Patch
> [^CR-4-drop-closed-pipelines-from-healthy-pipeline-threshold.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> a5aeb764d80. It and the patch on HDDS-16903 add the same two imports to
> {{TestHealthyPipelineSafeModeRule}}, so whichever lands second needs a
> trivial rebase.
> {{initializeRule}} clears {{unProcessedPipelineSet}} before adding the
> pipelines that are OPEN now, so the threshold is again based on the current
> OPEN RATIS/THREE count, as it was from HDDS-5263 until HDDS-9478, and the
> refresh thread lowers it on the running SCM. If no pipeline closes or is
> deactivated during safe mode, the threshold is the same as before: the old
> set already re-added the counted OPEN pipelines on every refresh.
> Covered by {{testClosedPipelinesAreDroppedFromThresholdOnRefresh}} in the
> existing {{TestHealthyPipelineSafeModeRule}}, with default thresholds: 11
> OPEN pipelines (threshold 2), 10 closed before they are reported, then
> {{SCMSafeModeManager.refreshAndValidate}}. The rule must then require 1, and
> pass on the remaining healthy pipeline. Without the change it fails
> ({{expected: 1 but was: 2}}). With it, the
> {{org.apache.hadoop.hdds.scm.safemode}} unit tests together with
> {{TestPipelineManagerImpl}} and {{TestPipelineActionHandler}} (108 tests)
> pass, as do {{TestSCMSafeModeWithPipelineRules}} and
> {{TestSafeModeCheckSubcommandHA}} on a mini cluster (8 tests), and checkstyle
> is clean.
> Found by code review of the SCM safe mode rules, as part of the TLA+
> verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16903 (the same rule counts DORMANT and CLOSED pipelines, a
> separate defect in {{process}}), HDDS-9478 (introduced the set), HDDS-5263
> (added the refresh so that pipeline thresholds follow the current OPEN
> pipelines, for example after Ratis log replay; HDDS-9478 undid that for this
> rule) and HDDS-11844 (open, proposes not waiting for all pipelines). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
