[
https://issues.apache.org/jira/browse/HDDS-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16894:
------------------------------
Attachment:
MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch
> OneReplicaPipelineSafeModeRule still counts reports of pipelines closed
> during safe mode after its refresh lowers the threshold
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16894
> URL: https://issues.apache.org/jira/browse/HDDS-16894
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch,
> TestBugMC3OneReplicaRuleCountsClosedPipeline.java
>
>
> h3. Mechanism
> While the rule is not met, {{OneReplicaPipelineSafeModeRule.refresh(false)}}
> calls {{initializeRule(true)}}, which rebuilds {{oldPipelineIDSet}} and
> {{thresholdCount}} from the RATIS/THREE pipelines that are open now.
> {{currentReportedPipelineCount}} and {{reportedPipelineIDSet}} are kept as
> they are, so reports of pipelines that have closed since still count towards
> a threshold that only covers open pipelines. With R reported pipelines of
> which C have closed, the rule passes once R reaches ceil(0.9 x open), so it
> covers C fewer open pipelines than configured. Once {{SCMSafeModeManager}}
> records the rule as validated it is never checked again.
> The class Javadoc says the rule ensures at least one reported datanode per
> open pipeline, so that open containers have a replica to read when SCM leaves
> safe mode. The container rules only count CLOSED and QUASI_CLOSED containers.
> h3. Trigger
> # SCM starts in safe mode with open RATIS/THREE pipelines, and the datanodes
> of some of them report.
> # One of the reported pipelines is closed while SCM is in safe mode: by
> {{StaleNodeHandler}} or {{DeadNodeHandler}} after a registered datanode goes
> silent, by a Ratis pipeline action from a datanode, or by {{ozone admin
> pipeline close}}.
> # The periodic refresh ({{hdds.scm.safemode.rule.refresh.interval}}, default
> 5s) runs. In releases before the periodic refresh thread, the refresh runs on
> {{SCMStateMachine.applyTransaction}} in safe mode, so the transaction that
> closes the pipeline triggers it (code reading, not run on a release).
> h3. Impact
> * Smallest case, run: two open pipelines p1 and p2 (threshold 2). Only p1's
> three datanodes register and report, and p1 is reported healthy. p1 is
> closed. The refresh sets the threshold to 1 with the count still 1, the rule
> validates with 0 of 1 open pipelines reported, and in the same refresh cycle
> SCM leaves safe mode, because every other rule already passes (3 datanodes
> registered, one healthy pipeline, no closed containers).
> * SCM leaves safe mode only if the other rules pass. If the unreported
> datanodes hold closed containers with keys, the container rule still holds
> SCM in safe mode unless those containers are under 1% of the total. The
> rule's own bound already allows 10% of open pipelines to stay unreported. The
> bug loosens that by one pipeline per reported pipeline closed during safe
> mode.
> * After an early exit, {{getContainerWithPipeline}} no longer refuses open
> containers without reported replicas, so reads of those containers go to
> datanodes that have not reported until they come back, and block allocation
> can pick the unreported open pipeline (code reading, not run). No data loss
> is expected.
> * The refresh path dates from HDDS-5263 (1.2.0). Related: HDDS-11844 proposes
> changing what this rule requires before SCM leaves safe mode. This report and
> patch keep the current requirement.
> h3. Reproduction
> [^TestBugMC3OneReplicaRuleCountsClosedPipeline.java]
> ({{hadoop-hdds/server-scm}}) runs the smallest case above with the real
> {{PipelineManagerImpl}}, the real {{SCMSafeModeManager}} with all its rules
> and its refresh thread (interval lowered to 100ms), real registration,
> pipeline report and open pipeline events, and
> {{PipelineManagerImpl.closePipeline}}. On the unmodified commit it fails
> every time with {{SCM left safe mode with 0 of 1 open pipelines reported}},
> and the log shows {{OneReplicaPipelineSafeModeRule rule is successfully
> validated}} followed by {{SCM exiting safe mode.}} With the patch it passes.
> Not run on a real cluster.
> h3. Patch
> [^MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch],
> against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master
> at 6818707698c.
> On refresh, {{initializeRule(true)}} keeps only the reported pipelines that
> are still open ({{reportedPipelineIDSet.retainAll(oldPipelineIDSet)}}) and
> lowers {{currentReportedPipelineCount}} by the number removed. Reports of
> pipelines that are still open are kept. The count is lowered rather than set
> to the set size because {{cleanup()}} empties the set after the rule has
> passed. From the code, this also covers the report free validation path
> ({{updateReportedPipelineSet}}), which uses the same set and count. That path
> is not tested. Nothing changes when no reported pipeline closes during safe
> mode. The metric {{currentPipelinesWithAtleastOneReplicaReportedCount}} is a
> counter and still includes the closed pipelines.
> Covered by {{testRefreshDropsReportsOfPipelinesNoLongerOpen}} in the existing
> {{TestOneReplicaPipelineSafeModeRule}}: three open pipelines, two report, one
> of those closes, and after the refresh the rule expects threshold 2 and count
> 1 and is not met. A report from the third pipeline then satisfies it. Without
> the change it fails ({{expected: <1> but was: <2>}}). With it, the
> {{org.apache.hadoop.hdds.scm.safemode}} unit tests (75) and
> {{TestSCMSafeModeWithPipelineRules}} on a mini cluster (2) pass, and
> checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM safe mode rules under
> HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
