[
https://issues.apache.org/jira/browse/HDDS-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16906:
------------------------------
Attachment: TestBugMC7DeadEcPipelineMemberAfterFailovers.java
> Pipeline with a DEAD datanode stays OPEN if SCM leadership changes while the
> node goes STALE and DEAD, and its EC container is not repaired
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16906
> URL: https://issues.apache.org/jira/browse/HDDS-16906
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-7-scrub-open-pipelines-with-dead-nodes.patch,
> TestBugMC7DeadEcPipelineMemberAfterFailovers.java
>
>
> h3. Mechanism
> Every SCM, leader or follower, runs its own node health check and fires
> {{STALE_NODE}} and {{DEAD_NODE}} once per transition. {{StaleNodeHandler}}
> and {{DeadNodeHandler}} close the node's pipelines through
> {{PipelineManager.closePipeline}}, which is a Ratis request: on a follower it
> fails with NotLeaderException. {{StaleNodeHandler}} logs "Could not finalize
> pipeline" at INFO without the exception, {{DeadNodeHandler}} logs the
> exception, and both drop it. No SCM fires the events again for a node that is
> already DEAD, and nothing re-runs the handlers when an SCM becomes leader.
> The only periodic check, {{PipelineManagerImpl.scrubPipelines}} on the
> leader, closes an OPEN pipeline only if a member is not registered
> (HDDS-7341); a DEAD datanode stays registered.
> So if the SCM that becomes leader saw the STALE transition while it was a
> follower, and the same happens for the DEAD transition, no SCM closes the
> pipeline. For RATIS/THREE pipelines the remaining datanodes can still trigger
> the close through pipeline actions (argued, not run). A RATIS/ONE pipeline on
> the dead datanode and EC pipelines have no such path, because no other member
> is left or because EC datanodes are not aware of pipelines.
> h3. Trigger
> # An OPEN EC pipeline, three SCMs, and a datanode of the pipeline stops.
> # The datanode goes STALE. A follower sees it first, its handler fails, and
> that follower becomes leader before the old leader's close has been applied.
> # The datanode goes DEAD. Again the SCM that becomes leader saw it first as a
> follower.
> Each SCM checks node health on its own timer
> ({{ozone.scm.heartbeat.thread.interval}}, default 3s), so each leader change
> has to fall in a window of a few seconds around the transition, at both
> transitions (5 minutes apart with the default stale and dead intervals). A
> close that fails on the leader for another reason is lost the same way
> (argued, not run).
> h3. Impact
> * The EC pipeline stays OPEN with a DEAD member.
> {{SCMBlockProtocolServer.allocateBlock}} keeps returning it to requests that
> do not exclude it. The client then fails over: it adds the pipeline to its
> exclude list and asks for another one (argued from the client code, not run).
> * Its OPEN container keeps 4 of 5 replicas. {{ReplicationManager}} does not
> reconstruct OPEN containers, and the container cannot fill, so it stays under
> replicated until the datanode returns, an admin closes the pipeline or
> container, or an SCM that was restarted after the node died becomes leader
> (then the node is unregistered there and the existing scrubber rule closes
> the pipeline).
> * In the same run, two RATIS/THREE pipelines and one RATIS/ONE pipeline with
> the dead datanode were also still OPEN on the new leader after a scrubber
> pass. Whether the datanodes close the RATIS/THREE ones later was not observed.
> h3. Reproduction
> PASS, unmodified source. [^TestBugMC7DeadEcPipelineMemberAfterFailovers.java]
> ({{hadoop-ozone/integration-test}}) starts 3 SCMs, 1 OM and 5 datanodes with
> the stale and dead intervals shortened to 6s and 40s, and writes one EC 3-2
> key with the client. It widens the two windows with the existing
> {{SCMNodeManager.pauseHealthCheck}} test hook on the current leader and moves
> leadership with the admin {{transferLeadership}} RPC: followers B and C see
> STALE, leadership moves A to B, followers A and C see DEAD, leadership moves
> B to C. On C it then runs one scrubber pass and one ReplicationManager pass.
> Output on the pin: the EC pipeline is OPEN with a DEAD member, its container
> is OPEN with 4 replicas, {{allocateBlock}} for EC 3-2 returns that pipeline,
> and the log shows the stale node handler's "Could not finalize pipeline"
> lines and the dead node handler failing with NotLeaderException on the
> followers. As a control, a second member stopped while C is leader gets the
> pipeline closed. A passing test means the defect is present. With the patch
> it fails at the first check, because the scrubber pass closes the pipeline
> and the container moves to CLOSING.
> h3. Patch
> [^MC-7-scrub-open-pipelines-with-dead-nodes.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> b9b33851b20.
> {{scrubPipelines}} also closes an OPEN pipeline that has a DEAD member, next
> to the existing rule for unregistered members. The scrubber runs only on the
> leader, so the close no longer depends on which SCM saw the transition, and
> it also covers a close that failed on a stable leader. {{closePipeline}}
> closes the pipeline's containers too, so ReplicationManager can repair them
> once they are closed. The scrubber runs only on a leader that is ready and
> out of safe mode. In normal operation the leader's stale or dead node handler
> has already closed every pipeline of a DEAD node, of any type, so the rule
> finds nothing. If such a close was lost, the scrubber does it at most one
> scrub interval (default 5 minutes) after the node is DEAD. It does not act on
> STALE members. Covered by {{testScrubOpenWithDeadNodes}} in the existing
> {{TestPipelineManagerImpl}} (a STALE member keeps the pipeline open, a DEAD
> member closes it), which fails without the change. With it the SCM pipeline
> unit suites, {{TestSCMSafeModeManager}}, {{TestSCMNodeManager}} and
> {{TestDeadNodeHandler}} (359 tests) and {{TestDecommissionAndMaintenance}} (9
> tests) pass, checkstyle is clean.
> Related: HDDS-16887. If {{DeadNodeHandler}} deletes the pipeline during a
> scrubber pass, the new close hits the {{PipelineNotFoundException}} that
> HDDS-16887 handles (today it ends the pass, as it already can for the
> existing rules). HDDS-16887 also wraps the loop body in {{scrubPipelines}} in
> a {{try}} block, so the patches conflict textually and whichever lands second
> needs a trivial rebase.
> Found by TLA+ model checking and code review of the SCM pipeline lifecycle
> paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
