Siyao Meng created HDDS-16887:
---------------------------------
Summary: Pipeline scrubber pass stops at a pipeline removed by
another thread and skips the remaining pipelines until the next pass
Key: HDDS-16887
URL: https://issues.apache.org/jira/browse/HDDS-16887
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-scrubber-skip-pipeline-removed-during-pass.patch,
TestBugCR1ScrubPassAbortedByRemovedPipeline.java
h3. Mechanism
{{PipelineManagerImpl.scrubPipelines}} loops over the pipeline list read at the
start of the pass with no exception handling per pipeline. If another thread
removes a listed pipeline before the scrubber reaches it, {{closePipeline}}
(through {{closeContainersForPipeline}}) or {{deletePipeline}} throws
{{PipelineNotFoundException}}, the loop ends, and
{{scrubAndClosePipelinesMissingDataStreamPort}} logs "Unexpected error during
pipeline scrubbing" at ERROR with the stack trace. Pipelines are removed
outside the scrubber by {{DeadNodeHandler.destroyPipelines}} and by
{{closeStalePipelines}} (from {{NewNodeHandler}} and
{{NodeAddressUpdateHandler}}). {{DeadNodeHandler}} already ignores
{{PipelineNotFoundException}} for the same {{closePipeline}} and
{{deletePipeline}} calls, and {{closePipelinesMissingDataStreamPort}} catches
errors per pipeline.
h3. Trigger
# A scrub pass has several candidates (ALLOCATED past
{{ozone.scm.pipeline.allocated.timeout}}, CLOSED past
{{ozone.scm.pipeline.destroy.timeout}}, or OPEN with unregistered nodes).
# While the scrubber works on one of them, another thread removes a later one,
for example on a dead node event.
# The scrubber reaches the removed pipeline and the pass ends.
The reproduction drives the removal from the scrubber's own thread at a fixed
point; a removal racing a pass on a cluster was not exercised.
h3. Impact
* The candidates after the removed pipeline wait for the next pass, one
{{ozone.scm.pipeline.scrub.interval}} later (default 5m). The next pass reads
the list again and handles them (run).
* An ERROR log with stack trace for a benign race.
* No data impact.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR1ScrubPassAbortedByRemovedPipeline.java]
({{hadoop-hdds/server-scm}}) uses a real {{PipelineManagerImpl}} on RocksDB
with the single node HA stub, a mock node manager and a mock clock. Four
pipelines are past the ALLOCATED timeout. A Mockito spy on {{closePipeline}}
closes and deletes the second pipeline through the public API when the scrubber
starts closing the first one, as {{DeadNodeHandler}} would. The pass runs
through {{scrubAndClosePipelinesMissingDataStreamPort}}, the method the
background scrubber calls. After the first pass the error is logged and the
third and fourth pipelines are still there; after the second pass none are
left. A passing test means the defect is present.
h3. Patch
[^CR-1-scrubber-skip-pipeline-removed-during-pass.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
The loop body of {{scrubPipelines}} is wrapped in a catch of
{{PipelineNotFoundException}} that logs at DEBUG and continues with the next
pipeline, as {{DeadNodeHandler}} does. Other exceptions, for example losing SCM
leadership during the pass, still end the pass and are logged once by the
caller as before. Nothing changes when no pipeline is removed concurrently. The
change reindents the loop body; {{git diff -w}} shows five added lines.
Covered by {{testScrubPipelinesSkipsPipelineRemovedDuringScrub}} in the
existing {{TestPipelineManagerImpl}}: without the change {{scrubPipelines}}
throws ({{Unexpected exception thrown: ... PipelineNotFoundException: Pipeline
not found: ...}}); with it the pipeline after the removed one is scrubbed in
the same pass. All test classes of the SCM pipeline package,
{{TestBackgroundSCMService}}, {{TestSCMSafeModeManager}} and
{{TestDeadNodeHandler}} pass, and checkstyle is clean. With the patch the
reproduction fails at its first assertion: no error is logged and all pipelines
are removed in the first pass.
Related: a separate issue for the same loop, where the scrubber closes and
deletes a pipeline that was opened during the pass after it was read as a timed
out ALLOCATED pipeline. Its proposed patch conflicts textually with this one
and will be rebased if this one lands first.
Found by code review of the SCM pipeline lifecycle paths, as part of the TLA+
verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
