[
https://issues.apache.org/jira/browse/HDDS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16890:
------------------------------
Attachment: MC-2-close-containers-allocated-during-pipeline-close.patch
> SCM closePipeline leaves a container allocated during the close OPEN on the
> CLOSED pipeline until the pipeline is deleted
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16890
> URL: https://issues.apache.org/jira/browse/HDDS-16890
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-2-close-containers-allocated-during-pipeline-close.patch,
> TestBugMC2ClosePipelineOrphanContainer.java
>
>
> h3. Mechanism
> {{PipelineManagerImpl.closePipeline}} first calls
> {{closeContainersForPipeline}}, which reads the pipeline's containers and
> FINALIZEs each OPEN one (FINALIZE moves it to CLOSING and removes it from the
> pipeline, then a CLOSE_CONTAINER event is fired). Only after that does it
> take the pipeline manager write lock and move the pipeline to CLOSED. No lock
> is held between the read and the CLOSED state. Allocation holds only the
> pipeline manager read lock ({{WritableRatisContainerProvider.getContainer}},
> used by {{BlockManagerImpl.allocateBlock}}, and
> {{ContainerManagerImpl.allocateContainer}}, used by the admin
> {{allocateContainer}} RPC), and
> {{PipelineStateMap.addContainerToOpenPipeline}} rejects only CLOSED
> pipelines. A container allocated in that window is therefore added to the
> pipeline and never FINALIZEd. It stays OPEN on the CLOSED pipeline, or on a
> deleted one when the caller deletes right after closing ({{DeadNodeHandler}},
> {{closeStalePipelines}}, the datastream port close).
> HDDS-5973 ordered the close as containers first, then pipeline, so that SCM
> never has an OPEN container on a closed pipeline; its review states the goal
> as "containers are in closing before pipeline remove/close log in ratis".
> Concurrent allocation was not discussed there. HDDS-5655 added the read lock,
> which keeps allocation away from the CLOSED transition itself but not from
> the window before it.
> h3. Trigger
> Any pipeline close (pipeline action from a datanode, stale or dead node,
> decommission or maintenance start, admin close, scrubber, full EC pipeline)
> while an allocation that creates a new container runs on the same pipeline.
> For Ratis this is ordinary block allocation. A Ratis pipeline in steady state
> is at its open container limit, and each FINALIZE in the first pass removes a
> container from the pipeline, so a block allocation that picks this still OPEN
> pipeline during the close finds it below the limit and creates a new
> container ({{ContainerManagerImpl.getMatchingContainer}}). The close itself
> makes allocation create the late container. The window lasts as long as the
> FINALIZE calls of the first pass. For EC, block allocation adds a container
> only to a pipeline it has just created and not yet opened
> ({{WritableECContainerProvider}}). That pipeline is already registered with
> the node manager, so a stale node, dead node or decommission handler that
> closes it in that moment leaves the container OPEN on it, and the following
> {{openPipeline}} fails with "Closed pipeline can not be opened". That is
> rare; otherwise only the admin {{allocateContainer}} RPC hits the EC route.
> h3. Impact
> * The late container gets no FINALIZE and no close command. Allocation offers
> only OPEN pipelines, so no later block goes into it.
> * {{OpenContainerHandler}} in {{ReplicationManager}} closes an OPEN container
> when {{ReplicationManager.hasHealthyPipeline}} is false or a replica is
> reported in another state. {{hasHealthyPipeline}} only checks that the
> pipeline exists, so a late container whose replicas are OPEN (or that has
> none yet) is closed only after the scrubber deletes the CLOSED pipeline. With
> the default values ({{ozone.scm.pipeline.destroy.timeout}} 66s,
> {{ozone.scm.pipeline.scrub.interval}} 5m,
> {{hdds.scm.replication.thread.interval}} 300s) that is up to about 11 minutes
> after the close. For callers that delete right away it is the next
> ReplicationManager run. If ReplicationManager or the scrubber is stopped, the
> container stays OPEN.
> * When the close command finally arrives the Ratis group is gone, so the
> datanode quasi closes a Ratis container ({{CloseContainerCommandHandler}}, no
> write channel and no force flag) instead of closing it through Ratis, and
> ReplicationManager has to resolve the quasi closed replicas later.
> * What the writer sees: {{BlockManagerImpl.newBlock}} reads the pipeline
> again after the container was chosen, so {{allocateBlock}} can return the
> CLOSED pipeline with a block in a container SCM still has as OPEN. After a
> close without delete the Ratis group stays up until the scrubber deletes the
> pipeline, so writes of that block into the OPEN container go to a live group
> and succeed. After a close with delete the group is removed, the write fails
> and the client excludes the pipeline and asks for a new block, as it does for
> the other containers of that pipeline. No path that acknowledges a write on a
> removed group was found.
> The ReplicationManager, scrubber, datanode and client steps above are argued
> from code. The reproduction stops after {{closePipeline}} returns.
> h3. Reproduction
> [^TestBugMC2ClosePipelineOrphanContainer.java] ({{hadoop-hdds/server-scm}},
> target path and Maven command in the class comment) runs a real
> {{PipelineManagerImpl}} and {{ContainerManagerImpl}} on RocksDB with
> {{MockNodeManager}} and two real threads. A latch in the container manager
> that the closer uses holds {{closePipeline}} just before it FINALIZEs the
> last container it read, which only fixes the timing. The test thread then
> allocates through {{WritableRatisContainerProvider.getContainer}} (Ratis) or
> {{ContainerManager.allocateContainer}} (EC 3+2, as the admin RPC does).
> Output on unmodified source, the same for both routes:
> {noformat}
> close race: closePipeline read [#1], pipeline state while it runs = OPEN
> close race: after closePipeline: #1 (read by closePipeline) state = CLOSING
> close race: allocation during close handed out #2 on Pipeline-...
> close race: after closePipeline: pipeline state = CLOSED, containers still in
> pipeline = [#2], #2 state = OPEN
> close race: OPEN pipelines offered to new allocations = []
> {noformat}
> No CLOSE_CONTAINER event was fired for #2. A passing test means the defect is
> present.
> h3. Suggested fix
> The patch removes the cause. {{ReplicationManager.hasHealthyPipeline}} could
> in addition treat a CLOSED pipeline (not a DORMANT one, whose containers are
> meant to stay OPEN) as unhealthy, so that any OPEN container left on a CLOSED
> pipeline is closed on the next ReplicationManager run while its Ratis group
> still exists. With the patch the only remaining way I found to get there is
> an SCM crash or leader change between the CLOSED transition and the second
> container pass (argued), so this is not part of the patch.
> h3. Patch
> [^MC-2-close-containers-allocated-during-pipeline-close.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 6818707698c.
> {{closePipeline}} calls {{closeContainersForPipeline}} a second time after
> the pipeline is CLOSED. FINALIZE removes a container from its pipeline, so
> the second call only finds containers added while the first one ran, and
> normally reads an empty set. {{addContainerToOpenPipeline}} checks for CLOSED
> under the same state manager lock as the CLOSED transition, so once the
> pipeline is CLOSED no container can be added and the second call finds every
> late one. The allocation path is unchanged and takes no new lock, and the
> HDDS-5973 order is kept for every container that existed when the close
> started. Holding the write lock from the read to the CLOSED transition would
> also work but blocks all allocation while the FINALIZE calls go through
> Ratis; moving the pipeline to a new non OPEN state before the read would
> change the pipeline state machine. The second call relies on CLOSED being
> final. At this commit {{activatePipeline}} can move a CLOSED pipeline back to
> OPEN, which is reported and fixed separately in HDDS-16889.
> Covered by {{testClosePipelineClosesContainerAllocatedDuringClose}} in the
> existing {{TestPipelineManagerImpl}}, which allocates a container from inside
> the first FINALIZE and checks that both containers are CLOSING, none is left
> in the pipeline and each got exactly one CLOSE_CONTAINER event. Without the
> change it fails with "expected: CLOSING but was: OPEN". With it the SCM
> pipeline, block, container manager, close container, dead node and client
> protocol suites pass (230 tests), as do {{TestPipelineClose}} on a mini
> cluster (5 tests) and the Recon pipeline manager tests (5 tests), and
> checkstyle is clean. With the patch both tests of the reproduction fail,
> because the late container is CLOSING.
> Found by TLA+ model checking and code review of the SCM pipeline lifecycle
> paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. Related but not duplicates: HDDS-5973 (resolved, close order
> for SCM restarts), HDDS-5655 (resolved, read lock around allocation),
> HDDS-7829 (resolved, rejects adding a container to a CLOSED pipeline),
> HDDS-10231 (resolved, ReplicationManager closes OPEN containers whose
> pipeline is missing) and HDDS-11521 (Not A Problem, client wait on pipeline
> creation). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
