[
https://issues.apache.org/jira/browse/HDDS-16897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16897:
------------------------------
Attachment: MC-6-skip-delete-of-container-added-after-command-received.patch
> Datanode runs a resent DeleteContainerCommand against a replica imported
> after the command arrived, so a balancer move can leave the container under
> replicated while reporting COMPLETED
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16897
> URL: https://issues.apache.org/jira/browse/HDDS-16897
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-6-skip-delete-of-container-added-after-command-received.patch,
> TestBugMC6StaleDeleteWipesMoveTarget.java
>
>
> h3. Mechanism
> SCM completes a pending DELETE op as soon as a report says the replica is
> gone ({{ContainerStateManagerImpl.removeContainerReplica}} calls
> {{ContainerReplicaPendingOps.completeDeleteReplica}}), whichever command
> removed it. Since HDDS-12127, {{ReplicationManager.opCompleted}} resends an
> expired DELETE as the same command with a new deadline. If the first copy
> already ran on the datanode but its DELETED report reaches SCM only after the
> SCM deadline, SCM resends the command, and the late report then completes the
> op while the resent copy is still queued on the datanode. SCM no longer knows
> about that copy.
> {{MoveManager.move}} checks the replicas and pending ops SCM knows of, but
> not commands already handed to the datanode, so it accepts that datanode as a
> move target. When the copy has been imported and reported,
> {{MoveManager.handleSuccessfulAdd}} sees a healthy replica set and deletes
> the source. The queued command then runs:
> {{DeleteContainerCommandHandler.handleInternal}} checks only the deadline and
> the SCM term and deletes by container ID, so it force deletes the replica the
> move just created.
> h3. Trigger
> With the default timeouts ({{hdds.scm.replication.event.timeout}} 12 minutes,
> datanode offset 6 minutes):
> # A CLOSED RATIS/THREE container has 4 replicas. ReplicationManager sends a
> force DELETE to datanode V.
> # V runs it, but cannot reach SCM for long enough that the DELETED report
> arrives after the 12 minute SCM deadline (so at least about 6 minutes, and
> less than the dead node interval).
> # The scrubber resends the DELETE to V.
> # V reconnects. The heartbeat carries the late report and the response
> carries the resent DELETE, which waits behind other work in V's delete
> container threads (2 by default).
> # Within the 6 minute datanode deadline of the resent command, the balancer
> moves this container to V, the copy completes and MoveManager deletes the
> source.
> # The queued DELETE runs on V and deletes the imported replica.
> h3. Impact
> The container is left with 2 of 3 replicas while the balancer is told the
> move {{COMPLETED}}. ReplicationManager re-replicates from the remaining two,
> so there is no data loss on its own, only a window with reduced redundancy.
> The trigger needs a long report delay, a backed up delete queue and the
> balancer choosing that container and datanode within a few minutes, so it is
> unlikely.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC6StaleDeleteWipesMoveTarget.java] ({{hadoop-hdds/server-scm}})
> runs the real ReplicationManager (over replication through its own health
> check and {{OverReplicatedProcessor}}), {{ContainerReplicaPendingOps}},
> {{ContainerManagerImpl}}, {{IncrementalContainerReportHandler}} and
> {{MoveManager}}, and gives each datanode a real
> {{DeleteContainerCommandHandler}}, {{ContainerController}} and
> {{ContainerSet}}. Only the disk work of the container handler is mocked. The
> test controls when the delayed report is processed and keeps V's delete
> thread (1 thread in the test) busy until the moved replica is imported.
> Output on the unmodified source:
> {noformat}
> late ICR completed (removed) the pending DELETE, DELETE #2 still queued on V
> (V=d2) | scmReplicas=[d1, d3, d4] pendingOps=[] dnHas=[d1,d3,d4]
> V.deleteQueue=1
> MoveManager accepted move d1 -> d2 (V=d2) | scmReplicas=[d1, d3, d4]
> pendingOps=[ADD@d2] dnHas=[d1,d3,d4] V.deleteQueue=1
> ADD completed, MoveManager sent DELETE to source d1 (V=d2) | scmReplicas=[d1,
> d2, d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d2,d3,d4] V.deleteQueue=1
> V delete thread drained, delete calls on V = 2 (V=d2) | scmReplicas=[d1, d2,
> d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d3,d4] V.deleteQueue=0
> RESULT: move=COMPLETED dnReplicas=2 scmReplicas=[d3, d4]
> health=UNDER_REPLICATED (RF 3)
> {noformat}
> A passing test means the defect is present. The heartbeat transport and the
> datanode disk work are simulated.
> h3. Suggested fix
> The patch makes the datanode skip a delete for a container that was not
> present when the command arrived but is present when it runs. Other ways a
> queued delete can meet a newer replica (the replica disappears for another
> reason, such as a failed volume, while the original command is still queued)
> are not covered. An SCM side alternative would keep a resent DELETE's target
> excluded from moves until the command's datanode deadline has passed, which
> needs new state in SCM; it is not part of this patch.
> h3. Patch
> [^MC-6-skip-delete-of-container-added-after-command-received.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 6818707698c.
> {{DeleteContainerCommandHandler.handle}} records whether the container exists
> when the command is received. If it did not, and it exists when the command
> runs, the command is skipped with an INFO log. In every other case the
> behaviour is unchanged: an absent container is still a no-op, and a replica
> that was present on receipt is deleted as before, including when the
> container object is replaced in between (for example by the disk balancer).
> No protocol change.
> The resend comes from HDDS-12127. HDDS-16447 is the same race on the pending
> op in the opposite order (a report removes the op, then the resend inserts it
> again); both follow from SCM tracking a DELETE per target and replica index
> rather than per command. This patch also keeps the orphan resends described
> there from deleting a later replica on that datanode, because the container
> is absent when they arrive. The linked issue covers the other defect of the
> same resend (sent without rechecking that other copies remain), which needs
> an SCM side fix.
> Covered by {{testContainerAddedAfterCommandReceivedIsNotDeleted}} and
> {{testContainerPresentWhenCommandReceivedIsDeleted}} in the existing
> {{TestDeleteContainerCommandHandler}}. Without the change the first fails
> ({{NeverWantedButInvoked}} on {{deleteContainer}}). With it the datanode
> state machine and command handler unit suites pass and checkstyle is clean.
> With the patch the reproduction fails: the queued DELETE is skipped, the
> moved replica survives and the container ends with 3 replicas, HEALTHY.
> Found by TLA+ model checking and code review of the SCM container move and
> balancer paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
