Siyao Meng created HDDS-16897:
---------------------------------
Summary: Datanode runs a resent DeleteContainerCommand against a
replica imported after the command arrived, so a balancer move can leave the
container under replicated while reporting COMPLETED
Key: HDDS-16897
URL: https://issues.apache.org/jira/browse/HDDS-16897
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-skip-delete-of-container-added-after-command-received.patch,
TestBugMC6StaleDeleteWipesMoveTarget.java
h3. Mechanism
SCM completes a pending DELETE op as soon as a report says the replica is gone
({{ContainerStateManagerImpl.removeContainerReplica}} calls
{{ContainerReplicaPendingOps.completeDeleteReplica}}), whichever command
removed it. Since HDDS-12127, {{ReplicationManager.opCompleted}} resends an
expired DELETE as the same command with a new deadline. If the first copy
already ran on the datanode but its DELETED report reaches SCM only after the
SCM deadline, SCM resends the command, and the late report then completes the
op while the resent copy is still queued on the datanode. SCM no longer knows
about that copy.
{{MoveManager.move}} checks the replicas and pending ops SCM knows of, but not
commands already handed to the datanode, so it accepts that datanode as a move
target. When the copy has been imported and reported,
{{MoveManager.handleSuccessfulAdd}} sees a healthy replica set and deletes the
source. The queued command then runs:
{{DeleteContainerCommandHandler.handleInternal}} checks only the deadline and
the SCM term and deletes by container ID, so it force deletes the replica the
move just created.
h3. Trigger
With the default timeouts ({{hdds.scm.replication.event.timeout}} 12 minutes,
datanode offset 6 minutes):
# A CLOSED RATIS/THREE container has 4 replicas. ReplicationManager sends a
force DELETE to datanode V.
# V runs it, but cannot reach SCM for long enough that the DELETED report
arrives after the 12 minute SCM deadline (so at least about 6 minutes, and less
than the dead node interval).
# The scrubber resends the DELETE to V.
# V reconnects. The heartbeat carries the late report and the response carries
the resent DELETE, which waits behind other work in V's delete container
threads (2 by default).
# Within the 6 minute datanode deadline of the resent command, the balancer
moves this container to V, the copy completes and MoveManager deletes the
source.
# The queued DELETE runs on V and deletes the imported replica.
h3. Impact
The container is left with 2 of 3 replicas while the balancer is told the move
{{COMPLETED}}. ReplicationManager re-replicates from the remaining two, so
there is no data loss on its own, only a window with reduced redundancy. The
trigger needs a long report delay, a backed up delete queue and the balancer
choosing that container and datanode within a few minutes, so it is unlikely.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC6StaleDeleteWipesMoveTarget.java] ({{hadoop-hdds/server-scm}}) runs
the real ReplicationManager (over replication through its own health check and
{{OverReplicatedProcessor}}), {{ContainerReplicaPendingOps}},
{{ContainerManagerImpl}}, {{IncrementalContainerReportHandler}} and
{{MoveManager}}, and gives each datanode a real
{{DeleteContainerCommandHandler}}, {{ContainerController}} and
{{ContainerSet}}. Only the disk work of the container handler is mocked. The
test controls when the delayed report is processed and keeps V's delete thread
(1 thread in the test) busy until the moved replica is imported. Output on the
unmodified source:
{noformat}
late ICR completed (removed) the pending DELETE, DELETE #2 still queued on V
(V=d2) | scmReplicas=[d1, d3, d4] pendingOps=[] dnHas=[d1,d3,d4] V.deleteQueue=1
MoveManager accepted move d1 -> d2 (V=d2) | scmReplicas=[d1, d3, d4]
pendingOps=[ADD@d2] dnHas=[d1,d3,d4] V.deleteQueue=1
ADD completed, MoveManager sent DELETE to source d1 (V=d2) | scmReplicas=[d1,
d2, d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d2,d3,d4] V.deleteQueue=1
V delete thread drained, delete calls on V = 2 (V=d2) | scmReplicas=[d1, d2,
d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d3,d4] V.deleteQueue=0
RESULT: move=COMPLETED dnReplicas=2 scmReplicas=[d3, d4]
health=UNDER_REPLICATED (RF 3)
{noformat}
A passing test means the defect is present. The heartbeat transport and the
datanode disk work are simulated.
h3. Suggested fix
The patch makes the datanode skip a delete for a container that was not present
when the command arrived but is present when it runs. Other ways a queued
delete can meet a newer replica (the replica disappears for another reason,
such as a failed volume, while the original command is still queued) are not
covered. An SCM side alternative would keep a resent DELETE's target excluded
from moves until the command's datanode deadline has passed, which needs new
state in SCM; it is not part of this patch.
h3. Patch
[^MC-6-skip-delete-of-container-added-after-command-received.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
{{DeleteContainerCommandHandler.handle}} records whether the container exists
when the command is received. If it did not, and it exists when the command
runs, the command is skipped with an INFO log. In every other case the
behaviour is unchanged: an absent container is still a no-op, and a replica
that was present on receipt is deleted as before, including when the container
object is replaced in between (for example by the disk balancer). No protocol
change.
The resend comes from HDDS-12127. HDDS-16447 is the same race on the pending op
in the opposite order (a report removes the op, then the resend inserts it
again); both follow from SCM tracking a DELETE per target and replica index
rather than per command. This patch also keeps the orphan resends described
there from deleting a later replica on that datanode, because the container is
absent when they arrive. The linked issue covers the other defect of the same
resend (sent without rechecking that other copies remain), which needs an SCM
side fix.
Covered by {{testContainerAddedAfterCommandReceivedIsNotDeleted}} and
{{testContainerPresentWhenCommandReceivedIsDeleted}} in the existing
{{TestDeleteContainerCommandHandler}}. Without the change the first fails
({{NeverWantedButInvoked}} on {{deleteContainer}}). With it the datanode state
machine and command handler unit suites pass and checkstyle is clean. With the
patch the reproduction fails: the queued DELETE is skipped, the moved replica
survives and the container ends with 3 replicas, HEALTHY.
Found by TLA+ model checking and code review of the SCM container move and
balancer paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
