[
https://issues.apache.org/jira/browse/HDDS-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17643:
------------------------------
Attachment: HDDS-17643-repro.patch
> Pipeline leader recreates a Ratis container replica that SCM force deleted,
> and SCM keeps the new copy over the real replicas
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17643
> URL: https://issues.apache.org/jira/browse/HDDS-17643
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, SCM
> Affects Versions: 2.2.2
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-17643-repro.patch
>
>
> h3. Mechanism
> SCM can delete a CLOSED replica on a datanode that is still the Ratis leader
> of the pipeline the container was written on, for example the source of a
> balancer move ({{MoveManager}} sends a forced delete) or an excess replica.
> Pipelines outlive the containers on them, so writes for that container can
> still reach the leader after the delete.
> {{KeyValueHandler}} deletes the replica through
> {{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
> the container ID table (HDDS-11650) and does not add it to the missing
> container set. Nothing on the datanode remembers that the container existed.
> When a WriteChunk for that container then reaches the leader:
> # {{ContainerStateMachine.startTransaction}} calls
> {{HddsDispatcher.validateContainerCommand}}, which returns without any check
> when the container is not in the container set, so the leader accepts the
> write.
> # In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
> missing container for WriteChunk and PutBlock. HDDS-15791 added
> {{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}}
> sets it to false; {{BlockOutputStream}} for Ratis always allows it. The
> leader creates an empty OPEN replica.
> # The followers still have the container CLOSED and return
> CLOSED_CONTAINER_IO, which {{ContainerStateMachine}} ignores in
> writeStateMachineData and applyTransaction. The entry commits.
> Two outcomes follow:
> * If the block's earlier chunks were written to the deleted replica, PutBlock
> fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
> leader closes the whole pipeline.
> * If all chunks of a block are written after the recreate, PutBlock succeeds
> on the new replica only. SCM then closes the mismatched OPEN replica, and it
> ends CLOSED with a BCSID higher than the real replicas, but holding only the
> blocks written after the recreate.
> SCM cannot tell the new replica from the real ones and prefers it:
> * {{AbstractContainerReportHandler}} raises the CLOSED container's sequence
> ID to the new replica's BCSID (it only logs an error first, see HDDS-16448).
> * {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
> regardless of its BCSID.
> * {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
> {{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
> * {{RatisUnderReplicationHandler}} and the balancer copy from the replica
> with the highest sequence ID, which is the new one.
> Repeated over moves and replica losses, the real replicas are replaced by
> copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
> Once the blocks written after the recreate are deleted, every replica is
> empty, {{EmptyContainerHandler}} deletes the container, and reads fail with
> NO_REPLICA_FOUND while OM still has the keys.
> HDDS-17435 is the same datanode gap reached through StreamInit replay after a
> restart. This one needs no restart and no data stream.
> h3. Trigger
> * Container balancer enabled (or any other forced or excess replica delete).
> * A container closes and SCM deletes its replica on the pipeline leader while
> the pipeline stays open.
> * A client keeps writing to a block in that container afterwards (a writer
> that allocated the block before the close).
> h3. Observed
> Seen on a production cluster running a release that contains HDDS-11650. One
> datanode logged 103 containers that were deleted and then OPEN again with
> BCSID 0, over about 2.5 months. A typical sequence for container C on leader
> D1 (followers D2, D3):
> {noformat}
> 00:00 C closes on D1, D2, D3 at BCSID N
> 00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
> 00:06 D1: C is OPEN, BCSID 0 (recreated)
> 00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
> 00:09 D1: C CLOSED at BCSID N + 824
> {noformat}
> For containers in a window where SCM logs were available, SCM then found four
> replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
> three sampled cases the over replication delete removed a real replica (the
> balancer target, or an original pipeline member) and kept the recreated one.
> In one case all real replicas were eventually removed and the container was
> later deleted as empty while OM still referenced its keys.
> Across all 219 datanodes of the same cluster, 11,420 containers were
> recreated this way over 13 months, on 137 datanodes. Classified by which
> replicas are left:
> * 12: every original replica was deleted and no replica remains.
> * 32: only recreated replicas remain.
> * 5,553: original and recreated replicas both remain.
> h3. Reproduction
> Not run yet. A mini cluster test should be able to show it: write a key so
> that the pipeline leader holds the container, close the container, force
> delete the replica on the leader with a delete container command while
> keeping the pipeline, then write another block into the same container ID
> through the same pipeline and check that the leader recreates the container
> and that the write succeeds.
> h3. Suggested fix
> Datanode, Ratis: {{ContainerStateMachine.container2BCSIDMap}} already records
> every container created over the pipeline, is not cleaned up on delete, and
> is saved in the Ratis snapshot. In the auto create path of
> {{dispatchRequest}}, refuse to create a container whose ID is already in that
> map. Return CONTAINER_NOT_OPEN, which followers already ignore in
> writeStateMachineData and applyTransaction (CONTAINER_NOT_FOUND would fail
> the pipeline on a follower). The client gets an error and allocates a new
> block. This needs no DB change. It does not cover a container created and
> deleted after the last snapshot and then replayed after a restart (see
> HDDS-17435).
> Datanode, general alternative: keep the ID of a deleted replica as a
> tombstone in the container ID table (HDDS-11650) and reject it the same way.
> Container import and EC RECOVERING create must clear the tombstone. This also
> covers the restart replay and EC, at the cost of tombstones that need cleanup.
> SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
> (HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
> container's sequence ID as mismatched instead of healthy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
