[
https://issues.apache.org/jira/browse/HDDS-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137691#comment-18137691
]
Siyao Meng commented on HDDS-17643:
-----------------------------------
Thanks [~sumitagrawl], good idea. It looks like
{{ContainerStateMachine.container2BCSIDMap}} could already serve as that cache.
It records every container created over the pipeline, is not cleaned up on
delete, and is saved in the Ratis snapshot.
So the fix could be: in {{HddsDispatcher.dispatchRequest}}, skip the auto
create when the container ID is already in {{container2BCSIDMap}}, and return
CONTAINER_NOT_OPEN, which followers already ignore. That way we wouldn't need a
DB change.
One potential gap, a container created and deleted after the last snapshot
would not be in the map after a restart, so log replay might still recreate it
(similar to HDDS-17435).
For EC, this wouldn't help since there is no state machine. HDDS-15791 turns
off auto create only for reconstruction, so a late EC write might run into the
same issue, though I haven't verified it. Maybe we could fix Ratis here and
track EC separately
> Pipeline leader recreates a Ratis container replica that SCM force deleted,
> and SCM keeps the new copy over the real replicas
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17643
> URL: https://issues.apache.org/jira/browse/HDDS-17643
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, SCM
> Affects Versions: 2.2.2
> Reporter: Siyao Meng
> Priority: Critical
>
> h3. Mechanism
> SCM can delete a CLOSED replica on a datanode that is still the Ratis leader
> of the pipeline the container was written on, for example the source of a
> balancer move ({{MoveManager}} sends a forced delete) or an excess replica.
> Pipelines outlive the containers on them, so writes for that container can
> still reach the leader after the delete.
> {{KeyValueHandler}} deletes the replica through
> {{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
> the container ID table (HDDS-11650) and does not add it to the missing
> container set. Nothing on the datanode remembers that the container existed.
> When a WriteChunk for that container then reaches the leader:
> # {{ContainerStateMachine.startTransaction}} calls
> {{HddsDispatcher.validateContainerCommand}}, which returns without any check
> when the container is not in the container set, so the leader accepts the
> write.
> # In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
> missing container for WriteChunk and PutBlock. HDDS-15791 added
> {{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}}
> sets it to false; {{BlockOutputStream}} for Ratis always allows it. The
> leader creates an empty OPEN replica.
> # The followers still have the container CLOSED and return
> CLOSED_CONTAINER_IO, which {{ContainerStateMachine}} ignores in
> writeStateMachineData and applyTransaction. The entry commits.
> Two outcomes follow:
> * If the block's earlier chunks were written to the deleted replica, PutBlock
> fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
> leader closes the whole pipeline.
> * If all chunks of a block are written after the recreate, PutBlock succeeds
> on the new replica only. SCM then closes the mismatched OPEN replica, and it
> ends CLOSED with a BCSID higher than the real replicas, but holding only the
> blocks written after the recreate.
> SCM cannot tell the new replica from the real ones and prefers it:
> * {{AbstractContainerReportHandler}} raises the CLOSED container's sequence
> ID to the new replica's BCSID (it only logs an error first, see HDDS-16448).
> * {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
> regardless of its BCSID.
> * {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
> {{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
> * {{RatisUnderReplicationHandler}} and the balancer copy from the replica
> with the highest sequence ID, which is the new one.
> Repeated over moves and replica losses, the real replicas are replaced by
> copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
> Once the blocks written after the recreate are deleted, every replica is
> empty, {{EmptyContainerHandler}} deletes the container, and reads fail with
> NO_REPLICA_FOUND while OM still has the keys.
> HDDS-17435 is the same datanode gap reached through StreamInit replay after a
> restart. This one needs no restart and no data stream.
> h3. Trigger
> * Container balancer enabled (or any other forced or excess replica delete).
> * A container closes and SCM deletes its replica on the pipeline leader while
> the pipeline stays open.
> * A client keeps writing to a block in that container afterwards (a writer
> that allocated the block before the close).
> h3. Observed
> Seen on a production cluster running a release that contains HDDS-11650. One
> datanode logged 103 containers that were deleted and then OPEN again with
> BCSID 0, over about 2.5 months. A typical sequence for container C on leader
> D1 (followers D2, D3):
> {noformat}
> 00:00 C closes on D1, D2, D3 at BCSID N
> 00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
> 00:06 D1: C is OPEN, BCSID 0 (recreated)
> 00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
> 00:09 D1: C CLOSED at BCSID N + 824
> {noformat}
> For containers in a window where SCM logs were available, SCM then found four
> replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
> three sampled cases the over replication delete removed a real replica (the
> balancer target, or an original pipeline member) and kept the recreated one.
> In one case all real replicas were eventually removed and the container was
> later deleted as empty while OM still referenced its keys.
> Across all 219 datanodes of the same cluster, 11,420 containers were
> recreated this way over 13 months, on 137 datanodes. Classified by which
> replicas are left:
> * 12: every original replica was deleted and no replica remains.
> * 32: only recreated replicas remain.
> * 5,553: original and recreated replicas both remain.
> h3. Reproduction
> Not run yet. A mini cluster test should be able to show it: write a key so
> that the pipeline leader holds the container, close the container, force
> delete the replica on the leader with a delete container command while
> keeping the pipeline, then write another block into the same container ID
> through the same pipeline and check that the leader recreates the container
> and that the write succeeds.
> h3. Suggested fix
> Datanode, Ratis: {{ContainerStateMachine.container2BCSIDMap}} already records
> every container created over the pipeline, is not cleaned up on delete, and
> is saved in the Ratis snapshot. In the auto create path of
> {{dispatchRequest}}, refuse to create a container whose ID is already in that
> map. Return CONTAINER_NOT_OPEN, which followers already ignore in
> writeStateMachineData and applyTransaction (CONTAINER_NOT_FOUND would fail
> the pipeline on a follower). The client gets an error and allocates a new
> block. This needs no DB change. It does not cover a container created and
> deleted after the last snapshot and then replayed after a restart (see
> HDDS-17435).
> Datanode, general alternative: keep the ID of a deleted replica as a
> tombstone in the container ID table (HDDS-11650) and reject it the same way.
> Container import and EC RECOVERING create must clear the tombstone. This also
> covers the restart replay and EC, at the cost of tombstones that need cleanup.
> SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
> (HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
> container's sequence ID as mismatched instead of healthy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
