[
https://issues.apache.org/jira/browse/HDDS-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17643:
------------------------------
Description:
h3. Mechanism
SCM can delete a CLOSED replica on a datanode that is still the Ratis leader of
the pipeline the container was written on, for example the source of a balancer
move ({{MoveManager}} sends a forced delete) or an excess replica. Pipelines
outlive the containers on them, so writes for that container can still reach
the leader after the delete.
{{KeyValueHandler}} deletes the replica through
{{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
the container ID table (HDDS-11650) and does not add it to the missing
container set. Nothing on the datanode remembers that the container existed.
When a WriteChunk for that container then reaches the leader:
# {{ContainerStateMachine.startTransaction}} calls
{{HddsDispatcher.validateContainerCommand}}, which returns without any check
when the container is not in the container set, so the leader accepts the write.
# In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
missing container for WriteChunk and PutBlock. HDDS-15791 added
{{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}} sets
it to false; {{BlockOutputStream}} for Ratis always allows it. The leader
creates an empty OPEN replica.
# The followers still have the container CLOSED and return CLOSED_CONTAINER_IO,
which {{ContainerStateMachine}} ignores in writeStateMachineData and
applyTransaction. The entry commits.
Two outcomes follow:
* If the block's earlier chunks were written to the deleted replica, PutBlock
fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
leader closes the whole pipeline.
* If all chunks of a block are written after the recreate, PutBlock succeeds on
the new replica only. SCM then closes the mismatched OPEN replica, and it ends
CLOSED with a BCSID higher than the real replicas, but holding only the blocks
written after the recreate.
SCM cannot tell the new replica from the real ones and prefers it:
* {{AbstractContainerReportHandler}} raises the CLOSED container's sequence ID
to the new replica's BCSID (it only logs an error first, see HDDS-16448).
* {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
regardless of its BCSID.
* {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
{{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
* {{RatisUnderReplicationHandler}} and the balancer copy from the replica with
the highest sequence ID, which is the new one.
Repeated over moves and replica losses, the real replicas are replaced by
copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
Once the blocks written after the recreate are deleted, every replica is empty,
{{EmptyContainerHandler}} deletes the container, and reads fail with
NO_REPLICA_FOUND while OM still has the keys.
HDDS-17435 is the same datanode gap reached through StreamInit replay after a
restart. This one needs no restart and no data stream.
h3. Trigger
* Container balancer enabled (or any other forced or excess replica delete).
* A container closes and SCM deletes its replica on the pipeline leader while
the pipeline stays open.
* A client keeps writing to a block in that container afterwards (a writer that
allocated the block before the close).
h3. Observed
Seen on a production cluster running a release that contains HDDS-11650. One
datanode logged 103 containers that were deleted and then OPEN again with BCSID
0, over about 2.5 months. A typical sequence for container C on leader D1
(followers D2, D3):
{noformat}
00:00 C closes on D1, D2, D3 at BCSID N
00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
00:06 D1: C is OPEN, BCSID 0 (recreated)
00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
00:09 D1: C CLOSED at BCSID N + 824
{noformat}
For containers in a window where SCM logs were available, SCM then found four
replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
three sampled cases the over replication delete removed a real replica (the
balancer target, or an original pipeline member) and kept the recreated one. In
one case all real replicas were eventually removed and the container was later
deleted as empty while OM still referenced its keys.
Across all 219 datanodes of the same cluster, 11,420 containers were recreated
this way over 13 months, on 137 datanodes. Classified by which replicas are
left:
* 12: every original replica was deleted and no replica remains.
* 32: only recreated replicas remain.
* 5,553: original and recreated replicas both remain.
h3. Reproduction
Reproduced on master (e345940c289) with
{{TestCloseContainerByPipeline#testWriteAfterForceDeleteOnLeaderRecreatesContainer}},
attached as HDDS-17643-repro.patch. The test passes on master, which shows the
bug:
# Write a key with RATIS/THREE and close its container on all three datanodes.
# Send a forced delete container command to the pipeline leader only. The
pipeline stays open.
# Through the same pipeline, send a WriteChunk and a PutBlock for a new block
in the same container ID.
Both writes return SUCCESS. The followers reject them because the container is
CLOSED, and that is ignored. The leader recreates the container: OPEN, 1 block,
BCSID above the closed replicas. The followers stay CLOSED at the old BCSID.
{noformat}
leader: WARN keyvalue.KeyValueContainer - Marked container DELETED from
CLOSED: KeyValueContainerData #1 (DELETED, non-empty, ri=0, ...)
follower: ERROR op=WRITE_CHUNK {"blockData":"conID: 1 locID: 117556328051441666
bcsId: 0 replicaIndex: null","offset":"0","size":"1024","stage":"WRITE_DATA"} |
ret=FAILURE
java.lang.Exception: Requested operation not allowed as
ContainerState is CLOSED
follower: ERROR op=PUT_BLOCK {"blockData":"[blockId=conID: 1 locID:
117556328051441666 bcsId: 0 replicaIndex: null, size=1024]"} | ret=FAILURE
{noformat}
h3. Suggested fix
Datanode, Ratis: {{ContainerStateMachine.container2BCSIDMap}} already records
every container created over the pipeline, is not cleaned up on delete, and is
saved in the Ratis snapshot. In the auto create path of {{dispatchRequest}},
refuse to create a container whose ID is already in that map. Return
CONTAINER_NOT_OPEN, which followers already ignore in writeStateMachineData and
applyTransaction (CONTAINER_NOT_FOUND would fail the pipeline on a follower).
The client gets an error and allocates a new block. This needs no DB change. It
does not cover a container created and deleted after the last snapshot and then
replayed after a restart (see HDDS-17435).
Datanode, general alternative: keep the ID of a deleted replica as a tombstone
in the container ID table (HDDS-11650) and reject it the same way. Container
import and EC RECOVERING create must clear the tombstone. This also covers the
restart replay and EC, at the cost of tombstones that need cleanup.
SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
(HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
container's sequence ID as mismatched instead of healthy.
was:
h3. Mechanism
SCM can delete a CLOSED replica on a datanode that is still the Ratis leader of
the pipeline the container was written on, for example the source of a balancer
move ({{MoveManager}} sends a forced delete) or an excess replica. Pipelines
outlive the containers on them, so writes for that container can still reach
the leader after the delete.
{{KeyValueHandler}} deletes the replica through
{{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
the container ID table (HDDS-11650) and does not add it to the missing
container set. Nothing on the datanode remembers that the container existed.
When a WriteChunk for that container then reaches the leader:
# {{ContainerStateMachine.startTransaction}} calls
{{HddsDispatcher.validateContainerCommand}}, which returns without any check
when the container is not in the container set, so the leader accepts the write.
# In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
missing container for WriteChunk and PutBlock. HDDS-15791 added
{{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}} sets
it to false; {{BlockOutputStream}} for Ratis always allows it. The leader
creates an empty OPEN replica.
# The followers still have the container CLOSED and return CLOSED_CONTAINER_IO,
which {{ContainerStateMachine}} ignores in writeStateMachineData and
applyTransaction. The entry commits.
Two outcomes follow:
* If the block's earlier chunks were written to the deleted replica, PutBlock
fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
leader closes the whole pipeline.
* If all chunks of a block are written after the recreate, PutBlock succeeds on
the new replica only. SCM then closes the mismatched OPEN replica, and it ends
CLOSED with a BCSID higher than the real replicas, but holding only the blocks
written after the recreate.
SCM cannot tell the new replica from the real ones and prefers it:
* {{AbstractContainerReportHandler}} raises the CLOSED container's sequence ID
to the new replica's BCSID (it only logs an error first, see HDDS-16448).
* {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
regardless of its BCSID.
* {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
{{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
* {{RatisUnderReplicationHandler}} and the balancer copy from the replica with
the highest sequence ID, which is the new one.
Repeated over moves and replica losses, the real replicas are replaced by
copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
Once the blocks written after the recreate are deleted, every replica is empty,
{{EmptyContainerHandler}} deletes the container, and reads fail with
NO_REPLICA_FOUND while OM still has the keys.
HDDS-17435 is the same datanode gap reached through StreamInit replay after a
restart. This one needs no restart and no data stream.
h3. Trigger
* Container balancer enabled (or any other forced or excess replica delete).
* A container closes and SCM deletes its replica on the pipeline leader while
the pipeline stays open.
* A client keeps writing to a block in that container afterwards (a writer that
allocated the block before the close).
h3. Observed
Seen on a production cluster running a release that contains HDDS-11650. One
datanode logged 103 containers that were deleted and then OPEN again with BCSID
0, over about 2.5 months. A typical sequence for container C on leader D1
(followers D2, D3):
{noformat}
00:00 C closes on D1, D2, D3 at BCSID N
00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
00:06 D1: C is OPEN, BCSID 0 (recreated)
00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
00:09 D1: C CLOSED at BCSID N + 824
{noformat}
For containers in a window where SCM logs were available, SCM then found four
replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
three sampled cases the over replication delete removed a real replica (the
balancer target, or an original pipeline member) and kept the recreated one. In
one case all real replicas were eventually removed and the container was later
deleted as empty while OM still referenced its keys.
Across all 219 datanodes of the same cluster, 11,420 containers were recreated
this way over 13 months, on 137 datanodes. Classified by which replicas are
left:
* 12: every original replica was deleted and no replica remains.
* 32: only recreated replicas remain.
* 5,553: original and recreated replicas both remain.
h3. Reproduction
Not run yet. A mini cluster test should be able to show it: write a key so that
the pipeline leader holds the container, close the container, force delete the
replica on the leader with a delete container command while keeping the
pipeline, then write another block into the same container ID through the same
pipeline and check that the leader recreates the container and that the write
succeeds.
h3. Suggested fix
Datanode, Ratis: {{ContainerStateMachine.container2BCSIDMap}} already records
every container created over the pipeline, is not cleaned up on delete, and is
saved in the Ratis snapshot. In the auto create path of {{dispatchRequest}},
refuse to create a container whose ID is already in that map. Return
CONTAINER_NOT_OPEN, which followers already ignore in writeStateMachineData and
applyTransaction (CONTAINER_NOT_FOUND would fail the pipeline on a follower).
The client gets an error and allocates a new block. This needs no DB change. It
does not cover a container created and deleted after the last snapshot and then
replayed after a restart (see HDDS-17435).
Datanode, general alternative: keep the ID of a deleted replica as a tombstone
in the container ID table (HDDS-11650) and reject it the same way. Container
import and EC RECOVERING create must clear the tombstone. This also covers the
restart replay and EC, at the cost of tombstones that need cleanup.
SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
(HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
container's sequence ID as mismatched instead of healthy.
> Pipeline leader recreates a Ratis container replica that SCM force deleted,
> and SCM keeps the new copy over the real replicas
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17643
> URL: https://issues.apache.org/jira/browse/HDDS-17643
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, SCM
> Affects Versions: 2.2.2
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-17643-repro.patch
>
>
> h3. Mechanism
> SCM can delete a CLOSED replica on a datanode that is still the Ratis leader
> of the pipeline the container was written on, for example the source of a
> balancer move ({{MoveManager}} sends a forced delete) or an excess replica.
> Pipelines outlive the containers on them, so writes for that container can
> still reach the leader after the delete.
> {{KeyValueHandler}} deletes the replica through
> {{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
> the container ID table (HDDS-11650) and does not add it to the missing
> container set. Nothing on the datanode remembers that the container existed.
> When a WriteChunk for that container then reaches the leader:
> # {{ContainerStateMachine.startTransaction}} calls
> {{HddsDispatcher.validateContainerCommand}}, which returns without any check
> when the container is not in the container set, so the leader accepts the
> write.
> # In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
> missing container for WriteChunk and PutBlock. HDDS-15791 added
> {{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}}
> sets it to false; {{BlockOutputStream}} for Ratis always allows it. The
> leader creates an empty OPEN replica.
> # The followers still have the container CLOSED and return
> CLOSED_CONTAINER_IO, which {{ContainerStateMachine}} ignores in
> writeStateMachineData and applyTransaction. The entry commits.
> Two outcomes follow:
> * If the block's earlier chunks were written to the deleted replica, PutBlock
> fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
> leader closes the whole pipeline.
> * If all chunks of a block are written after the recreate, PutBlock succeeds
> on the new replica only. SCM then closes the mismatched OPEN replica, and it
> ends CLOSED with a BCSID higher than the real replicas, but holding only the
> blocks written after the recreate.
> SCM cannot tell the new replica from the real ones and prefers it:
> * {{AbstractContainerReportHandler}} raises the CLOSED container's sequence
> ID to the new replica's BCSID (it only logs an error first, see HDDS-16448).
> * {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
> regardless of its BCSID.
> * {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
> {{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
> * {{RatisUnderReplicationHandler}} and the balancer copy from the replica
> with the highest sequence ID, which is the new one.
> Repeated over moves and replica losses, the real replicas are replaced by
> copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
> Once the blocks written after the recreate are deleted, every replica is
> empty, {{EmptyContainerHandler}} deletes the container, and reads fail with
> NO_REPLICA_FOUND while OM still has the keys.
> HDDS-17435 is the same datanode gap reached through StreamInit replay after a
> restart. This one needs no restart and no data stream.
> h3. Trigger
> * Container balancer enabled (or any other forced or excess replica delete).
> * A container closes and SCM deletes its replica on the pipeline leader while
> the pipeline stays open.
> * A client keeps writing to a block in that container afterwards (a writer
> that allocated the block before the close).
> h3. Observed
> Seen on a production cluster running a release that contains HDDS-11650. One
> datanode logged 103 containers that were deleted and then OPEN again with
> BCSID 0, over about 2.5 months. A typical sequence for container C on leader
> D1 (followers D2, D3):
> {noformat}
> 00:00 C closes on D1, D2, D3 at BCSID N
> 00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
> 00:06 D1: C is OPEN, BCSID 0 (recreated)
> 00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
> 00:09 D1: C CLOSED at BCSID N + 824
> {noformat}
> For containers in a window where SCM logs were available, SCM then found four
> replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
> three sampled cases the over replication delete removed a real replica (the
> balancer target, or an original pipeline member) and kept the recreated one.
> In one case all real replicas were eventually removed and the container was
> later deleted as empty while OM still referenced its keys.
> Across all 219 datanodes of the same cluster, 11,420 containers were
> recreated this way over 13 months, on 137 datanodes. Classified by which
> replicas are left:
> * 12: every original replica was deleted and no replica remains.
> * 32: only recreated replicas remain.
> * 5,553: original and recreated replicas both remain.
> h3. Reproduction
> Reproduced on master (e345940c289) with
> {{TestCloseContainerByPipeline#testWriteAfterForceDeleteOnLeaderRecreatesContainer}},
> attached as HDDS-17643-repro.patch. The test passes on master, which shows
> the bug:
> # Write a key with RATIS/THREE and close its container on all three datanodes.
> # Send a forced delete container command to the pipeline leader only. The
> pipeline stays open.
> # Through the same pipeline, send a WriteChunk and a PutBlock for a new block
> in the same container ID.
> Both writes return SUCCESS. The followers reject them because the container
> is CLOSED, and that is ignored. The leader recreates the container: OPEN, 1
> block, BCSID above the closed replicas. The followers stay CLOSED at the old
> BCSID.
> {noformat}
> leader: WARN keyvalue.KeyValueContainer - Marked container DELETED from
> CLOSED: KeyValueContainerData #1 (DELETED, non-empty, ri=0, ...)
> follower: ERROR op=WRITE_CHUNK {"blockData":"conID: 1 locID:
> 117556328051441666 bcsId: 0 replicaIndex:
> null","offset":"0","size":"1024","stage":"WRITE_DATA"} | ret=FAILURE
> java.lang.Exception: Requested operation not allowed as
> ContainerState is CLOSED
> follower: ERROR op=PUT_BLOCK {"blockData":"[blockId=conID: 1 locID:
> 117556328051441666 bcsId: 0 replicaIndex: null, size=1024]"} | ret=FAILURE
> {noformat}
> h3. Suggested fix
> Datanode, Ratis: {{ContainerStateMachine.container2BCSIDMap}} already records
> every container created over the pipeline, is not cleaned up on delete, and
> is saved in the Ratis snapshot. In the auto create path of
> {{dispatchRequest}}, refuse to create a container whose ID is already in that
> map. Return CONTAINER_NOT_OPEN, which followers already ignore in
> writeStateMachineData and applyTransaction (CONTAINER_NOT_FOUND would fail
> the pipeline on a follower). The client gets an error and allocates a new
> block. This needs no DB change. It does not cover a container created and
> deleted after the last snapshot and then replayed after a restart (see
> HDDS-17435).
> Datanode, general alternative: keep the ID of a deleted replica as a
> tombstone in the container ID table (HDDS-11650) and reject it the same way.
> Container import and EC RECOVERING create must clear the tombstone. This also
> covers the restart replay and EC, at the cost of tombstones that need cleanup.
> SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
> (HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
> container's sequence ID as mismatched instead of healthy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
