Chu Cheng Li created HDDS-17400:
-----------------------------------
Summary: DELETED container moves back to QUASI_CLOSED for a stale
OPEN or CLOSING replica, so its deleted blocks are never removed
Key: HDDS-17400
URL: https://issues.apache.org/jira/browse/HDDS-17400
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
h3. Problem
* SCM moves a DELETING or DELETED Ratis container back to QUASI_CLOSED when a
datanode reports a non-empty OPEN or CLOSING replica
({{AbstractContainerReportHandler.updateContainerState}}, HDDS-12421 and
HDDS-14684). SCM does not compare the replica BCSID with the container sequence
ID first.
* For a CLOSED or QUASI_CLOSED replica at or below the sequence ID, SCM sends a
force delete instead (HDDS-14416, HDDS-14684).
* A datanode that is away while a container is emptied and deleted keeps its
OPEN replica. This replica still holds the deleted blocks, because the datanode
did not get the block deletes.
h3. Example (SCM simulation of HDDS-16627, seed 2039)
# Container #1051 (RATIS/THREE) is OPEN on dn2, dn4 and dn7. dn2 loses its
connection to SCM. SCM marks dn2 dead and removes its replica. The other
replicas become QUASI_CLOSED.
# The client deletes all 6 blocks of the container. The other replicas apply
the deletes. SCM then removes the delete transactions.
# The replication manager deletes the empty container. The container becomes
DELETED.
# dn2 comes back. It reports an OPEN replica with BCSID 171 (the same as the
container) and with the 6 deleted blocks. SCM moves the container to
QUASI_CLOSED.
h3. Effect
* The deleted blocks come back. No delete transaction refers to them, thus
nothing removes them. The replication manager copies them to more datanodes.
* The container is not empty, thus SCM does not delete it again.
* In the example, dn5 also came back later, with an empty replica from a second
origin. The replication manager then wanted 3 copies of each origin, but only 5
datanodes were in service. The container stayed
QUASI_CLOSED_STUCK_UNDER_REPLICATED to the end of the run.
* HDDS-14774 saw the same move in {{TestContainerReportHandling}}, caused by a
late CLOSING report. That fix changed only the test.
h3. Possible fix
* The BCSID of an OPEN or CLOSING replica is not always final, because its
datanode can still apply Ratis log entries. One option: keep the container
DELETED and send a close command. When the replica reports QUASI_CLOSED or
CLOSED, the existing BCSID check decides.
* This needs HDDS-16809 first. {{updateContainerStats}} raises the container
sequence ID before the check. Thus, today the check force-deletes every CLOSED
or QUASI_CLOSED replica, also a replica with newer data.
h3. Test
A new {{TestContainerReportHandler}} case fails on master (f804bf2686) for OPEN
and CLOSING. A non-empty replica with a BCSID equal to the container sequence
ID moves a DELETED container to QUASI_CLOSED. The existing
{{testDeletingContainerResurrectionToQuasiClosedWithOpenReplica}} expects this
behavior.
Related: HDDS-12305 (why SCM lets a container come back), HDDS-13928,
HDDS-16809, HDDS-17060.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
