[
https://issues.apache.org/jira/browse/HDDS-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16144:
------------------------------
Attachment: HDDS-16144.001.patch
> SCM may purge a delete transaction before a newly copied replica has
> acknowledged it
> ------------------------------------------------------------------------------------
>
> Key: HDDS-16144
> URL: https://issues.apache.org/jira/browse/HDDS-16144
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16144.001.patch
>
>
> {{SCMDeletedBlockTransactionStatusManager.commitTransactions}} decides
> whether a delete transaction can be purged from the deleted-block log using a
> replica set snapshot taken earlier in the same call, and then removes the
> transaction from the durable log without re-checking membership. A replica
> that is copied to a new datanode after the snapshot but before the purge, and
> that never received or acknowledged the deletion, can therefore keep the
> blocks forever: SCM has already purged the transaction and never re-issues it.
> h3. Steps to reproduce
> # A delete transaction for a container is outstanding and has been sent to
> the current replicas.
> # {{commitTransactions}} captures the replica set via
> {{containerManager.getContainerReplicas(containerId)}} and evaluates
> {{canPurge}} from that snapshot (all snapshot replicas have committed).
> # Around the same time, replication manager copies the container to a new
> datanode. The copy carries the container RocksDB, so the new replica inherits
> whatever delete state the source had; if the source had not yet physically
> deleted the blocks, the new replica holds them with no pending SCM delete
> transaction.
> # {{commitTransactions}} proceeds to purge:
> {{transactionToDNsCommitMap.remove(txID)}} and
> {{removeTransactions(txIDsToBeDeleted)}} delete the row from the durable
> deleted-block log. There is no re-validation of the current replica set at
> removal time.
> The transaction is gone from the SCM log and is never recreated. SCM does not
> reconcile a replica reported {{deleteTransactionId}} against the container
> and does not re-issue delete transactions from it (the field is effectively
> unused in server-scm), so the lagging replica is never told to delete.
> h3. Root cause
> The purge decision and the durable removal are not guarded by a consistent,
> current replica set. {{commitTransactions}} evaluates {{canPurge}} against a
> replica snapshot and then {{removeTransactions}} removes the row with no
> membership re-check. SCM has no catch-up mechanism that compares a replica's
> {{deleteTransactionId}} to the container's and re-issues missed deletions;
> once the transaction is purged it is never re-delivered.
> h3. Impact
> A container replica that becomes current in the snapshot-to-purge window
> without having applied the deletion permanently retains the deleted blocks,
> an undeleted-block space leak on that datanode with no automatic recovery at
> the SCM level. The leak is confined to datanode capacity; there is no client
> visible data corruption. The SCM-level premature purge reproduces
> deterministically; demonstrating the physical on-disk leak requires a
> mini-cluster with adversarial copy-versus-delete timing, so this is filed as
> an environment-limited (not fully cluster-reproduced) finding.
> h3. Relation to HDDS-11498
> HDDS-11498 improved deletion efficiency and the "send to all replicas or not
> at all" handling in {{commitTransactions}}, but does not address a replica
> that becomes current after the ACK-set snapshot is taken and before the
> durable removal. No filed issue reports this stale-snapshot premature purge.
> h3. Suggested fix
> Re-validate the current replica set immediately before removing a transaction
> from the durable log (recompute {{canPurge}} against a fresh
> {{getContainerReplicas}} under the same critical section), or reconcile a
> newly added replica's {{deleteTransactionId}} against the container and
> re-issue any missed delete transactions before allowing the purge, so a
> replica that has not acknowledged the deletion cannot be orphaned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
