[
https://issues.apache.org/jira/browse/HDDS-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16142:
------------------------------
Attachment: HDDS-16142.001.patch
> SCM deleted block transaction summary can be double counted after a Ratis
> submit timeout
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16142
> URL: https://issues.apache.org/jira/browse/HDDS-16142
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16142.001.patch
>
>
> The volatile deleted-block transaction summary in
> {{SCMDeletedBlockTransactionStatusManager}} (the {{totalTxCount}},
> {{totalBlockCount}}, {{totalBlockSize}}, {{totalReplicatedBlockSize}}
> counters surfaced by {{getDeletedBlockSummary}}) can be decremented twice for
> a single transaction, driving the summary below the set of transactions it
> actually accounts for. The undercount is persisted durably and reloaded on
> leader transfer, so it is permanent.
> h3. Steps to reproduce
> # Add a delete transaction so the summary is incremented and the transaction
> is queued for the durable deleted-block log.
> # The Ratis submit for the add times out.
> {{SCMRatisServerImpl.submitRequest}} does
> {{submitClientRequestAsync(req).get(timeout)}}, which abandons the future
> without cancelling the request, so {{addTransactions}} takes the
> {{IOException}} path and rolls the summary back down, assuming the durable
> add never landed.
> # The abandoned request still commits and applies afterwards, so the
> transaction is durably present in the deleted-block log even though the
> summary was reverted.
> # The container later becomes terminal. {{getTransactions}} re-caches the
> transaction size and {{removeTransactions}} decrements the summary a second
> time.
> After this sequence the volatile summary is at least 1 below the number of
> transactions accounted for. {{removeTransactionsFromDB(txIDs, getSummary())}}
> then persists the undercount durably, so it survives leader transfer and
> restart.
> h3. Root cause
> {{SCMDeletedBlockTransactionStatusManager.addTransactions}} increments the
> summary, calls {{deletedBlockLogStateManager.addTransactionsToDB(txList,
> getSummary())}}, and on {{IOException}} reverts the increment in its catch
> block on the assumption that the durable add did not happen. A Ratis submit
> timeout breaks that assumption: {{submitRequest}} abandons the client request
> future after the timeout without cancelling it, so the request can still be
> committed and applied, persisting the transaction. The now durable
> transaction is later removed normally and {{removeTransactions}} decrements
> the summary again for the same transaction, a double decrement for a single
> transaction.
> h3. Impact
> The deleted-block transaction summary is observability only. It is surfaced
> by the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}},
> Recon, and JMX metrics, and does not affect deletion correctness or
> {{getNumOfValidTransactions()}}. The consequence is a persistent, operator
> visible undercount of pending deleted-block work that survives leader
> transfer and restart. There is no data loss and no premature deletion.
> h3. Relation to HDDS-13184, HDDS-15726, HDDS-14572
> HDDS-13184 introduced the deleted-block summary accounting. HDDS-15726 (the
> most recent change to this file) added the {{addTransactions}} and
> {{removeTransactions}} rollback-on-DB-failure catch blocks that this bug
> exploits; it does not account for the submit-timeout-but-still-applied case.
> HDDS-14572 added the leader-transfer summary reconstruction that makes the
> undercount survive failover. None of these reports or fixes the double
> decrement.
> h3. Suggested fix
> Do not treat an {{IOException}} from {{addTransactionsToDB}} as proof that
> the durable mutation did not land. Prefer reconciling the summary from the
> authoritative durable state (the deleted-block log and the persisted summary)
> on the failure path and on leader transfer, rather than maintaining it purely
> through incremental increment and rollback. Alternatively, make the submit
> path settle or cancel the request so a timed-out submit cannot later commit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
