Siyao Meng created HDDS-16145:
---------------------------------
Summary: Checkpoint flush during delete transaction removal can
double count the SCM deleted block summary
Key: HDDS-16145
URL: https://issues.apache.org/jira/browse/HDDS-16145
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
A checkpoint or leader-transfer flush that interposes in the middle of
{{DeletedBlockLogStateManagerImpl.removeTransactionsFromDB}} can re-expose a
delete-transaction row that is still durable, causing the block-deletion
scanner to select it again and decrement the volatile summary a second time, to
a permanent and impossible negative value.
h3. Steps to reproduce
# The block-deletion scanner drives terminal cleanup, which calls
{{removeTransactionsFromDB(txIDs, summary)}}. That method first marks the ids
in {{deletingTxIDs}} (the scanner-hide set), then removes each row from the
transaction buffer, then buffers the updated summary, as separate steps.
# A checkpoint download ({{InterSCMGrpcService}}) or leader-transfer flush runs
{{transactionBuffer.flush()}} between the mark and the buffered removal.
{{onFlush}} then resets {{deletingTxIDs}} to a fresh empty set, clearing the
hide marker while the row is still durable (its {{removeFromBuffer}} had not
been flushed).
# The next scan no longer sees the row as hidden, re-selects it, and terminal
cleanup decrements the summary again for the same transaction.
The summary is driven to a negative value (for a single transaction it reaches
-1), which is persisted durably and reloaded verbatim on restart and leader
transfer.
h3. Root cause
{{removeTransactionsFromDB}} performs the hide marking
({{deletingTxIDs.addAll}}), the row removal ({{removeFromBuffer}}), and the
summary update as separate, non-atomic steps.
{{SCMHADBTransactionBufferImpl.flush()}} takes only the write lock and does not
consult the in-progress apply/removal state, and {{onFlush}} resets
{{deletingTxIDs}}. A flush interposed mid-removal therefore clears the hide
marker for a row that is still durable, so the scanner re-selects it and
double-decrements the summary. A negative deleted-block count is an impossible
undercount.
h3. Impact
The deleted-block transaction summary is observability only. It is surfaced by
the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}}, Recon
{{PendingDeletionEndpoint}}, and JMX metrics, and does not affect deletion
correctness. The consequence is a persistent, operator visible accounting
corruption (a negative pending-deletion count) that survives restart and leader
transfer. There is no data loss and no premature deletion.
h3. Relation to HDDS-15665
HDDS-15665 stabilizes an already-open deleted-block iterator against concurrent
modification, but does not make {{removeTransactionsFromDB}} atomic with
respect to {{flush()}}, nor does it protect a new iterator opened after a
mid-removal flush clears the hide marker. No filed issue reports this
checkpoint-flush re-exposure and summary double decrement.
h3. Suggested fix
Make the hide marking, row removal, and summary update in
{{removeTransactionsFromDB}} atomic with respect to flush, or have
{{flush()}}/{{onFlush}} preserve the {{deletingTxIDs}} hide markers for rows
whose removal has not yet been flushed, so a flush cannot re-expose a
still-durable row that is being removed. Recomputing the summary from the
durable rows on reload would also prevent a negative value from being persisted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
