[
https://issues.apache.org/jira/browse/HDDS-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16143:
------------------------------
Attachment: HDDS-16143.001.patch
> SCM leader transfer flush can split a delete transaction apply and persist a
> summary undercount
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16143
> URL: https://issues.apache.org/jira/browse/HDDS-16143
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16143.001.patch
>
>
> When SCM leadership changes, {{SCMStateMachine.notifyLeaderChanged}} calls
> {{transactionBuffer.flush()}} unconditionally. If that flush interposes in
> the middle of a delete-transaction apply, it can persist the transaction rows
> without the matching summary update, and the subsequent leader-side
> reconstruction reads the resulting durable summary undercount into the
> in-memory summary, baking in a permanent undercount that survives restart.
> h3. Steps to reproduce
> # A delete-transaction apply is in progress.
> {{DeletedBlockLogStateManagerImpl.addTransactionsToDB}} buffers the
> transaction rows first and then buffers the updated summary as a separate
> {{addToBuffer}} call.
> # An SCM leader transfer fires and {{SCMStateMachine.notifyLeaderChanged}}
> calls {{transactionBuffer.flush()}} between the row buffering and the summary
> buffering.
> # The flush persists the rows with the stale (pre-update) summary, so the
> durable summary is now lower than the durable row count.
> # {{onBecomeLeader}} then runs {{initDataDistributionData}}, which reloads
> the in-memory summary from that undercounted durable value rather than
> recomputing it from the rows.
> The reconstructed in-memory summary is permanently short; the next durable
> write re-persists the undercount, so it survives further leader transfers and
> restart.
> h3. Root cause
> {{addTransactionsToDB}} performs the row mutation and the summary mutation as
> two separate buffer operations, so they are not atomic with respect to a
> flush. The leader-transfer path uses the unguarded
> {{transactionBuffer.flush()}} in {{notifyLeaderChanged}} (not the
> {{flushIfNeeded}} variant that skips when an apply is in progress), so a
> flush can split an in-progress apply. {{onBecomeLeader}} then reconstructs
> the in-memory summary from the split durable value via
> {{initDataDistributionData}}, converting a transient durable inconsistency
> into a permanent in-memory one.
> h3. Impact
> The deleted-block transaction summary is observability only. It is surfaced
> by the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}},
> Recon, and JMX metrics, and does not affect deletion correctness. The
> consequence is a persistent, operator visible summary undercount that
> survives leader transfer and restart. There is no data loss and no premature
> deletion.
> h3. Relation to HDDS-14572, HDDS-15065
> HDDS-14572 added the leader-transfer summary reconstruction
> ({{onBecomeLeader}} to {{initDataDistributionData}}) that this bug relies on
> to make the undercount permanent. HDDS-15065 added the guarded
> {{flushIfNeeded}} path (which skips flushing while an apply is in progress)
> but did not guard the {{flush()}} in {{notifyLeaderChanged}}. Neither reports
> or fixes this split-then-reconstruct undercount.
> h3. Suggested fix
> Make the row-and-summary durable mutation atomic with respect to flush
> (buffer both under the same guarded section), or guard the leader-transfer
> {{flush()}} the same way {{flushIfNeeded}} is guarded so it cannot split an
> in-progress apply, or have {{initDataDistributionData}} recompute the summary
> from the durable rows rather than trusting a possibly-split persisted summary
> value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
