[
https://issues.apache.org/jira/browse/HDDS-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16743:
------------------------------
Attachment: MC-2-retry-cache-restart-reproduction.patch
> OM retry cache does not survive a restart, so a client retry of an applied
> write is applied again and reports a false failure
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16743
> URL: https://issues.apache.org/jira/browse/HDDS-16743
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: MC-2-retry-cache-restart-reproduction.patch
>
>
> h3. Mechanism
> What makes an OM write at most once is the Ratis retry cache record keyed by
> {{ClientInvocationId(clientId, callId)}}, and that record lives only in
> process memory. Ratis writes it when it applies the entry, nothing persists
> it, and nothing rebuilds it. The Raft log and the OM's applied index are
> durable, and after a clean restart the state machine is already at the end of
> the log, so Ratis replays nothing.
> A restarted OM therefore still holds the committed entry proving the call
> ran, and has lost the only record that would stop it running again.
> {{checkRetryCache()}} ({{OzoneManagerRatisServer.java:546-548}}) misses,
> {{OMExecutionFlow}} has no second layer of dedup, and Ratis consults the same
> empty cache on submit.
> h3. Trigger
> A client write is committed and applied, and the reply is lost because the OM
> goes down (rolling restart, crash, pod replacement). The OM comes back with
> an empty retry cache, and the Hadoop RPC retry presents the original client
> ID and call ID.
> This holds whenever the retry reaches an OM process that did not apply the
> entry itself: a single OM, a restart of every OM, or a restarted node that
> regains leadership. A plain failover to a peer that stayed up is covered,
> because every replica that applies the entry creates the record. That is the
> case {{TestOzoneManagerHAWithStoppedNodes#testRetryCacheWithDownedOM}} tests.
> It also needs the entry to have been flushed to the OM DB before the OM went
> down. An OM that stops with the entry applied but not yet flushed replays it
> from the Raft log on restart, and the replay recreates the record. So a crash
> right after the write is partly self healing, while a restart a moment later
> is not.
> h3. Impact
> A second log entry with the same {{ClientInvocationId}} is committed and
> applied, and the client gets a different answer for the same call. For
> {{RenameKey}} the rename that durably succeeded is reported as
> {{KEY_ALREADY_EXISTS}}. Nothing here is specific to rename: any request type
> that is not idempotent on its own is exposed in the same way, and
> {{CommitKey}} is the case described in HDDS-16736.
> h3. Reproduction
> PASS. With the real client and a real OM restart, {{OzoneBucket.renameKey}}
> threw {{KEY_ALREADY_EXISTS}} for a rename that had happened, in 2 of 3
> iterations, with only a short delay before the restart as timing help. With
> the client ID and call ID pinned the way {{testRetryCacheWithDownedOM}} does
> it: one committed entry for the invocation and no retry cache record after
> the restart, two committed entries after the retry, and the retry answered
> {{KEY_ALREADY_EXISTS}}. The control, the same duplicate without a restart, is
> absorbed and answered OK. Artifacts under {{confirmation/MC-2/}} and
> {{repro/}}, driven from the counterexample
> {{spec/output/r3_MC_deep_s3restart_apply.out}} against invariant
> {{AtMostOnceApply}}.
> h3. Relation to HDDS-16736
> HDDS-16736 is about the gRPC transport never being able to use the retry
> cache, and names OM restart in passing as a further route to the same loss.
> This issue is that route on the default Hadoop RPC transport, for all request
> types, with its own reproduction.
> h3. Suggested fix
> Make the record as durable as the log it guards, or make the requests
> idempotent.
> * Rebuild the retry cache on startup from the log entries of the last retry
> cache expiry window, without applying them again. This is a change in Ratis.
> * Or persist the invocation ID and result for the recent window in the OM DB,
> in the same batch as the transaction, and have {{checkRetryCache()}} fall
> back to it on a miss. That also closes the expiry and leader change routes
> named in HDDS-16736.
> * Or make each non idempotent request recognise its own earlier success,
> which needs a writer identity on the stored record, as discussed in
> HDDS-16736.
> h3. No fix patch attached
> All three options are design decisions for reviewers: a Ratis change, a new
> persisted table, or a per request change to the stored record. None is a
> mechanical change.
> [^MC-2-retry-cache-restart-reproduction.patch] is a reproduction only,
> against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It adds
> {{testRetryCacheWithRestartedOMs}} to the existing
> {{TestOzoneManagerHAWithStoppedNodes}}, next to the failover test it mirrors:
> the same rename, a wait for every OM to apply and flush it, a restart of
> every OM, then the same call again. It asserts the retry is answered OK, and
> fails at this commit with {{KEY_ALREADY_EXISTS}}. Without the wait for the
> flush the same test passes, because the OMs replay the rename on restart.
> Found by TLA+ model checking and code review of the OM client retry cache and
> at most once mutation paths under HDDS-15926, on commit
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Checked against open HDDS issues
> for duplicates before filing, in addition to the repository's merged git
> history. Generated with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
