[
https://issues.apache.org/jira/browse/HDDS-16919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16919:
------------------------------
Attachment: TestBugCR3InterruptFlagAfterDownloadFailure.java
> SCMSnapshotProvider sets the thread interrupt flag when the checkpoint
> download fails without an interrupt
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16919
> URL: https://issues.apache.org/jira/browse/HDDS-16919
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-3-do-not-interrupt-thread-on-checkpoint-download-failure.patch,
> TestBugCR3InterruptFlagAfterDownloadFailure.java
>
>
> h3. Mechanism
> {{SCMSnapshotProvider.getSCMDBSnapshot}} catches {{ExecutionException}} and
> {{InterruptedException}} in one block and calls
> {{Thread.currentThread().interrupt()}} for both. An ordinary download failure
> (for example gRPC UNAVAILABLE or a deadline) completes the download future
> exceptionally, so {{get()}} throws {{ExecutionException}}, and the install
> thread is left interrupted although nothing interrupted it. The shared catch
> came with HDDS-5103; HDDS-5467, which restored the interrupt flag after
> {{InterruptedException}} across the code base, added the call to it.
> h3. Trigger
> # A follower SCM gets an install snapshot notification and downloads the
> leader's checkpoint on the {{SCMInstallSnapshot}} thread.
> # The download fails, for example because the leader's gRPC server is not
> reachable.
> h3. Impact
> * The install thread has its interrupt flag set after a failed download. Any
> blocking call later in the same task fails with {{InterruptedException}}.
> * Today this has no visible effect:
> {{SCMStateMachine.notifyInstallSnapshotFromLeader}} returns right away on the
> null checkpoint, and {{ThreadPoolExecutor}} clears the flag before the next
> install task. It breaks any future change that does blocking work after a
> failed download (a retry or a cleanup) in the same task.
> h3. Reproduction
> PASS on the unmodified source.
> [^TestBugCR3InterruptFlagAfterDownloadFailure.java]
> ({{hadoop-hdds/server-scm}}) points a real {{SCMSnapshotProvider}} at a local
> port with nothing listening and runs two tasks on a single thread executor,
> as {{SCMStateMachine}} does. The first task prints {{interrupted=true | sleep
> threw InterruptedException}} after the failed download. The second task on
> the same thread starts with the flag cleared. A passing test means the defect
> is present.
> h3. Patch
> [^CR-3-do-not-interrupt-thread-on-checkpoint-download-failure.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> The patch splits the catch so that only {{InterruptedException}} restores the
> interrupt flag. Log message and the thrown {{IOException}} are unchanged.
> With the patch the reproduction fails because the first task is no longer
> interrupted.
> Covered by {{testFailedDownloadDoesNotInterruptThread}} and
> {{testInterruptedDownloadKeepsInterruptFlag}} in the existing
> {{TestSCMSnapshotProvider}}. Without the change the first fails with
> "Expecting value to be false but was true". With it
> {{TestSCMSnapshotProvider}}, {{TestSCMStateMachine}} and
> {{TestSCMHAManagerImpl}} (23 tests) pass and checkstyle is clean.
> Found by code review of the SCM Ratis snapshot and checkpoint install paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
