Siyao Meng created HDDS-16919:
---------------------------------
Summary: SCMSnapshotProvider sets the thread interrupt flag when
the checkpoint download fails without an interrupt
Key: HDDS-16919
URL: https://issues.apache.org/jira/browse/HDDS-16919
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-3-do-not-interrupt-thread-on-checkpoint-download-failure.patch,
TestBugCR3InterruptFlagAfterDownloadFailure.java
h3. Mechanism
{{SCMSnapshotProvider.getSCMDBSnapshot}} catches {{ExecutionException}} and
{{InterruptedException}} in one block and calls
{{Thread.currentThread().interrupt()}} for both. An ordinary download failure
(for example gRPC UNAVAILABLE or a deadline) completes the download future
exceptionally, so {{get()}} throws {{ExecutionException}}, and the install
thread is left interrupted although nothing interrupted it. The shared catch
came with HDDS-5103; HDDS-5467, which restored the interrupt flag after
{{InterruptedException}} across the code base, added the call to it.
h3. Trigger
# A follower SCM gets an install snapshot notification and downloads the
leader's checkpoint on the {{SCMInstallSnapshot}} thread.
# The download fails, for example because the leader's gRPC server is not
reachable.
h3. Impact
* The install thread has its interrupt flag set after a failed download. Any
blocking call later in the same task fails with {{InterruptedException}}.
* Today this has no visible effect:
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} returns right away on the
null checkpoint, and {{ThreadPoolExecutor}} clears the flag before the next
install task. It breaks any future change that does blocking work after a
failed download (a retry or a cleanup) in the same task.
h3. Reproduction
PASS on the unmodified source.
[^TestBugCR3InterruptFlagAfterDownloadFailure.java]
({{hadoop-hdds/server-scm}}) points a real {{SCMSnapshotProvider}} at a local
port with nothing listening and runs two tasks on a single thread executor, as
{{SCMStateMachine}} does. The first task prints {{interrupted=true | sleep
threw InterruptedException}} after the failed download. The second task on the
same thread starts with the flag cleared. A passing test means the defect is
present.
h3. Patch
[^CR-3-do-not-interrupt-thread-on-checkpoint-download-failure.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
The patch splits the catch so that only {{InterruptedException}} restores the
interrupt flag. Log message and the thrown {{IOException}} are unchanged. With
the patch the reproduction fails because the first task is no longer
interrupted.
Covered by {{testFailedDownloadDoesNotInterruptThread}} and
{{testInterruptedDownloadKeepsInterruptFlag}} in the existing
{{TestSCMSnapshotProvider}}. Without the change the first fails with "Expecting
value to be false but was true". With it {{TestSCMSnapshotProvider}},
{{TestSCMStateMachine}} and {{TestSCMHAManagerImpl}} (23 tests) pass and
checkstyle is clean.
Found by code review of the SCM Ratis snapshot and checkpoint install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
