Siyao Meng created HDDS-17669:
---------------------------------
Summary: Interrupting a Ratis key write, hsync or close that waits
for a watch for commit loses the interrupt, and the data is retried on a new
block
Key: HDDS-17669
URL: https://issues.apache.org/jira/browse/HDDS-17669
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-5-restore-interrupt-flag-in-block-output-stream.patch,
TestBugCR5InterruptedWriteRetries.java
h3. Mechanism
{{KeyOutputStream.handleRetry}} stops retrying a failed block when the thread
is interrupted ("Interrupted while trying for retry"). Three catch sites in
{{BlockOutputStream}} turn an {{InterruptedException}} into an {{IOException}}
without restoring the interrupt flag, unlike the other four in the same class:
{{waitForAllPendingFlushes}}, {{writeOnRetry}} and
{{allocateNewBufferIfNeeded}}. {{Future.get}} or {{Lock.lockInterruptibly}} has
already cleared the flag, so it is lost.
Every {{close()}} and {{hsync()}} of a Ratis key waits in
{{BlockOutputStream.handleFlushInternal}} for the watch for commit of its own
PutBlock, and a {{write}} on a full buffer pool waits in {{handleFullBuffer}}
for the watch of its last PutBlock. If the thread is interrupted during either
wait, the catch site there restores the flag and fails the block stream.
{{KeyOutputStream.handleExceptionInternal}} then waits in
{{waitForAllPendingFlushes}} for the same pending watch. That wait sees the
flag, throws {{InterruptedException}}, clears the flag, and the resulting
{{IOException}} is ignored there. The failure handling goes on, the guard in
{{handleRetry}} no longer sees the interrupt, and the buffered data is written
again to a new block.
h3. Trigger
# Interrupt a thread in {{close()}} or {{hsync()}} of a Ratis key while it
waits for the watch for commit of its PutBlock ({{Future.cancel(true)}},
{{ExecutorService.shutdownNow}}, a framework task kill). No datanode error and
no slow follower are needed.
# Or interrupt a thread in {{write}} while it waits on a full buffer pool for a
slow watch for commit.
The same happens when the interrupt arrives while the failure handling of an
earlier error (for example a PutBlock timeout) waits for a pending watch of the
same block, or while a retry waits for its watch in {{writeOnRetry}}.
h3. Impact
* An interrupted {{close()}} returns normally and commits the key. An
interrupted {{hsync()}} or {{write}} returns normally, and a later {{close()}}
commits the key with all the data. In each case the interrupt flag is cleared
and OM allocated a new block for the data, and the old pipeline is added to the
stream's exclude list. The cancellation is ignored, and the caller has no way
to see that it was interrupted (run, see Reproduction).
* No data loss or wrong data was observed. The key that is committed contains
everything the application wrote.
* Argued from the code, not run: the failure handling continues while the old
block's watch is still pending, so a late watch reply can race with the retry.
Without the interrupt this cannot occur, because {{waitForAllPendingFlushes}}
returns only after every pending watch has completed.
h3. Reproduction
PASS on unmodified source, deterministic, no cluster, about 4 s.
[^TestBugCR5InterruptedWriteRetries.java] ({{hadoop-ozone/client}}) uses the
real {{KeyOutputStream}}, {{RatisBlockOutputStream}} and {{CommitWatcher}},
with a scripted {{XceiverClientSpi}} whose watches for commit on the first
pipeline stay pending, and a mocked OM.
* {{interruptWhileCloseWaitsForWatch}}: after a 100 byte write, {{close()}}
blocks on the watch. After the interrupt it returns normally, the flag is
clear, one new block was allocated, and the key is committed with 100 bytes.
* {{interruptWhileHsyncWaitsForWatch}}: the same for {{hsync()}}. A later
{{close()}} commits 100 bytes.
* {{interruptWhileWriteWaitsOnFullBuffer}}: a 16 KB write fills the four 4 KB
buffers and blocks. After the interrupt it returns normally, the flag is clear,
one new block was allocated, and {{close()}} commits 16,384 bytes.
* {{interruptWhileFailureHandlingWaitsForPendingFlushes}}: a 4 KB write leaves
its watch pending, the PutBlock of the next 4 KB fails, and a third write
blocks in {{waitForAllPendingFlushes}}. After the interrupt the same happens,
while the old watch is still pending.
* {{withoutInterruptFailureHandlingWaitsForPendingWatch}}: control without the
interrupt. The failure handling waits until the old watch is answered, then the
key commits with all 8,292 bytes.
A passing test means the defect is present.
h3. Patch
[^CR-5-restore-interrupt-flag-in-block-output-stream.patch], against
ec5ed5f7ea0e4b406aee53aae4906616819cba2b. It also applies to master at
b683fd3f294.
The three catch sites call {{Thread.currentThread().interrupt()}} before
{{handleInterruptedException}}, as the other catch sites of
{{BlockOutputStream}} do. With the patch an interrupted {{write()}},
{{hsync()}} or {{close()}} fails with the thread's interrupt flag kept and the
key is not committed, so an application that interrupts writer threads will now
see that failure where the call used to succeed silently. The exception is the
one the existing guard in {{handleRetry}} already throws: the original error,
not an {{InterruptedIOException}}.
Covered by three tests in the existing {{TestBlockOutputStreamCorrectness}},
one per catch site, and {{testInterruptedWriteFailsWithoutRetry}} in the
existing {{TestKeyOutputStream}}, which interrupts a write blocked on a full
buffer pool and checks that it fails, keeps the flag and asks OM for no new
block. All four fail without the change because the flag is cleared. With it,
the hdds-client storage unit tests (225), the ozone-client unit tests (650) and
the mini cluster suites {{TestCommitWatcher}},
{{TestBlockOutputStreamWithFailures}} and {{TestFailureHandlingByClient}} (28
tests) pass, and checkstyle is clean. With the patch the four interrupt tests
of the reproduction fail, because the call now throws, the flag is set, no
block is allocated and nothing is committed. The control still passes.
Related: HDDS-17648 (a watch that never completes when the Ratis client is
closed) leaves {{waitForAllPendingFlushes}} waiting forever. Until that is
fixed an interrupt is the only way out of the wait, and today it turns into a
silent retry. The patches touch different files.
Found by code review of the Ozone client Ratis block write and failure handling
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
ec5ed5f7ea0e4b406aee53aae4906616819cba2b. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]