Siyao Meng created HDDS-17002:
---------------------------------
Summary: Streaming block read after unbuffer() requests nothing at
the current position, times out and excludes a healthy datanode
Key: HDDS-17002
URL: https://issues.apache.org/jira/browse/HDDS-17002
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-4-reset-stream-read-request-mark-on-reader-close.patch,
TestBugMC4StreamReadUnbuffer.java
h3. Mechanism
{{StreamBlockInputStream.unbuffer}} calls {{releaseClient}}, which calls
{{closeReader}}. That discards the {{StreamingReader}} together with every
response still owed to it, but leaves {{requestedLength}} unchanged. On the
next read a new reader is opened and {{StreamBlockInputStream.readBlock}}
counts {{requestedLength - position}} as already requested: for a block shorter
than the pre-read size {{readLength}} is clamped to 0, for a longer block the
"already requested, above refill threshold" branch usually returns early.
Either way nothing covering the current position is requested on the new stream
(in the remaining case, a longer block whose stale window has drained just
below the refill threshold, a ReadBlock starts at the old mark, past the
position, and its responses are discarded; this case is from code reading).
{{StreamingReader.poll}} then times out, and {{handleExceptions}} adds the
healthy datanode to {{failedStreamingDatanodes}}, which is never cleared for
that stream. After one such cycle per replica
{{XceiverClientGrpc.initStreamRead}} has no datanode left and throws.
{{handleExceptions}} already resets {{requestedLength = position}} after
{{releaseClient}}, so a new reader is expected to start from the current
position; the unbuffer path skips that reset.
h3. Trigger
# {{ozone.client.stream.readblock.enable=true}}, a RATIS/THREE key, all
datanodes healthy.
# Open the key and {{read()}}.
# {{unbuffer()}} (for example through {{OzoneFSInputStream}}, which implements
{{CanUnbuffer}}), then {{read()}} again. Repeat.
No fault is needed. This is the usual pattern of {{CanUnbuffer}} callers that
keep a stream open and release resources between reads. Only a sequential read
that continues from the same position after {{unbuffer()}} is affected:
positioned reads ({{readPositioned}}) do not use {{requestedLength}}, and a
{{seek()}} to another position after {{unbuffer()}} resets it.
h3. Impact
* Every read after {{unbuffer()}} stalls for
{{ozone.client.stream.read.timeout}} plus the read retry interval (4.1 s
observed with a 3 s timeout, so about 11 s with the default 10 s timeout, not
run) and excludes one more healthy datanode from that stream.
* After one unbuffer and read cycle per replica the read throws {{IOException:
Failed to start streaming read to any available DataNodes}} on a healthy
cluster. Reopening the key starts with an empty exclusion set.
* Only with streaming block reads enabled (off by default). No wrong data was
observed: the bytes read after each stall were correct.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugMC4StreamReadUnbuffer.java]
({{hadoop-ozone/integration-test}}) starts a mini cluster with three datanodes,
writes a 32 KiB RATIS/THREE key and reads it with a streaming read client (3 s
stream read timeout), using only {{read()}} and {{unbuffer()}}; reflection is
used only to inspect the exclusion set. A passing test means the defect is
present. Output on the unmodified source:
{noformat}
MC-4 cycle 1: read after unbuffer took 4114 ms, excluded datanodes
[a1638c9a-...]
MC-4 cycle 2: read after unbuffer took 4102 ms, excluded datanodes
[a1638c9a-..., 8bbfd668-...]
MC-4 cycle 3: read after unbuffer threw java.io.IOException: Failed to start
streaming read to any available DataNodes
{noformat}
Each stalled cycle logs {{Failed to poll a response, timed out PT3S}} followed
by {{Excluding DataNode ... from streaming read retries}}. With the patch the
test fails at cycle 1 (2 ms, nothing excluded).
h3. Patch
[^MC-4-reset-stream-read-request-mark-on-reader-close.patch], against
1fb64c8e499161d1452ae6948ffd461fc9d2c302. It also applies to master at
8664dc4fc3c.
{{closeReader}} sets {{requestedLength = position}} when it discards the
reader, so whatever path discards a reader ({{unbuffer}}, {{close}}, the retry
path, end of block), the next reader requests again from the current position.
Behaviour of the other callers does not change: the retry path already reset,
and at end of block both values are already the block length.
Covered by {{testStreamReadAfterUnbuffer}} in the existing
{{TestStreamReadDatanodeFailover}}, which unbuffers before the first read and
then between reads, once more than there are replicas, and checks the rest of
the key. It runs with a pre-read window of 16 KiB and of 32 MiB on a 32 KiB
key, so both the clamp and the refill threshold branch are exercised. Without
the change both cases fail with {{IOException: Failed to start streaming read
to any available DataNodes}}. With it {{TestStreamReadDatanodeFailover}} (5
tests), {{TestStreamRead}} (4), the integration {{TestStreamBlockInputStream}}
(3) and the unit {{TestStreamBlockInputStream}} (29) and
{{TestMultipartInputStream}} (6) pass, checkstyle is clean.
Found by TLA+ model checking and code review of the Ozone client read retry and
failover paths under HDDS-15926, on commit
1fb64c8e499161d1452ae6948ffd461fc9d2c302. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
