[
https://issues.apache.org/jira/browse/HDDS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17002:
------------------------------
Attachment: MC-4-reset-stream-read-request-mark-on-reader-close.patch
> Streaming block read after unbuffer() requests nothing at the current
> position, times out and excludes a healthy datanode
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17002
> URL: https://issues.apache.org/jira/browse/HDDS-17002
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-4-reset-stream-read-request-mark-on-reader-close.patch,
> TestBugMC4StreamReadUnbuffer.java
>
>
> h3. Mechanism
> {{StreamBlockInputStream.unbuffer}} calls {{releaseClient}}, which calls
> {{closeReader}}. That discards the {{StreamingReader}} together with every
> response still owed to it, but leaves {{requestedLength}} unchanged. On the
> next read a new reader is opened and {{StreamBlockInputStream.readBlock}}
> counts {{requestedLength - position}} as already requested: for a block
> shorter than the pre-read size {{readLength}} is clamped to 0, for a longer
> block the "already requested, above refill threshold" branch usually returns
> early. Either way nothing covering the current position is requested on the
> new stream (in the remaining case, a longer block whose stale window has
> drained just below the refill threshold, a ReadBlock starts at the old mark,
> past the position, and its responses are discarded; this case is from code
> reading). {{StreamingReader.poll}} then times out, and {{handleExceptions}}
> adds the healthy datanode to {{failedStreamingDatanodes}}, which is never
> cleared for that stream. After one such cycle per replica
> {{XceiverClientGrpc.initStreamRead}} has no datanode left and throws.
> {{handleExceptions}} already resets {{requestedLength = position}} after
> {{releaseClient}}, so a new reader is expected to start from the current
> position; the unbuffer path skips that reset.
> h3. Trigger
> # {{ozone.client.stream.readblock.enable=true}}, a RATIS/THREE key, all
> datanodes healthy.
> # Open the key and {{read()}}.
> # {{unbuffer()}} (for example through {{OzoneFSInputStream}}, which
> implements {{CanUnbuffer}}), then {{read()}} again. Repeat.
> No fault is needed. This is the usual pattern of {{CanUnbuffer}} callers that
> keep a stream open and release resources between reads. Only a sequential
> read that continues from the same position after {{unbuffer()}} is affected:
> positioned reads ({{readPositioned}}) do not use {{requestedLength}}, and a
> {{seek()}} to another position after {{unbuffer()}} resets it.
> h3. Impact
> * Every read after {{unbuffer()}} stalls for
> {{ozone.client.stream.read.timeout}} plus the read retry interval (4.1 s
> observed with a 3 s timeout, so about 11 s with the default 10 s timeout, not
> run) and excludes one more healthy datanode from that stream.
> * After one unbuffer and read cycle per replica the read throws
> {{IOException: Failed to start streaming read to any available DataNodes}} on
> a healthy cluster. Reopening the key starts with an empty exclusion set.
> * Only with streaming block reads enabled (off by default). No wrong data was
> observed: the bytes read after each stall were correct.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugMC4StreamReadUnbuffer.java]
> ({{hadoop-ozone/integration-test}}) starts a mini cluster with three
> datanodes, writes a 32 KiB RATIS/THREE key and reads it with a streaming read
> client (3 s stream read timeout), using only {{read()}} and {{unbuffer()}};
> reflection is used only to inspect the exclusion set. A passing test means
> the defect is present. Output on the unmodified source:
> {noformat}
> MC-4 cycle 1: read after unbuffer took 4114 ms, excluded datanodes
> [a1638c9a-...]
> MC-4 cycle 2: read after unbuffer took 4102 ms, excluded datanodes
> [a1638c9a-..., 8bbfd668-...]
> MC-4 cycle 3: read after unbuffer threw java.io.IOException: Failed to start
> streaming read to any available DataNodes
> {noformat}
> Each stalled cycle logs {{Failed to poll a response, timed out PT3S}}
> followed by {{Excluding DataNode ... from streaming read retries}}. With the
> patch the test fails at cycle 1 (2 ms, nothing excluded).
> h3. Patch
> [^MC-4-reset-stream-read-request-mark-on-reader-close.patch], against
> 1fb64c8e499161d1452ae6948ffd461fc9d2c302. It also applies to master at
> 8664dc4fc3c.
> {{closeReader}} sets {{requestedLength = position}} when it discards the
> reader, so whatever path discards a reader ({{unbuffer}}, {{close}}, the
> retry path, end of block), the next reader requests again from the current
> position. Behaviour of the other callers does not change: the retry path
> already reset, and at end of block both values are already the block length.
> Covered by {{testStreamReadAfterUnbuffer}} in the existing
> {{TestStreamReadDatanodeFailover}}, which unbuffers before the first read and
> then between reads, once more than there are replicas, and checks the rest of
> the key. It runs with a pre-read window of 16 KiB and of 32 MiB on a 32 KiB
> key, so both the clamp and the refill threshold branch are exercised. Without
> the change both cases fail with {{IOException: Failed to start streaming read
> to any available DataNodes}}. With it {{TestStreamReadDatanodeFailover}} (5
> tests), {{TestStreamRead}} (4), the integration
> {{TestStreamBlockInputStream}} (3) and the unit
> {{TestStreamBlockInputStream}} (29) and {{TestMultipartInputStream}} (6)
> pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the Ozone client read retry
> and failover paths under HDDS-15926, on commit
> 1fb64c8e499161d1452ae6948ffd461fc9d2c302. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
