[
https://issues.apache.org/jira/browse/HDDS-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17004:
------------------------------
Attachment: CR-1-fail-over-streaming-read-on-checksum-mismatch.patch
> Streaming block read does not try another replica after a client side
> checksum mismatch
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-17004
> URL: https://issues.apache.org/jira/browse/HDDS-17004
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-1-fail-over-streaming-read-on-checksum-mismatch.patch,
> TestBugCR1StreamReadChecksumNoFailover.java
>
>
> h3. Mechanism
> With {{ozone.client.verify.checksum=true}} (default)
> {{StreamBlockInputStream.StreamingReader.onNext}} verifies each ReadBlock
> response and fails the stream with {{OzoneChecksumException}} on a mismatch.
> The reader surfaces it as {{IOException("Streaming read failed")}} wrapping
> an {{ExecutionException}}. {{ConnectionFailureUtils.unwrapCause}} only stops
> at {{StorageContainerException}} and {{TimeoutIOException}}, and
> {{StreamBlockInputStream.handleExceptions}} (sequential reads) and
> {{handlePreadException}} (positioned reads) retry only those and connectivity
> errors. So the checksum error is rethrown and no other replica is tried. On
> the sequential path the failed reader is also kept, so later sequential reads
> of other ranges on the same stream fail with the same error until the stream
> is unbuffered or reopened.
> The non streaming path validates checksums inside the per datanode loop of
> {{XceiverClientGrpc.sendCommandWithRetry}} and moves on to the next replica
> for that request. Datanodes do not verify checksums for client reads by
> default, so silent corruption on one replica reaches the client as successful
> responses with bad bytes.
> h3. Trigger
> # {{ozone.client.stream.readblock.enable=true}}, a RATIS/THREE key.
> # Silent corruption of the block data on one replica (one byte flipped in
> place).
> # Read the key with a client whose stream starts on that replica.
> Which replica a stream starts on follows the order OM returns
> ({{ozone.network.topology.aware.read=true}} by default). Replicas at the same
> distance are shuffled per lookup, so a client hits the corrupt replica first
> on every read only when it is strictly the closest one (for example on the
> same host), and otherwise on a share of the reads.
> h3. Impact
> * A streaming read that starts on the corrupt replica fails, although two
> healthy replicas exist. The non streaming read of the same key succeeds.
> * No wrong data is returned.
> * Only with streaming block reads enabled (off by default).
> * On the current code the failure appears as a false EOF ({{Inconsistent read
> ... numBytesRead=-1}}) instead of the checksum error, because of HDDS-16680:
> the error arrives while {{StreamingReader.poll}} waits and is turned into end
> of stream before the retry decision. The fix for this issue depends on
> HDDS-16680 (apache/ozone PR #11388): only with both does the read succeed.
> h3. Reproduction
> PASS, deterministic in a mini cluster, unmodified source.
> [^TestBugCR1StreamReadChecksumNoFailover.java]
> ({{hadoop-ozone/integration-test}}) writes a 32 KiB RATIS/THREE key on three
> datanodes, flips one byte of the block file on the first pipeline node, then
> reads the key 5 times without and 5 times with streaming block reads. All
> datanodes have the client's address there, so topology aware read puts the
> corrupted node first for every read. A passing test means the defect is
> present. Results:
> ||Code||Non streaming||Streaming||Streaming failure||
> |unmodified|5/5|0/5|false EOF|
> |with the attached patch|5/5|0/5|false EOF (HDDS-16680)|
> |with PR #11388 (HDDS-16680)|5/5|0/5|{{Streaming read failed}} caused by
> {{OzoneChecksumException}}|
> |with both|5/5|5/5|none, each read skips the corrupt replica once|
> The short circuit read path appears to have the same gap
> ({{BlockInputStream}} rereads the same local file after a checksum error
> until its retries are used up); this is from code reading only, was not run
> and is not covered by the attached patch.
> h3. Patch
> [^CR-1-fail-over-streaming-read-on-checksum-mismatch.patch], against
> 1fb64c8e499161d1452ae6948ffd461fc9d2c302. It also applies to master at
> fd2f6092c6d, and together with PR #11388.
> {{ConnectionFailureUtils.unwrapCause}} (used only by
> {{StreamBlockInputStream}}) also returns an {{OzoneChecksumException}} found
> in the cause chain. On such an error the sequential path
> ({{handleExceptions}}) and the positioned path ({{readPositioned}}) release
> the failed reader and skip that replica, then open a stream on the next
> replica, as the non streaming path does per request. The skip lasts for one
> positioned read, or for one refill of the sequential read buffer (the set is
> cleared when the buffer is refilled). This cannot loop: within that scope
> every checksum failure either adds a replica not tried yet or rethrows, and a
> refill that succeeds returns data that the read then consumes. A checksum
> mismatch is a fault of the data in one range, not of the datanode, so the
> replica is not added to the stream's {{failedStreamingDatanodes}}: reads of
> other ranges, and later reads of the same range, start again from the first
> replica. There is no retry sleep, no block location refresh and no use of the
> read retry limit ({{ozone.client.max.read.retries}}) for this case, on either
> path; it is bounded by the number of replicas. When every replica fails the
> check, the read throws {{IOException("Streaming read failed")}} with the
> {{OzoneChecksumException}} of the last replica tried as its root cause
> (through an {{ExecutionException}}), the same exception as today, and later
> reads of other ranges still work.
> Covered in the existing unit {{TestStreamBlockInputStream}} with three mocked
> replicas, each case for sequential and positioned reads:
> {{testChecksumMismatchFailsOverToAnotherReplica}} (first replica corrupt, the
> read returns the data),
> {{testChecksumMismatchOnAllReplicasSurfacesChecksumError}} (all replicas
> corrupt, {{OzoneChecksumException}} is the root cause),
> {{testChecksumMismatchInOneRangeDoesNotAffectOtherRanges}} (offset 0 corrupt
> on all replicas, a read of offset 2 on the same stream returns the data) and
> {{testChecksumSkipDoesNotUseReadRetry}} (first replica corrupt, second
> answers CONTAINER_NOT_FOUND, one read retry allowed, the read returns the
> data from the third). The responses arrive synchronously, so the tests do not
> depend on HDDS-16680. Without the change the first fails with {{IOException:
> Streaming read failed}} caused by {{OzoneChecksumException}}, and the third
> fails on its sequential half because the failed reader is kept. With it
> {{TestStreamBlockInputStream}} (33 tests), {{TestConnectionFailureUtils}}
> (21), {{TestBlockInputStream}} (17), {{TestChunkInputStream}} (10),
> {{TestMultipartInputStream}} (6), and in a mini cluster
> {{TestStreamReadDatanodeFailover}} (3), {{TestStreamRead}} (4) and the
> integration {{TestStreamBlockInputStream}} (3) pass, checkstyle is clean.
> Related: HDDS-16873 proposes reporting replicas that fail client side
> checksum verification; it assumes the client already moves to another replica.
> Found by code review of the Ozone client read retry and failover paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 1fb64c8e499161d1452ae6948ffd461fc9d2c302. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
