[
https://issues.apache.org/jira/browse/HDDS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-16258:
--------------------------------
Description:
Streaming ReadBlock can fail checksum verification for valid data after
flush/hsync produces uneven chunks. Checksum intervals restart at each chunk,
so flattening the stored checksums and verifying the response as one continuous
sequence gives incorrect boundaries.
HDDS-15857 added server-side range alignment and shared per-chunk verification.
It did not add chunk metadata to the response protocol or update client
verification; the production path still uses the legacy flattened checksum
collector.
For example, with bytesPerChecksum=4:
{noformat}
Stored chunks: [A B C D E] [F G H I J K L]
Stored CRC ranges: [ABCD] [E] [FGHI] [JKL]
Flat client ranges: [ABCD] [EFGH] [IJKL]
{noformat}
The fix has two parts:
# Refactor {{KeyValueHandler#readBlockImpl}} by evolving
{{ReadBlockComputation}} into one {{BlockReadCursor}} that owns aligned
offsets, response lengths, overlapping chunk selection, and read progress.
# Replace the flattened response checksum field with metadata for only the
chunks overlapping each response. Preserve their original offsets and stored
checksums, and use {{Checksum.validateChecksums}} on both the server and
{{StreamBlockInputStream}} to verify each range relative to its chunk.
Cover shifted starts, short final checksum intervals, {{NONE}}, empty data, and
missing or incomplete metadata. Verification must preserve buffer positions and
reject checksum failures before the client enqueues data. Keep the existing
verification settings and stream error/cleanup behavior; fail unexpected EOF
before sending an incomplete checksum interval. Remove {{testVariableChunks}}
and exercise the same production path in tests, including uneven persisted
chunks and full/seek/range reads.
The proposed streaming response change intentionally requires matching client
and datanode implementations, with no legacy fallback. Streaming reads are
disabled by default. Keep {{proto.lock}} unchanged until the next release. This
ticket is limited to the cursor refactor and checksum correctness in streaming
ReadBlock; general request validation, ordinary chunk reads, and the stored
format are outside its scope.
was:
*Description:*
*Background* As part of HDDS-15857, we fixed the server-side checksum
calculation in {{KeyValueHandler}} for variable-sized chunks and updated the
protocol to optionally include {{chunkInfoList}} in the
{{{}ReadBlockResponseProto{}}}. To keep the original patch focused and
manageable, the client-side verification logic has been split into this ticket.
*Problem* Currently, when a client actively triggers a flush or hsync, the
resulting chunk size may not perfectly align with {{{}bytesPerChunk{}}}. The
client-side {{StreamBlockInputStream}} incorrectly assumes all chunks (except
the last one) have a fixed size when verifying checksums.
Because of this fixed-size assumption, the client cannot correctly map the
offsets for variable-sized chunks. This leads to misaligned checksum
verification, causing false {{{}OzoneChecksumException{}}}s when processing
chunks that are variable-sized or smaller than {{{}bytesPerChunk{}}}.
*Proposed Solution* Update the client-side streaming read architecture to be
aware of exact chunk boundaries:
# *Boundary-Aware Verification:* In {{{}StreamBlockInputStream#onNext{}}}, if
the server response contains {{chunkInfoList}} and checksum verification is
enabled:
** Iterate through each chunk to identify the exact overlapping region between
the read block and the chunk boundaries.
** Calculate the correct start and end checksum indices using
{{bytesPerChecksum}} relative to that specific chunk's starting offset.
# *Backward Compatibility:* If {{chunkInfoList}} is absent from the response
(e.g., when reading from an older DataNode version), the client must gracefully
fall back to the legacy verification path ({{{}Checksum.verifyChecksum(data,
checksumData, 0){}}}) to avoid breaking upgrades.
*Testing*
* Update {{TestStreamBlockInputStream.java}} to validate boundary-aware
verification using {{{}chunkInfoList{}}}.
* Add integration tests in {{TestStreamRead.java}} (e.g.,
{{testSmallChunksWithLargeChecksum}} and
{{{}testShiftedChecksumBoundaryVerification{}}}) to cover varying/small chunk
sizes with larger {{{}bytesPerChecksum{}}}.
Summary: Refactor streaming block reads and fix checksum verification
for variable-sized chunks (was: Support boundary-aware checksum verification
for variable-sized chunks in StreamBlockInputStream)
> Refactor streaming block reads and fix checksum verification for
> variable-sized chunks
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16258
> URL: https://issues.apache.org/jira/browse/HDDS-16258
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chung-En Lee
> Assignee: Chung-En Lee
> Priority: Major
> Labels: pull-request-available
>
> Streaming ReadBlock can fail checksum verification for valid data after
> flush/hsync produces uneven chunks. Checksum intervals restart at each chunk,
> so flattening the stored checksums and verifying the response as one
> continuous sequence gives incorrect boundaries.
> HDDS-15857 added server-side range alignment and shared per-chunk
> verification. It did not add chunk metadata to the response protocol or
> update client verification; the production path still uses the legacy
> flattened checksum collector.
> For example, with bytesPerChecksum=4:
> {noformat}
> Stored chunks: [A B C D E] [F G H I J K L]
> Stored CRC ranges: [ABCD] [E] [FGHI] [JKL]
> Flat client ranges: [ABCD] [EFGH] [IJKL]
> {noformat}
> The fix has two parts:
> # Refactor {{KeyValueHandler#readBlockImpl}} by evolving
> {{ReadBlockComputation}} into one {{BlockReadCursor}} that owns aligned
> offsets, response lengths, overlapping chunk selection, and read progress.
> # Replace the flattened response checksum field with metadata for only the
> chunks overlapping each response. Preserve their original offsets and stored
> checksums, and use {{Checksum.validateChecksums}} on both the server and
> {{StreamBlockInputStream}} to verify each range relative to its chunk.
> Cover shifted starts, short final checksum intervals, {{NONE}}, empty data,
> and missing or incomplete metadata. Verification must preserve buffer
> positions and reject checksum failures before the client enqueues data. Keep
> the existing verification settings and stream error/cleanup behavior; fail
> unexpected EOF before sending an incomplete checksum interval. Remove
> {{testVariableChunks}} and exercise the same production path in tests,
> including uneven persisted chunks and full/seek/range reads.
> The proposed streaming response change intentionally requires matching client
> and datanode implementations, with no legacy fallback. Streaming reads are
> disabled by default. Keep {{proto.lock}} unchanged until the next release.
> This ticket is limited to the cursor refactor and checksum correctness in
> streaming ReadBlock; general request validation, ordinary chunk reads, and
> the stored format are outside its scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
