[
https://issues.apache.org/jira/browse/HDDS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng resolved HDDS-16258.
-------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> Refactor streaming block reads and fix checksum verification for
> variable-sized chunks
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16258
> URL: https://issues.apache.org/jira/browse/HDDS-16258
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chung-En Lee
> Assignee: Chung-En Lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> Streaming ReadBlock can fail checksum verification for valid data after
> flush/hsync produces uneven chunks. Checksum intervals restart at each chunk,
> so flattening the stored checksums and verifying the response as one
> continuous sequence gives incorrect boundaries.
> HDDS-15857 added server-side range alignment and shared per-chunk
> verification. It did not add chunk metadata to the response protocol or
> update client verification; the production path still uses the legacy
> flattened checksum collector.
> For example, with bytesPerChecksum=4:
> {noformat}
> Stored chunks: [A B C D E] [F G H I J K L]
> Stored CRC ranges: [ABCD] [E] [FGHI] [JKL]
> Flat client ranges: [ABCD] [EFGH] [IJKL]
> {noformat}
> The fix has two parts:
> # Refactor {{KeyValueHandler#readBlockImpl}} by evolving
> {{ReadBlockComputation}} into one {{BlockReadCursor}} that owns aligned
> offsets, response lengths, overlapping chunk selection, and read progress.
> # Replace the flattened response checksum field with metadata for only the
> chunks overlapping each response. Preserve their original offsets and stored
> checksums, and use {{Checksum.validateChecksums}} on both the server and
> {{StreamBlockInputStream}} to verify each range relative to its chunk.
> Cover shifted starts, short final checksum intervals, {{NONE}}, empty data,
> and missing or incomplete metadata. Verification must preserve buffer
> positions and reject checksum failures before the client enqueues data. Keep
> the existing verification settings and stream error/cleanup behavior; fail
> unexpected EOF before sending an incomplete checksum interval. Remove
> {{testVariableChunks}} and exercise the same production path in tests,
> including uneven persisted chunks and full/seek/range reads.
> The proposed streaming response change intentionally requires matching client
> and datanode implementations, with no legacy fallback. Streaming reads are
> disabled by default. Keep {{proto.lock}} unchanged until the next release.
> This ticket is limited to the cursor refactor and checksum correctness in
> streaming ReadBlock; general request validation, ordinary chunk reads, and
> the stored format are outside its scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
