[
https://issues.apache.org/jira/browse/HDDS-16869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16869:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: Degraded reads should read and decode only the requested range
> ------------------------------------------------------------------
>
> Key: HDDS-16869
> URL: https://issues.apache.org/jira/browse/HDDS-16869
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> In reconstruction mode, \{{ECBlockReconstructedInputStream}} works in whole
> stripes. \{{ECBlockReconstructedStripeInputStream}} only accepts
> stripe-aligned
> positions (\{{seek}} throws if \{{pos % stripeSize != 0}}) and reads and
> decodes
> the full stripe, so any read, however small, costs data-unit-count full cells
> from the datanodes plus a decode of the whole stripe. This includes reads on
> indexes that are healthy, which could be served directly. Small reads at
> scattered offsets, such as columnar-file footer and index reads, are the worst
> case.
> h2. HDFS behavior
> {\{StripedBlockUtil.divideByteRangeIntoStripes}} splits the requested range
> into
> {\{AlignedStripe}}s that cover only the requested cell offsets. The reader
> reads
> those spans from healthy data indexes as plain data and, for a missing index,
> reads the same span from enough other indexes and decodes only that span. The
> stateful reader reads from the current position to the end of the stripe, not
> from the stripe start; pread reads exactly the requested range.
> h2. Proposal
> * For a requested range, compute per-index spans (offset and length within the
> cell, per stripe).
> * Read spans on available data indexes directly, without decoding.
> * For spans on missing indexes, take the union span in cell-offset space, read
> that same span from data-unit-count available indexes (data or parity) and
> decode only that span. The Reed-Solomon decoder works on equal-length inputs,
> so any equal-offset span can be decoded.
> * Keep reading full stripes when the request covers whole stripes (large
> sequential reads), where the two approaches read the same amount.
> * Datanode reads of a span still respect checksum boundaries, which the
> per-index block stream already handles.
> h2. Compatibility
> Client only. No wire, configuration or data change.
> h2. Testing
> * Counting stream factory: a small unaligned read in reconstruction mode reads
> only the requested span from healthy indexes, and only the decode span from
> the inputs needed for a missing index.
> * Correctness matrix: in-cell, cross-cell, cross-stripe and
> partial-last-stripe
> ranges, with 1 to parity-count missing indexes, for \{{rs-3-2}}, \{{rs-6-3}}
> and
> {\{rs-10-4}}, comparing bytes with the written data.
> * Large sequential reads read the same number of bytes as today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
