[
https://issues.apache.org/jira/browse/HDDS-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16868:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: Recover from a failed cell read within the stripe instead of re-reading
> ---------------------------------------------------------------------------
>
> Key: HDDS-16868
> URL: https://issues.apache.org/jira/browse/HDDS-16868
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> When one cell read fails, the EC readers throw away data they have already
> read
> and read it again:
> # *Failover discards the whole call.* \{{ECBlockInputStreamProxy.read}} marks
> the
> caller buffer and records the block position before calling the healthy
> reader. On \{{BadDataLocationException}} it switches to the reconstruction
> reader at the *start* of the call, calls \{{buf.reset()}} and reads again, so
> every cell already delivered by the call is read a second time (and decoded).
> The spare-location retry inside \{{ECBlockInputStream.read}} does the same: it
> resets the buffer and seeks back to the start of the call.
> # *Whole-stripe retry.* In
> \{{ECBlockReconstructedStripeInputStream.read(ByteBuffer[])}},
> any \{{IOException}} while loading a stripe re-seeks to the stripe start,
> clears all buffers and re-reads every cell of the stripe, including the ones
> that were read successfully.
> # *Submit-order waits.* \{{loadDataBuffersFromStream}} waits on the futures in
> the order they were submitted and waits for all of them even after one has
> failed. Only after the whole round does it pick a replacement index, so a
> failure that happened early still waits for the slowest read before recovery
> starts.
> h2. HDFS behavior
> In \{{StripeReader.readStripe}}, a failed chunk read marks that reader
> {\{shouldSkip}}, schedules one more parity read (\{{readParityChunks}} with
> the
> number of missing chunks), keeps the chunks already fetched, takes futures in
> completion order (\{{getNextCompletedStripedRead}}), cancels the rest once
> enough
> chunks are present (\{{clearFutures}}) and decodes only the missing cells.
> Later
> stripes skip the failed index from the start.
> h2. Proposal
> *Option A (incremental, recommended).*
> * Proxy failover resumes at the first byte not yet delivered. The healthy
> reader's position at the failure already counts only completed reads; seek
> the reconstruction reader there and keep filling the caller buffer from the
> matching offset, without \{{buf.reset()}}. Apply the same rule to the spare
> retry in \{{ECBlockInputStream}}.
> * In \{{ECBlockReconstructedStripeInputStream}}, consume reads through an
> {\{ExecutorCompletionService}}. On a failed index, record it, submit one
> replacement read (next parity, or spare location) at once, and keep the
> buffers that are already complete. Once the data unit count of buffers is
> ready, cancel or ignore the rest and decode. Retry the whole stripe only when
> not enough indexes remain.
> *Option B.* Replace the two readers with one stripe reader, modeled on HDFS
> {\{StripeReader}}, used by both healthy and degraded reads, that decodes
> inline
> per stripe and skips failed indexes in later stripes. This removes the
> proxy-level failover but is a large refactor of the EC read path; agree on it
> on the Jira before starting.
> Either option can also replace the byte-by-byte copy in
> {\{ECBlockReconstructedInputStream.readBufferToDest}} with a bulk
> \{{ByteBuffer}}
> copy.
> h2. Compatibility
> Client only. No wire, configuration or data change.
> h2. Testing
> * Counting stream factory: after a failure in the middle of a multi-cell read,
> no cell that was read successfully before the failure is read again, and the
> returned bytes match the written data.
> * A failed index triggers exactly one replacement read; good buffers of the
> stripe are not re-read.
> * Latch test: with one index slow and another failing, the replacement read
> starts before the slow read completes.
> * Surplus reads are cancelled or ignored once enough buffers are ready.
> * Existing \{{TestECBlockInputStreamProxy}} and
> {\{TestECBlockReconstructedStripeInputStream}} failover cases keep passing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
