Andrey Yarovoy created HDDS-16868:
-------------------------------------
Summary: EC: Recover from a failed cell read within the stripe
instead of re-reading
Key: HDDS-16868
URL: https://issues.apache.org/jira/browse/HDDS-16868
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
When one cell read fails, the EC readers throw away data they have already read
and read it again:
# *Failover discards the whole call.* \{{ECBlockInputStreamProxy.read}} marks
the
caller buffer and records the block position before calling the healthy
reader. On \{{BadDataLocationException}} it switches to the reconstruction
reader at the *start* of the call, calls \{{buf.reset()}} and reads again, so
every cell already delivered by the call is read a second time (and decoded).
The spare-location retry inside \{{ECBlockInputStream.read}} does the same: it
resets the buffer and seeks back to the start of the call.
# *Whole-stripe retry.* In
\{{ECBlockReconstructedStripeInputStream.read(ByteBuffer[])}},
any \{{IOException}} while loading a stripe re-seeks to the stripe start,
clears all buffers and re-reads every cell of the stripe, including the ones
that were read successfully.
# *Submit-order waits.* \{{loadDataBuffersFromStream}} waits on the futures in
the order they were submitted and waits for all of them even after one has
failed. Only after the whole round does it pick a replacement index, so a
failure that happened early still waits for the slowest read before recovery
starts.
h2. HDFS behavior
In \{{StripeReader.readStripe}}, a failed chunk read marks that reader
{\{shouldSkip}}, schedules one more parity read (\{{readParityChunks}} with the
number of missing chunks), keeps the chunks already fetched, takes futures in
completion order (\{{getNextCompletedStripedRead}}), cancels the rest once
enough
chunks are present (\{{clearFutures}}) and decodes only the missing cells. Later
stripes skip the failed index from the start.
h2. Proposal
*Option A (incremental, recommended).*
* Proxy failover resumes at the first byte not yet delivered. The healthy
reader's position at the failure already counts only completed reads; seek
the reconstruction reader there and keep filling the caller buffer from the
matching offset, without \{{buf.reset()}}. Apply the same rule to the spare
retry in \{{ECBlockInputStream}}.
* In \{{ECBlockReconstructedStripeInputStream}}, consume reads through an
{\{ExecutorCompletionService}}. On a failed index, record it, submit one
replacement read (next parity, or spare location) at once, and keep the
buffers that are already complete. Once the data unit count of buffers is
ready, cancel or ignore the rest and decode. Retry the whole stripe only when
not enough indexes remain.
*Option B.* Replace the two readers with one stripe reader, modeled on HDFS
{\{StripeReader}}, used by both healthy and degraded reads, that decodes inline
per stripe and skips failed indexes in later stripes. This removes the
proxy-level failover but is a large refactor of the EC read path; agree on it
on the Jira before starting.
Either option can also replace the byte-by-byte copy in
{\{ECBlockReconstructedInputStream.readBufferToDest}} with a bulk
\{{ByteBuffer}}
copy.
h2. Compatibility
Client only. No wire, configuration or data change.
h2. Testing
* Counting stream factory: after a failure in the middle of a multi-cell read,
no cell that was read successfully before the failure is read again, and the
returned bytes match the written data.
* A failed index triggers exactly one replacement read; good buffers of the
stripe are not re-read.
* Latch test: with one index slow and another failing, the replacement read
starts before the slow read completes.
* Surplus reads are cancelled or ignored once enough buffers are ready.
* Existing \{{TestECBlockInputStreamProxy}} and
{\{TestECBlockReconstructedStripeInputStream}} failover cases keep passing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
