Sergey Soldatov created HDDS-16680:
--------------------------------------
Summary: StreamBlockInputStream returns EOF instead of failing
over when an error response arrives during poll()
Key: HDDS-16680
URL: https://issues.apache.org/jira/browse/HDDS-16680
Project: Apache Ozone
Issue Type: Bug
Affects Versions: 2.2.1
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
A streaming read that gets an error response from a datanode (e.g.
UNKNOWN_BCSID, CONTAINER_NOT_FOUND, or a token failure) can return EOF instead
of throwing. Retry and failover to another replica are skipped, and
KeyInputStream fails with:
{code:java}
java.io.IOException: Inconsistent read for blockID=conID: 1 locID:
117883640217600001 bcsId: 2 replicaIndex: null length=32768 position=0
numBytesToRead=32768 numBytesRead=-1
{code}
Root cause: StreamingReader.poll() runs checkError() at the top of its loop,
then waits up to 100 ms on the response queue, then returns null (end of
stream) if future.isDone(). If onNext() (via failOnErrorResponse()) or
onError() fails the future during that wait, poll() returns null for a stream
that failed. The error is lost, and read() turns the null into EOF.
Regression: two changes combined to cause this:
- HDDS-15479 moved the isDone() check to after the queue wait (to avoid
dropping the last queued item). That put the whole wait between checkError()
and isDone().
- HDDS-16207 made error responses and gRPC errors other than CANCELLED fail the
future right away. Before that, they ended in the read timeout, and
TimeoutIOException triggered a retry/failover (HDDS-15521). After HDDS-16207,
the failure almost always lands during the wait.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
