Andrey Yarovoy created HDDS-16873:
-------------------------------------
Summary: Report replicas that fail client-side checksum
verification
Key: HDDS-16873
URL: https://issues.apache.org/jira/browse/HDDS-16873
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
The datanode does not verify chunk checksums for client reads.
{\{KeyValueHandler}} validates them only for Ratis-internal reads; for client
reads "the client is expected to validate". When the client finds a mismatch
(\{{OzoneChecksumException}}), it reads another replica (Ratis) or treats the
location as failed and reconstructs (EC), and the read succeeds. Nobody is told:
* the read returned \{{SUCCESS}} on the datanode, so \{{HddsDispatcher}} does
not
trigger an on-demand container scan (it does so only for non-\{{SUCCESS}}
results);
* SCM is not informed.
The corrupt replica stays in service until the background data scanner reaches
it. Until then every reader of that data pays the retry or the reconstruction,
and redundancy is lower than SCM believes.
h2. HDFS behavior
{\{DFSInputStream}} / \{{DFSStripedInputStream}} collect checksum failures in
{\{CorruptedBlocks}} and call \{{reportCheckSumFailure}} in \{{finally}}, which
reports the replica to the NameNode; the NameNode marks it corrupt and
re-replicates or reconstructs it.
h2. Proposal
Client asks the datanode to scan.* Add a container
request with which the client reports a checksum failure for a block on that
datanode. The datanode does not trust the report: it runs an on-demand scan of
the container (\{{OnDemandContainerScanner}}), rate-limited by the scanner's
existing per-container minimum gap. If the scan confirms the corruption, the
container is marked unhealthy and the existing ICR -> SCM ReplicationManager
path repairs it. The client sends the report asynchronously, at most once per
datanode and container per stream, and it never fails or delays the read.
Security: the request must be authorized like a read of the block (block /
container token in secure mode), so a client can only trigger scans on data it
can read, and the scanner's rate limit bounds the load it can cause. Assess it
against \{{THREAT_MODEL.md}} before implementation.
h2. Testing
* Unit: a checksum failure on read sends one report per datanode and container,
asynchronously; the read still returns correct data; no report goes to a
datanode version without support.
* Datanode unit: the request triggers an on-demand scan and is rate-limited by
the scanner gap; an unauthorized request is rejected in secure mode.
* Integration, extending the existing container scanner integration tests
(\{{TestOnDemandContainerScannerIntegration}}): corrupt a chunk on one replica,
read the key, and check that the container becomes unhealthy on that datanode
and is repaired; cover a Ratis key and an EC key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
