Siyao Meng created HDDS-17327:
---------------------------------
Summary: Topology-aware read tries an unresponsive nearest
datanode first for every ReadChunk
Key: HDDS-17327
URL: https://issues.apache.org/jira/browse/HDDS-17327
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Siyao Meng
Assignee: Siyao Meng
With {{ozone.network.topology.aware.read}}=true (the default),
{{XceiverClientGrpc.sortDatanodes}} orders the replicas by distance for every
ReadChunk, and {{sendCommandWithRetry}} tries them in that order. Nothing
remembers that a datanode just failed: {{XceiverClientGrpc}} keeps no failed
datanodes, and the exclusion in {{ContainerProtocolCalls.tryEachDatanode}} only
lasts for one call. If the nearest datanode hangs (e.g. a stuck disk or a long
GC pause) instead of refusing connections, every ReadChunk waits
{{ozone.client.read.timeout}} (30s) before the next replica answers. A
ReadChunk is sent for each read call within a chunk, so by calculation reading
a 256MB block with 4MB reads takes up to 64 × 30s, about 32 minutes, longer.
HDDS-16419 removes the nested N × N failover, but keeps the same order for each
call.
Proposed fix: keep a set of datanodes that failed recently, with an expiry like
{{ozone.client.exclude.nodes.expiry.time}} on the write side, and try them last.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
