Janus Chow created HDDS-17877:
---------------------------------
Summary: Datanode should skip queued gRPC requests whose client
deadline has already expired
Key: HDDS-17877
URL: https://issues.apache.org/jira/browse/HDDS-17877
Project: Apache Ozone
Issue Type: Improvement
Reporter: Janus Chow
Assignee: Janus Chow
h3. Problem
{{XceiverServerGrpc}} serves all client requests on one fixed-size read
executor (ChunkReader, {{hdds.datanode.read.chunk.threads.per.volume}} ×
volumes) with an unbounded {{LinkedBlockingQueue}}. When the datanode is
overloaded (slow disk, high iowait, page cache pressure), requests wait in this
queue for longer than the client timeout ({{ozone.client.read.timeout}}, 30s,
sent as the gRPC deadline by {{XceiverClientGrpc}}).
The client then fails with {{DEADLINE_EXCEEDED}} and retries on another
replica, but the datanode still executes the stale request when it is dequeued:
it opens the block file, looks up the block in RocksDB, reads the chunk from
disk, and only then finds that the call is gone:
{code} ERROR GrpcXceiverService: ContainerCommand send on error. Exception:
StatusRuntimeException: CANCELLED: client cancelled {code}
In one incident we saw ~53k such errors on a single datanode within ten
minutes, all from ChunkReader threads. Each of them is disk I/O spent on a
request nobody waits for, taken exactly when the disks are already saturated,
which makes the queue even longer and causes more timeouts.
h3. Proposal
In {{GrpcXceiverService.send().onNext()}}, before dispatching, check the call
context:
{code:java} if (Context.current().isCancelled()) { // client deadline expired
or call cancelled while queued
metrics.incSkippedExpiredRequests(request.getCmdType()); return; } {code}
This works for the timeout case without depending on the busy executor: the
server creates the call context with {{withDeadline(...,
transport.getScheduledExecutorService())}} (grpc-java
{{ServerImpl.createContext}}), so it is cancelled by the transport timer when
the deadline passes, even while the request is still queued. On expiry gRPC
also closes the stream itself, so nothing needs to be sent back.
Notes:
Explicit cancels before the deadline are not fully covered: grpc-java delivers
the {{ContextCloser}} through the same application executor, so it may also
wait in the queue. The deadline case is the one that matters under overload.
Streaming {{ReadBlock}} calls have no deadline, so they are not affected.
Add a counter for skipped requests per command type so the overload is visible
(complements HDDS-17168).
h3. Related
HDDS-17168: expose read executor utilization and queue length
HDDS-17635: cancelled streaming ReadBlock should not be treated as a container
failure
HDDS-11461: improve the impact of DataNode I/O
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]