Janus Chow created HDDS-17877:
---------------------------------

             Summary: Datanode should skip queued gRPC requests whose client 
deadline has already expired
                 Key: HDDS-17877
                 URL: https://issues.apache.org/jira/browse/HDDS-17877
             Project: Apache Ozone
          Issue Type: Improvement
            Reporter: Janus Chow
            Assignee: Janus Chow


h3. Problem

{{XceiverServerGrpc}} serves all client requests on one fixed-size read 
executor (ChunkReader, {{hdds.datanode.read.chunk.threads.per.volume}} × 
volumes) with an unbounded {{LinkedBlockingQueue}}. When the datanode is 
overloaded (slow disk, high iowait, page cache pressure), requests wait in this 
queue for longer than the client timeout ({{ozone.client.read.timeout}}, 30s, 
sent as the gRPC deadline by {{XceiverClientGrpc}}).

The client then fails with {{DEADLINE_EXCEEDED}} and retries on another 
replica, but the datanode still executes the stale request when it is dequeued: 
it opens the block file, looks up the block in RocksDB, reads the chunk from 
disk, and only then finds that the call is gone:

{code} ERROR GrpcXceiverService: ContainerCommand send on error. Exception: 
StatusRuntimeException: CANCELLED: client cancelled {code}

In one incident we saw ~53k such errors on a single datanode within ten 
minutes, all from ChunkReader threads. Each of them is disk I/O spent on a 
request nobody waits for, taken exactly when the disks are already saturated, 
which makes the queue even longer and causes more timeouts.

h3. Proposal

In {{GrpcXceiverService.send().onNext()}}, before dispatching, check the call 
context:

{code:java} if (Context.current().isCancelled()) { // client deadline expired 
or call cancelled while queued 
metrics.incSkippedExpiredRequests(request.getCmdType()); return; } {code}

This works for the timeout case without depending on the busy executor: the 
server creates the call context with {{withDeadline(..., 
transport.getScheduledExecutorService())}} (grpc-java 
{{ServerImpl.createContext}}), so it is cancelled by the transport timer when 
the deadline passes, even while the request is still queued. On expiry gRPC 
also closes the stream itself, so nothing needs to be sent back.

Notes:

Explicit cancels before the deadline are not fully covered: grpc-java delivers 
the {{ContextCloser}} through the same application executor, so it may also 
wait in the queue. The deadline case is the one that matters under overload.
Streaming {{ReadBlock}} calls have no deadline, so they are not affected.
Add a counter for skipped requests per command type so the overload is visible 
(complements HDDS-17168).
h3. Related

HDDS-17168: expose read executor utilization and queue length
HDDS-17635: cancelled streaming ReadBlock should not be treated as a container 
failure
HDDS-11461: improve the impact of DataNode I/O



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to