[ 
https://issues.apache.org/jira/browse/HDDS-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-17877:
----------------------------------
    Labels: pull-request-available  (was: )

> Datanode should skip queued gRPC requests whose client deadline has already 
> expired
> -----------------------------------------------------------------------------------
>
>                 Key: HDDS-17877
>                 URL: https://issues.apache.org/jira/browse/HDDS-17877
>             Project: Apache Ozone
>          Issue Type: Improvement
>            Reporter: Janus Chow
>            Assignee: Janus Chow
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> {{XceiverServerGrpc}} serves all client requests on one fixed-size read 
> executor (ChunkReader, {{hdds.datanode.read.chunk.threads.per.volume}} × 
> volumes) with an unbounded {{LinkedBlockingQueue}}. When the datanode is 
> overloaded (slow disk, high iowait, page cache pressure), requests wait in 
> this queue for longer than the client timeout ({{ozone.client.read.timeout}}, 
> 30s, sent as the gRPC deadline by {{XceiverClientGrpc}}).
> The client then fails with {{DEADLINE_EXCEEDED}} and retries on another 
> replica, but the datanode still executes the stale request when it is 
> dequeued: it opens the block file, looks up the block in RocksDB, reads the 
> chunk from disk, and only then finds that the call is gone:
> {code} ERROR GrpcXceiverService: ContainerCommand send on error. Exception: 
> StatusRuntimeException: CANCELLED: client cancelled {code}
> In one incident we saw ~53k such errors on a single datanode within ten 
> minutes, all from ChunkReader threads. Each of them is disk I/O spent on a 
> request nobody waits for, taken exactly when the disks are already saturated, 
> which makes the queue even longer and causes more timeouts.
> h3. Proposal
> In {{GrpcXceiverService.send().onNext()}}, before dispatching, check the call 
> context:
> {code:java} if (Context.current().isCancelled()) { // client deadline expired 
> or call cancelled while queued 
> metrics.incSkippedExpiredRequests(request.getCmdType()); return; } {code}
> This works for the timeout case without depending on the busy executor: the 
> server creates the call context with {{withDeadline(..., 
> transport.getScheduledExecutorService())}} (grpc-java 
> {{ServerImpl.createContext}}), so it is cancelled by the transport timer when 
> the deadline passes, even while the request is still queued. On expiry gRPC 
> also closes the stream itself, so nothing needs to be sent back.
> Notes:
> Explicit cancels before the deadline are not fully covered: grpc-java 
> delivers the {{ContextCloser}} through the same application executor, so it 
> may also wait in the queue. The deadline case is the one that matters under 
> overload.
> Streaming {{ReadBlock}} calls have no deadline, so they are not affected.
> Add a counter for skipped requests per command type so the overload is 
> visible (complements HDDS-17168).
> h3. Related
> HDDS-17168: expose read executor utilization and queue length
> HDDS-17635: cancelled streaming ReadBlock should not be treated as a 
> container failure
> HDDS-11461: improve the impact of DataNode I/O



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to