[
https://issues.apache.org/jira/browse/HDDS-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17877:
----------------------------------
Labels: pull-request-available (was: )
> Datanode should skip queued gRPC requests whose client deadline has already
> expired
> -----------------------------------------------------------------------------------
>
> Key: HDDS-17877
> URL: https://issues.apache.org/jira/browse/HDDS-17877
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {{XceiverServerGrpc}} serves all client requests on one fixed-size read
> executor (ChunkReader, {{hdds.datanode.read.chunk.threads.per.volume}} ×
> volumes) with an unbounded {{LinkedBlockingQueue}}. When the datanode is
> overloaded (slow disk, high iowait, page cache pressure), requests wait in
> this queue for longer than the client timeout ({{ozone.client.read.timeout}},
> 30s, sent as the gRPC deadline by {{XceiverClientGrpc}}).
> The client then fails with {{DEADLINE_EXCEEDED}} and retries on another
> replica, but the datanode still executes the stale request when it is
> dequeued: it opens the block file, looks up the block in RocksDB, reads the
> chunk from disk, and only then finds that the call is gone:
> {code} ERROR GrpcXceiverService: ContainerCommand send on error. Exception:
> StatusRuntimeException: CANCELLED: client cancelled {code}
> In one incident we saw ~53k such errors on a single datanode within ten
> minutes, all from ChunkReader threads. Each of them is disk I/O spent on a
> request nobody waits for, taken exactly when the disks are already saturated,
> which makes the queue even longer and causes more timeouts.
> h3. Proposal
> In {{GrpcXceiverService.send().onNext()}}, before dispatching, check the call
> context:
> {code:java} if (Context.current().isCancelled()) { // client deadline expired
> or call cancelled while queued
> metrics.incSkippedExpiredRequests(request.getCmdType()); return; } {code}
> This works for the timeout case without depending on the busy executor: the
> server creates the call context with {{withDeadline(...,
> transport.getScheduledExecutorService())}} (grpc-java
> {{ServerImpl.createContext}}), so it is cancelled by the transport timer when
> the deadline passes, even while the request is still queued. On expiry gRPC
> also closes the stream itself, so nothing needs to be sent back.
> Notes:
> Explicit cancels before the deadline are not fully covered: grpc-java
> delivers the {{ContextCloser}} through the same application executor, so it
> may also wait in the queue. The deadline case is the one that matters under
> overload.
> Streaming {{ReadBlock}} calls have no deadline, so they are not affected.
> Add a counter for skipped requests per command type so the overload is
> visible (complements HDDS-17168).
> h3. Related
> HDDS-17168: expose read executor utilization and queue length
> HDDS-17635: cancelled streaming ReadBlock should not be treated as a
> container failure
> HDDS-11461: improve the impact of DataNode I/O
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]