[ 
https://issues.apache.org/jira/browse/HDDS-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated HDDS-17561:
-----------------------------------
    Description: 
h3. Problem
A blocking {{XceiverClientGrpc#sendCommand}} (ReadChunk, GetBlock and the other 
read-path commands) is built on top of the async path. For every request it 
opens a new bi-directional gRPC stream, sends one request, half-closes the 
stream, and waits on a {{CompletableFuture}}. The reply then travels:
# gRPC transport (Netty event loop) ->
# gRPC's default executor, where the {{StreamObserver}} parses the response and 
completes the future ->
# the caller thread, woken up from {{get()}}.

So each blocking request pays two thread hand-offs plus the setup and teardown 
of an HTTP/2 stream on both the client and the datanode, even though the caller 
is just sitting idle waiting for the reply.

h3. Proposal
* *Reply on the waiting thread.* New {{CallerRunsReplyFuture}}: a 
{{CompletableFuture}} that is also the call's {{Executor}}. While a thread 
waits in {{get()}}, the gRPC callbacks of the call are queued and run by that 
thread, so the response is parsed and the future completed on the caller (the 
same technique gRPC blocking stubs use). Once the waiter returns, queued and 
later callbacks run directly. Only {{get()}} drives the callbacks, so 
{{join()}} or a dependent stage alone never completes the future; 
{{sendCommandWithRetry}} always calls {{get()}}.
* *Pooled streams for untraced blocking calls.* A blocking call with no valid 
span on the calling thread leases an idle stream of the channel, or opens one, 
and sends exactly one request on it at a time. That is the default while 
tracing is off, but an application's own span also counts, and those calls keep 
using one stream per call. The datanode serves the requests of a stream in 
order, so the next reply always belongs to the current request and no request 
IDs are needed. After the reply the stream goes back to the pool. The number of 
streams is bounded by the client semaphore (max outstanding requests).
* *Client-side deadline.* A pooled stream has no gRPC deadline, so the request 
deadline is enforced by the waiter in {{get()}}. On timeout the request fails 
with {{DEADLINE_EXCEEDED}}, as before, and the stream is cancelled and dropped.
* *Stream lifecycle.*
** Idle streams are half-closed after one minute by a shared daemon thread, and 
immediately when the channel leaves {{READY}} (e.g. the datanode sends GOAWAY 
on shutdown), so open streams do not hold up the datanode's graceful shutdown.
** A stream is dropped, never reused, when it fails, is completed by the 
datanode, receives an unsolicited reply, or when its request is abandoned 
(timeout, interrupt). An abandon acts only while the request still owns the 
stream, so it cannot cancel a stream that has gone on to another request.
* *Configuration.* {{ozone.client.grpc.pooled.streams.enabled}} (default true). 
When false, these calls open one stream per call with a gRPC deadline. Replies 
are still delivered on the waiting thread.
* *Unchanged paths.* Traced blocking calls still open one stream per call 
(trace headers are per stream), but also complete on the waiting thread. 
{{sendCommandOnAllNodes}} and the async path 
({{sendCommandAsync(ContainerCommandRequestProto)}}) also use one stream per 
call and complete on gRPC's executor.
* *Smaller fix along the way.* {{sortDatanodes}} returns early for single-node 
pipelines.
* {{README.gRPC.md}} is updated to describe the new behavior.

h3. Compatibility
Client-only change, with one new client configuration key. No protobuf, RPC or 
datanode changes; the datanode already handles several sequential requests on 
one stream.

h3. Testing
New {{TestXceiverClientGrpcCallerRuns}} (22 tests, most against a local Netty 
gRPC server) covers:
* replies completing on the caller for {{get()}};
* stream reuse for sequential calls, distinct streams for concurrent calls, no 
more streams than semaphore permits, and pooling turned off by configuration;
* deadline expiry and interrupted waiters, including an abandon that must not 
cancel the next request on the stream;
* stream errors, a stream that ends after it was leased, and unsolicited and 
duplicate replies;
* idle-timeout half-close and server shutdown not being held up;
* traced calls using a stream per call, dead channels failing fast without 
leaking permits, and fan-out delivery and permit release.


  was:
h3. Problem
A blocking {{XceiverClientGrpc#sendCommand}} (ReadChunk, GetBlock and the other 
read-path commands) is built on top of the async path. For every request it 
opens a new bi-directional gRPC stream, sends one request, half-closes the 
stream, and waits on a {{CompletableFuture}}. The reply then travels:
# gRPC transport (Netty event loop) ->
# gRPC's default executor, where the {{StreamObserver}} parses the response and 
completes the future ->
# the caller thread, woken up from {{get()}}.

So each blocking request pays two thread hand-offs plus the setup and teardown 
of an HTTP/2 stream on both the client and the datanode, even though the caller 
is just sitting idle waiting for the reply.

h3. Proposal
* *Reply on the waiting thread.* New {{CallerRunsReplyFuture}}: a 
{{CompletableFuture}} that is also the call's {{Executor}}. While a thread 
waits in {{get()}} / {{join()}}, the gRPC callbacks of the call are queued and 
run by that thread, so the response is parsed and the future completed on the 
caller (the same technique gRPC blocking stubs use). Before the caller starts 
waiting and after it returns, callbacks run on a shared daemon executor, never 
on the transport thread.
* *Pooled streams for untraced blocking calls.* A blocking call without a trace 
context (the production default) leases an idle stream of the channel, or opens 
one, and sends exactly one request on it at a time. The datanode serves the 
requests of a stream in order, so the next reply always belongs to the current 
request and no request IDs are needed. After the reply the stream goes back to 
the pool. The number of streams is bounded by the client semaphore (max 
outstanding requests).
* *Client-side deadline.* A pooled stream has no gRPC deadline, so the request 
deadline is enforced by the waiter (and by a timer when nobody waits). On 
timeout the request fails with {{DEADLINE_EXCEEDED}}, as before, and the stream 
is cancelled and dropped.
* *Stream lifecycle.*
** Idle streams are half-closed after {{scm.container.client.idle.threshold}} 
by a shared reaper thread, and immediately when the channel leaves {{READY}} 
(e.g. the datanode sends GOAWAY on shutdown), so open streams do not hold up 
the datanode's graceful shutdown.
** Half-closed streams that the datanode never completes are cancelled.
** A stream is dropped, never reused, when it fails, is completed by the 
datanode, receives an unsolicited reply, or when its request is abandoned 
(timeout, interrupt, cancel).
* *Unchanged paths.* Traced blocking calls still open one stream per call 
(trace headers are per stream), but also complete on the waiting thread. The 
async path ({{sendCommandAsync(ContainerCommandRequestProto)}}) is unchanged.
* *Smaller fixes along the way.* If issuing a fan-out {{sendCommand}} to one of 
the datanodes fails, the requests already issued to the others are cancelled 
instead of being left in flight. {{sortDatanodes}} returns early for 
single-node pipelines.
* {{README.gRPC.md}} is updated to describe the new behavior.

h3. Compatibility
Client-only change. No protobuf, RPC or datanode changes; the datanode already 
handles several sequential requests on one stream.

h3. Testing
New {{TestXceiverClientGrpcCallerRuns}} (33 tests against an in-process gRPC 
server) covers:
* replies completing on the caller for {{get()}} and {{join()}}, and off the 
submitter when nobody waits;
* stream reuse for sequential calls, distinct streams for concurrent calls, no 
more streams than semaphore permits;
* deadline expiry, timed waits shorter than the request deadline, interrupted 
waiters, cancel from another thread, late waiters;
* stream errors, datanode-completed streams, unsolicited replies, cancel racing 
with the reply;
* idle-threshold reaping, half-closed streams never completed by the datanode, 
server shutdown not being held up;
* traced calls using a stream per call, dead channels failing fast without 
leaking permits, and fan-out cancellation.



> Deliver blocking XceiverClientGrpc replies on the waiting thread
> ----------------------------------------------------------------
>
>                 Key: HDDS-17561
>                 URL: https://issues.apache.org/jira/browse/HDDS-17561
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Datanode
>    Affects Versions: 2.2.1
>            Reporter: Sergey Soldatov
>            Assignee: Sergey Soldatov
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> A blocking {{XceiverClientGrpc#sendCommand}} (ReadChunk, GetBlock and the 
> other read-path commands) is built on top of the async path. For every 
> request it opens a new bi-directional gRPC stream, sends one request, 
> half-closes the stream, and waits on a {{CompletableFuture}}. The reply then 
> travels:
> # gRPC transport (Netty event loop) ->
> # gRPC's default executor, where the {{StreamObserver}} parses the response 
> and completes the future ->
> # the caller thread, woken up from {{get()}}.
> So each blocking request pays two thread hand-offs plus the setup and 
> teardown of an HTTP/2 stream on both the client and the datanode, even though 
> the caller is just sitting idle waiting for the reply.
> h3. Proposal
> * *Reply on the waiting thread.* New {{CallerRunsReplyFuture}}: a 
> {{CompletableFuture}} that is also the call's {{Executor}}. While a thread 
> waits in {{get()}}, the gRPC callbacks of the call are queued and run by that 
> thread, so the response is parsed and the future completed on the caller (the 
> same technique gRPC blocking stubs use). Once the waiter returns, queued and 
> later callbacks run directly. Only {{get()}} drives the callbacks, so 
> {{join()}} or a dependent stage alone never completes the future; 
> {{sendCommandWithRetry}} always calls {{get()}}.
> * *Pooled streams for untraced blocking calls.* A blocking call with no valid 
> span on the calling thread leases an idle stream of the channel, or opens 
> one, and sends exactly one request on it at a time. That is the default while 
> tracing is off, but an application's own span also counts, and those calls 
> keep using one stream per call. The datanode serves the requests of a stream 
> in order, so the next reply always belongs to the current request and no 
> request IDs are needed. After the reply the stream goes back to the pool. The 
> number of streams is bounded by the client semaphore (max outstanding 
> requests).
> * *Client-side deadline.* A pooled stream has no gRPC deadline, so the 
> request deadline is enforced by the waiter in {{get()}}. On timeout the 
> request fails with {{DEADLINE_EXCEEDED}}, as before, and the stream is 
> cancelled and dropped.
> * *Stream lifecycle.*
> ** Idle streams are half-closed after one minute by a shared daemon thread, 
> and immediately when the channel leaves {{READY}} (e.g. the datanode sends 
> GOAWAY on shutdown), so open streams do not hold up the datanode's graceful 
> shutdown.
> ** A stream is dropped, never reused, when it fails, is completed by the 
> datanode, receives an unsolicited reply, or when its request is abandoned 
> (timeout, interrupt). An abandon acts only while the request still owns the 
> stream, so it cannot cancel a stream that has gone on to another request.
> * *Configuration.* {{ozone.client.grpc.pooled.streams.enabled}} (default 
> true). When false, these calls open one stream per call with a gRPC deadline. 
> Replies are still delivered on the waiting thread.
> * *Unchanged paths.* Traced blocking calls still open one stream per call 
> (trace headers are per stream), but also complete on the waiting thread. 
> {{sendCommandOnAllNodes}} and the async path 
> ({{sendCommandAsync(ContainerCommandRequestProto)}}) also use one stream per 
> call and complete on gRPC's executor.
> * *Smaller fix along the way.* {{sortDatanodes}} returns early for 
> single-node pipelines.
> * {{README.gRPC.md}} is updated to describe the new behavior.
> h3. Compatibility
> Client-only change, with one new client configuration key. No protobuf, RPC 
> or datanode changes; the datanode already handles several sequential requests 
> on one stream.
> h3. Testing
> New {{TestXceiverClientGrpcCallerRuns}} (22 tests, most against a local Netty 
> gRPC server) covers:
> * replies completing on the caller for {{get()}};
> * stream reuse for sequential calls, distinct streams for concurrent calls, 
> no more streams than semaphore permits, and pooling turned off by 
> configuration;
> * deadline expiry and interrupted waiters, including an abandon that must not 
> cancel the next request on the stream;
> * stream errors, a stream that ends after it was leased, and unsolicited and 
> duplicate replies;
> * idle-timeout half-close and server shutdown not being held up;
> * traced calls using a stream per call, dead channels failing fast without 
> leaking permits, and fan-out delivery and permit release.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to