ss77892 commented on code in PR #11874:
URL: https://github.com/apache/ozone/pull/11874#discussion_r4514418096


##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {

Review Comment:
   added  switch `ozone.client.grpc.pooled.streams.enabled`, default true. When 
false, these calls open one stream per call with a gRPC deadline, like traced 
calls. Replies are still delivered on the waiting thread, so this turns off the 
pool rather than the whole change.
   
   I didn't add the read only limit. Reads and writes take the same path on the 
datanode (dispatcher.dispatch followed by one onNext in GrpcXceiverService), so 
the limit wouldn't reduce the risk, and the switch covers rollback.
   
   I kept span stuff as is, and documented in `README.gRPC.md`. The trace 
header is set once when a stream opens, and GrpcServerInterceptor uses it for 
every message on that stream. If we pooled with an application span current, 
the first caller's span would stick to all later requests on the stream. It's 
not a regression, since those calls behave as they did before this PR.
   
   Having the datanode prefer the per-message traceID is the right way to pool 
traced calls, but it's a datanode change, so I'd handle it in a follow-up Jira. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to