yandrey321 commented on code in PR #11874:
URL: https://github.com/apache/ozone/pull/11874#discussion_r4507453236


##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {

Review Comment:
   [Major] no switch to turn pooling off; applies to writes too
   
   ```
   Every untraced blocking sendCommand now goes through a pooled stream. That 
includes write and admin commands
   (createContainer, closeContainer, putBlock/writeSmallFile when sent 
synchronously), not just ReadChunk/GetBlock.
   Correctness then depends on the datanode answering every command type with 
exactly one onNext and never ending the
   stream after a successful reply. That holds for GrpcXceiverService today, 
but nothing enforces it.
   
   Could we add a client config to fall back to one stream per call, e.g.
   `ozone.client.grpc.pooled.streams.enabled` (default true, or false for the 
first release)? Also consider limiting
   pooling to HddsUtils.isReadOnly(request) commands at first. That covers the 
measured hot path (ReadChunk/GetBlock)
   with a smaller blast radius.
   ```
   
   [Minor] pooling silently disabled by any current OTel span
   
   ```
   `!Span.current().getSpanContext().isValid()` is true only when no span 
context at all is current. With Ozone tracing
   disabled, TracingUtil's no-op tracer still propagates the parent's span 
context (PropagatedSpan), and with
   ozone tracing application-aware mode it adopts the application's 
GlobalOpenTelemetry. So any client that runs
   inside an application span uses one stream per call for every command, even 
with Ozone tracing off. Examples: an
   unshaded Ozone client under the OTel Java agent, or a service with its own 
spans.
   
   (The shaded ozone-filesystem jar relocates io.opentelemetry, so HBase's own 
spans don't affect it. That's how we saw
   pooling work in the HBase profile. Unshaded users would not get the win.)
   
   Options:
   - gate on whether *Ozone* exports a trace context (e.g. 
TracingUtil.isTracingEnabled(), or isSampled());
   - better, make the datanode's GrpcServerInterceptor prefer the per-message 
traceID, which the request already
     carries (XceiverClientGrpc sets it on every command and HddsDispatcher 
already uses msg.getTraceID()). Traced calls
     could then be pooled too, and this check could go away.
   
   Either way, a debug log or counter of pooled vs per-call sends would help 
operators see which path they're on.
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {
+      // No trace context to send, so a pooled stream carries the same headers 
as a new one would.
+      channelInfo.leaseStream().send(request, reply, (CallerRunsReplyFuture) 
replyFuture,
+          System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout));
+      return new XceiverClientReply(replyFuture);
+    }
+
+    XceiverClientProtocolServiceStub stub = 
channelInfo.getStub().withDeadlineAfter(timeout, TimeUnit.MILLISECONDS);
+    if (replyFuture instanceof Executor) {
+      stub = stub.withExecutor((Executor) replyFuture);
+    }
     // create a new grpc message stream pair for each call.
-    final StreamObserver<ContainerCommandRequestProto> requestObserver =
-        dnChannelInfoMap.get(dnId).getStub().withDeadlineAfter(timeout, 
TimeUnit.MILLISECONDS)
-            .send(new StreamObserver<ContainerCommandResponseProto>() {
-              @Override
-              public void onNext(ContainerCommandResponseProto value) {
-                replyFuture.complete(value);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onError(Throwable t) {
-                replyFuture.completeExceptionally(t);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onCompleted() {
-                if (!replyFuture.isDone()) {
-                  replyFuture.completeExceptionally(new IOException(
-                      "Stream completed but no reply for request " +
-                          processForDebug(request)));
-                }
-              }
-
-              private void decreasePendingMetricsAndReleaseSemaphore() {
-                metrics.decrPendingContainerOpsMetrics(request.getCmdType());
-                long cost = Time.monotonicNow() - requestTime;
-                metrics.addContainerOpsLatency(request.getCmdType(), cost);
-                if (LOG.isDebugEnabled()) {
-                  LOG.debug("Executed command {} on datanode {}, cost = {}, 
cmdType = {}",
-                      processForDebug(request), dn, cost, 
request.getCmdType());
-                }
-                semaphore.release();
-              }
-            });
+    final StreamObserver<ContainerCommandRequestProto> requestObserver = 
stub.send(reply);
     requestObserver.onNext(request);
     requestObserver.onCompleted();
     return new XceiverClientReply(replyFuture);
   }
 
+  /**
+   * Receives the reply to one request: completes its future, then releases
+   * its semaphore permit and updates the metrics.
+   */
+  @VisibleForTesting
+  final class ReplyObserver implements 
StreamObserver<ContainerCommandResponseProto> {
+    private final ContainerCommandRequestProto request;
+    private final DatanodeDetails dn;
+    private final CompletableFuture<ContainerCommandResponseProto> replyFuture;
+    private final long requestTime;
+
+    ReplyObserver(ContainerCommandRequestProto request, DatanodeDetails dn,
+        CompletableFuture<ContainerCommandResponseProto> replyFuture, long 
requestTime) {
+      this.request = request;
+      this.dn = dn;
+      this.replyFuture = replyFuture;
+      this.requestTime = requestTime;
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      replyFuture.complete(value);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      replyFuture.completeExceptionally(t);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onCompleted() {
+      if (!replyFuture.isDone()) {
+        replyFuture.completeExceptionally(new IOException(
+            "Stream completed but no reply for request " +
+                processForDebug(request)));
+      }
+    }
+
+    private void decreasePendingMetricsAndReleaseSemaphore() {
+      metrics.decrPendingContainerOpsMetrics(request.getCmdType());
+      long cost = Time.monotonicNow() - requestTime;
+      metrics.addContainerOpsLatency(request.getCmdType(), cost);
+      if (LOG.isDebugEnabled()) {
+        LOG.debug("Executed command {} on datanode {}, cost = {}, cmdType = 
{}",
+            processForDebug(request), dn, cost, request.getCmdType());
+      }
+      semaphore.release();
+    }
+  }
+
+  /**
+   * A send stream kept open across requests, so a request does not pay for
+   * opening and closing an HTTP/2 stream. It carries one request at a time:
+   * the datanode serves the requests of a stream in order and replies to each
+   * one, so the next reply always belongs to the current request. The gRPC
+   * callbacks run on the current request's {@link CallerRunsReplyFuture}, as
+   * for a stream opened per request.
+   */
+  @VisibleForTesting
+  static final class PooledStream implements Executor,
+      ClientResponseObserver<ContainerCommandRequestProto, 
ContainerCommandResponseProto> {
+    private final ChannelInfo owner;
+    private ClientCallStreamObserver<ContainerCommandRequestProto> 
requestStream;
+    private final AtomicReference<ReplyObserver> current = new 
AtomicReference<>();
+    private volatile CallerRunsReplyFuture currentFuture;
+    private volatile boolean closed;
+    private volatile long idleSinceNanos;
+
+    PooledStream(ChannelInfo owner) {
+      this.owner = owner;
+    }
+
+    @Override
+    public void 
beforeStart(ClientCallStreamObserver<ContainerCommandRequestProto> stream) {
+      requestStream = stream;
+    }
+
+    void send(ContainerCommandRequestProto request, ReplyObserver reply, 
CallerRunsReplyFuture future,
+        long deadlineNanos) {
+      future.setDeadline(deadlineNanos, this::cancel);
+      currentFuture = future;
+      current.set(reply);
+      if (closed && current.compareAndSet(reply, null)) {
+        // the stream ended after it was leased, before the request was 
attached, so its callbacks did not fail it
+        currentFuture = null;
+        reply.onError(Status.UNAVAILABLE.withDescription("pooled stream closed 
before the request was sent")
+            .asRuntimeException());
+        return;
+      }
+      requestStream.onNext(request);
+    }
+
+    @Override
+    public void execute(Runnable task) {
+      final CallerRunsReplyFuture future = current.get() != null ? 
currentFuture : null;
+      if (future != null) {
+        future.execute(task);
+      } else {
+        task.run();
+      }
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {

Review Comment:
   [Major] unsolicited reply re-pools the stream and NPEs
   
   ```
   PooledStream.onNext() assumes a current request:
   
       final ReplyObserver reply = current.getAndSet(null);
       currentFuture = null;
       owner.releaseStream(this);
       reply.onNext(value);
   
   If a message arrives while no request is current, `reply` is null. That 
would be a datanode bug, a future command
   type that answers with more than one message, or a stray message after a 
reply. Two things then go wrong:
   - releaseStream(this) has already pushed the stream into idleStreams. If the 
stream was already idle, it is now in
     the deque twice.
   - The NPE is thrown inside the gRPC callback, so gRPC cancels the call 
("Failed to read message") only some time later.
   
   In that window, two threads can lease the same PooledStream. The second 
current.set(reply) overwrites the first:
   - the first request's ReplyObserver is never called, so its semaphore permit 
is never released (a permanent leak);
   - the reply sent for the first request is delivered to the second caller. 
For ReadChunk that is data for the wrong
     block or offset. Its checksums are self-consistent, so the validators 
would not catch it.
   
   The Jira says streams are "dropped rather than reused after ... unsolicited 
replies". Suggest:
   
       final ReplyObserver reply = current.getAndSet(null);
       currentFuture = null;
       if (reply == null) {
         // a reply nobody asked for: the stream is out of step, never reuse it
         closed = true;
         requestStream.cancel("unsolicited reply", null);
         return;
       }
       owner.releaseStream(this);
       reply.onNext(value);
   
   Plus a test where the fake server sends two responses for one request: the 
stream must not be reused, and the
   available permits must be back to the starting value.
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -334,7 +347,8 @@ public ContainerCommandResponseProto sendCommand(
     for (DatanodeDetails dn : datanodeList) {
       try {
         request = reconstructRequestIfNeeded(request, dn);
-        futureHashMap.put(dn, sendCommandAsync(request, dn).getResponse());
+        // the caller waits for the replies one by one, so the requests to the 
other nodes must not wait for it
+        futureHashMap.put(dn, sendCommandAsync(request, dn, new 
CompletableFuture<>()).getResponse());

Review Comment:
   [Minor] the fan-out does not cancel issued requests (the Jira says it does)
   
   ```
   The Jira lists "Requests already sent in a fan-out are cancelled if one 
sendCommand fails". In sendCommandOnAllNodes,
   when a later node throws, the requests already issued keep running. Their 
permits come back when the replies arrive,
   which is what fanOutThatCannotReachANode...ReleasesThePermits checks, but 
nothing cancels them. Either cancel the
   issued futures' calls on failure, or drop that line from the description.
   ```
   



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {
+      // No trace context to send, so a pooled stream carries the same headers 
as a new one would.
+      channelInfo.leaseStream().send(request, reply, (CallerRunsReplyFuture) 
replyFuture,
+          System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout));
+      return new XceiverClientReply(replyFuture);
+    }
+
+    XceiverClientProtocolServiceStub stub = 
channelInfo.getStub().withDeadlineAfter(timeout, TimeUnit.MILLISECONDS);
+    if (replyFuture instanceof Executor) {
+      stub = stub.withExecutor((Executor) replyFuture);
+    }
     // create a new grpc message stream pair for each call.
-    final StreamObserver<ContainerCommandRequestProto> requestObserver =
-        dnChannelInfoMap.get(dnId).getStub().withDeadlineAfter(timeout, 
TimeUnit.MILLISECONDS)
-            .send(new StreamObserver<ContainerCommandResponseProto>() {
-              @Override
-              public void onNext(ContainerCommandResponseProto value) {
-                replyFuture.complete(value);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onError(Throwable t) {
-                replyFuture.completeExceptionally(t);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onCompleted() {
-                if (!replyFuture.isDone()) {
-                  replyFuture.completeExceptionally(new IOException(
-                      "Stream completed but no reply for request " +
-                          processForDebug(request)));
-                }
-              }
-
-              private void decreasePendingMetricsAndReleaseSemaphore() {
-                metrics.decrPendingContainerOpsMetrics(request.getCmdType());
-                long cost = Time.monotonicNow() - requestTime;
-                metrics.addContainerOpsLatency(request.getCmdType(), cost);
-                if (LOG.isDebugEnabled()) {
-                  LOG.debug("Executed command {} on datanode {}, cost = {}, 
cmdType = {}",
-                      processForDebug(request), dn, cost, 
request.getCmdType());
-                }
-                semaphore.release();
-              }
-            });
+    final StreamObserver<ContainerCommandRequestProto> requestObserver = 
stub.send(reply);
     requestObserver.onNext(request);
     requestObserver.onCompleted();
     return new XceiverClientReply(replyFuture);
   }
 
+  /**
+   * Receives the reply to one request: completes its future, then releases
+   * its semaphore permit and updates the metrics.
+   */
+  @VisibleForTesting
+  final class ReplyObserver implements 
StreamObserver<ContainerCommandResponseProto> {
+    private final ContainerCommandRequestProto request;
+    private final DatanodeDetails dn;
+    private final CompletableFuture<ContainerCommandResponseProto> replyFuture;
+    private final long requestTime;
+
+    ReplyObserver(ContainerCommandRequestProto request, DatanodeDetails dn,
+        CompletableFuture<ContainerCommandResponseProto> replyFuture, long 
requestTime) {
+      this.request = request;
+      this.dn = dn;
+      this.replyFuture = replyFuture;
+      this.requestTime = requestTime;
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      replyFuture.complete(value);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      replyFuture.completeExceptionally(t);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onCompleted() {
+      if (!replyFuture.isDone()) {
+        replyFuture.completeExceptionally(new IOException(
+            "Stream completed but no reply for request " +
+                processForDebug(request)));
+      }
+    }
+
+    private void decreasePendingMetricsAndReleaseSemaphore() {
+      metrics.decrPendingContainerOpsMetrics(request.getCmdType());
+      long cost = Time.monotonicNow() - requestTime;
+      metrics.addContainerOpsLatency(request.getCmdType(), cost);
+      if (LOG.isDebugEnabled()) {
+        LOG.debug("Executed command {} on datanode {}, cost = {}, cmdType = 
{}",
+            processForDebug(request), dn, cost, request.getCmdType());
+      }
+      semaphore.release();
+    }
+  }
+
+  /**
+   * A send stream kept open across requests, so a request does not pay for
+   * opening and closing an HTTP/2 stream. It carries one request at a time:
+   * the datanode serves the requests of a stream in order and replies to each
+   * one, so the next reply always belongs to the current request. The gRPC
+   * callbacks run on the current request's {@link CallerRunsReplyFuture}, as
+   * for a stream opened per request.
+   */
+  @VisibleForTesting
+  static final class PooledStream implements Executor,
+      ClientResponseObserver<ContainerCommandRequestProto, 
ContainerCommandResponseProto> {
+    private final ChannelInfo owner;
+    private ClientCallStreamObserver<ContainerCommandRequestProto> 
requestStream;
+    private final AtomicReference<ReplyObserver> current = new 
AtomicReference<>();
+    private volatile CallerRunsReplyFuture currentFuture;
+    private volatile boolean closed;
+    private volatile long idleSinceNanos;
+
+    PooledStream(ChannelInfo owner) {
+      this.owner = owner;
+    }
+
+    @Override
+    public void 
beforeStart(ClientCallStreamObserver<ContainerCommandRequestProto> stream) {
+      requestStream = stream;
+    }
+
+    void send(ContainerCommandRequestProto request, ReplyObserver reply, 
CallerRunsReplyFuture future,
+        long deadlineNanos) {
+      future.setDeadline(deadlineNanos, this::cancel);
+      currentFuture = future;
+      current.set(reply);
+      if (closed && current.compareAndSet(reply, null)) {
+        // the stream ended after it was leased, before the request was 
attached, so its callbacks did not fail it
+        currentFuture = null;
+        reply.onError(Status.UNAVAILABLE.withDescription("pooled stream closed 
before the request was sent")
+            .asRuntimeException());
+        return;
+      }
+      requestStream.onNext(request);
+    }
+
+    @Override
+    public void execute(Runnable task) {
+      final CallerRunsReplyFuture future = current.get() != null ? 
currentFuture : null;
+      if (future != null) {
+        future.execute(task);
+      } else {
+        task.run();
+      }
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      // back in the pool before the permit is released, so the thread woken 
by the permit finds the stream
+      owner.releaseStream(this);
+      reply.onNext(value);
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      closed = true;
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      if (reply != null) {
+        reply.onError(t);
+      }
+    }
+
+    @Override
+    public void onCompleted() {
+      closed = true;
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      if (reply != null) {
+        reply.onCompleted();
+      }
+    }
+
+    boolean isClosed() {
+      return closed;
+    }
+
+    void setIdleSince(long nanos) {
+      idleSinceNanos = nanos;
+    }
+
+    long getIdleSince() {
+      return idleSinceNanos;
+    }
+
+    /** Ends an idle stream, the datanode then completes it. */
+    void halfClose() {

Review Comment:
   [Minor] code differs from the Jira: idle threshold, half-closed streams
   
   ```
   The Jira says idle streams are half-closed after 
scm.container.client.idle.threshold, and that half-closed streams
   the datanode never completes get cancelled. The code doesn't match:
   
   - IDLE_STREAM_TIMEOUT_NANOS is a hard-coded 1 minute (only settable through 
the @VisibleForTesting setter). Either
     wire it to the config or update the Jira/README.
   - halfClose() sets `closed` and calls onCompleted(); nothing cancels the 
call if the datanode never completes it. A
     pooled call has no gRPC deadline, so such a call (and its HTTP/2 stream) 
stays open until the transport dies. It
     also keeps the connection "active", so the server's maxConnectionIdle 
never reaps it. GrpcXceiverService completes
     immediately today, so this is defensive. A follow-up cancel after a grace 
period, or simply cancel() instead of
     halfClose() for idle streams, would close the gap.
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {
+      // No trace context to send, so a pooled stream carries the same headers 
as a new one would.
+      channelInfo.leaseStream().send(request, reply, (CallerRunsReplyFuture) 
replyFuture,
+          System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout));
+      return new XceiverClientReply(replyFuture);
+    }
+
+    XceiverClientProtocolServiceStub stub = 
channelInfo.getStub().withDeadlineAfter(timeout, TimeUnit.MILLISECONDS);
+    if (replyFuture instanceof Executor) {
+      stub = stub.withExecutor((Executor) replyFuture);
+    }
     // create a new grpc message stream pair for each call.
-    final StreamObserver<ContainerCommandRequestProto> requestObserver =
-        dnChannelInfoMap.get(dnId).getStub().withDeadlineAfter(timeout, 
TimeUnit.MILLISECONDS)
-            .send(new StreamObserver<ContainerCommandResponseProto>() {
-              @Override
-              public void onNext(ContainerCommandResponseProto value) {
-                replyFuture.complete(value);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onError(Throwable t) {
-                replyFuture.completeExceptionally(t);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onCompleted() {
-                if (!replyFuture.isDone()) {
-                  replyFuture.completeExceptionally(new IOException(
-                      "Stream completed but no reply for request " +
-                          processForDebug(request)));
-                }
-              }
-
-              private void decreasePendingMetricsAndReleaseSemaphore() {
-                metrics.decrPendingContainerOpsMetrics(request.getCmdType());
-                long cost = Time.monotonicNow() - requestTime;
-                metrics.addContainerOpsLatency(request.getCmdType(), cost);
-                if (LOG.isDebugEnabled()) {
-                  LOG.debug("Executed command {} on datanode {}, cost = {}, 
cmdType = {}",
-                      processForDebug(request), dn, cost, 
request.getCmdType());
-                }
-                semaphore.release();
-              }
-            });
+    final StreamObserver<ContainerCommandRequestProto> requestObserver = 
stub.send(reply);
     requestObserver.onNext(request);
     requestObserver.onCompleted();
     return new XceiverClientReply(replyFuture);
   }
 
+  /**
+   * Receives the reply to one request: completes its future, then releases
+   * its semaphore permit and updates the metrics.
+   */
+  @VisibleForTesting
+  final class ReplyObserver implements 
StreamObserver<ContainerCommandResponseProto> {
+    private final ContainerCommandRequestProto request;
+    private final DatanodeDetails dn;
+    private final CompletableFuture<ContainerCommandResponseProto> replyFuture;
+    private final long requestTime;
+
+    ReplyObserver(ContainerCommandRequestProto request, DatanodeDetails dn,
+        CompletableFuture<ContainerCommandResponseProto> replyFuture, long 
requestTime) {
+      this.request = request;
+      this.dn = dn;
+      this.replyFuture = replyFuture;
+      this.requestTime = requestTime;
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      replyFuture.complete(value);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      replyFuture.completeExceptionally(t);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onCompleted() {
+      if (!replyFuture.isDone()) {
+        replyFuture.completeExceptionally(new IOException(
+            "Stream completed but no reply for request " +
+                processForDebug(request)));
+      }
+    }
+
+    private void decreasePendingMetricsAndReleaseSemaphore() {
+      metrics.decrPendingContainerOpsMetrics(request.getCmdType());
+      long cost = Time.monotonicNow() - requestTime;
+      metrics.addContainerOpsLatency(request.getCmdType(), cost);
+      if (LOG.isDebugEnabled()) {
+        LOG.debug("Executed command {} on datanode {}, cost = {}, cmdType = 
{}",
+            processForDebug(request), dn, cost, request.getCmdType());
+      }
+      semaphore.release();
+    }
+  }
+
+  /**
+   * A send stream kept open across requests, so a request does not pay for
+   * opening and closing an HTTP/2 stream. It carries one request at a time:
+   * the datanode serves the requests of a stream in order and replies to each
+   * one, so the next reply always belongs to the current request. The gRPC
+   * callbacks run on the current request's {@link CallerRunsReplyFuture}, as
+   * for a stream opened per request.
+   */
+  @VisibleForTesting
+  static final class PooledStream implements Executor,
+      ClientResponseObserver<ContainerCommandRequestProto, 
ContainerCommandResponseProto> {
+    private final ChannelInfo owner;
+    private ClientCallStreamObserver<ContainerCommandRequestProto> 
requestStream;
+    private final AtomicReference<ReplyObserver> current = new 
AtomicReference<>();
+    private volatile CallerRunsReplyFuture currentFuture;
+    private volatile boolean closed;
+    private volatile long idleSinceNanos;
+
+    PooledStream(ChannelInfo owner) {
+      this.owner = owner;
+    }
+
+    @Override
+    public void 
beforeStart(ClientCallStreamObserver<ContainerCommandRequestProto> stream) {
+      requestStream = stream;
+    }
+
+    void send(ContainerCommandRequestProto request, ReplyObserver reply, 
CallerRunsReplyFuture future,
+        long deadlineNanos) {
+      future.setDeadline(deadlineNanos, this::cancel);
+      currentFuture = future;
+      current.set(reply);
+      if (closed && current.compareAndSet(reply, null)) {
+        // the stream ended after it was leased, before the request was 
attached, so its callbacks did not fail it
+        currentFuture = null;
+        reply.onError(Status.UNAVAILABLE.withDescription("pooled stream closed 
before the request was sent")
+            .asRuntimeException());
+        return;
+      }
+      requestStream.onNext(request);
+    }
+
+    @Override
+    public void execute(Runnable task) {

Review Comment:
   [Minor] a stream's callbacks can run on another request's thread or on the 
event loop
   
   ```
   gRPC wraps the call executor in a SerializingExecutor. The Runnable that 
PooledStream.execute() receives is that
   executor's drain loop, which runs every task queued for the call while it is 
running. Two consequences:
   - A waiter that runs the drain can also process the next request's reply on 
the same stream. That includes
     deserializing a ReadChunk response of up to the chunk size, after its own 
future is already done. complete()
     wakes the other waiter correctly, but the first caller's get() returns 
later than it needs to.
   - When no request is current, the drain runs inline on the calling thread, 
i.e. the netty event loop (e.g. an
     onReady or onClose). If a new request is leased and answered meanwhile, 
its message parse can happen on the event
     loop and stall I/O for every stream on that connection.
   
   Both are unlikely, since one request per stream means the next reply is at 
least an RTT later. But it would be good
   to state them in the class comment, and the benchmark's p99 should show 
whether it matters. One way to bound it:
   when current == null, hand the task to gRPC's default executor instead of 
running it inline.
   ```



##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/TestXceiverClientGrpcCallerRuns.java:
##########
@@ -0,0 +1,585 @@
+/*

Review Comment:
   [Minor] tests to add
   
   ```
   Nice coverage of reply threading, reuse, deadline, interrupt, idle reaping 
and fan-out. Suggested additions for the
   cases above:
   - server sends two responses for one request: the stream is dropped, no 
permit leak, and the next request gets its
     own reply;
   - join() on the future from sendCommandAsync(request, dn): completes, or 
fails fast instead of hanging;
   - a write command (e.g. CreateContainer) and a datanode error response 
(result != SUCCESS) on a pooled stream: the
     stream is reused after an error *response*, and dropped after an onError;
   - a span from a non-Ozone OTel context is current: documents which path is 
taken.
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {
+      // No trace context to send, so a pooled stream carries the same headers 
as a new one would.
+      channelInfo.leaseStream().send(request, reply, (CallerRunsReplyFuture) 
replyFuture,
+          System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout));
+      return new XceiverClientReply(replyFuture);
+    }
+
+    XceiverClientProtocolServiceStub stub = 
channelInfo.getStub().withDeadlineAfter(timeout, TimeUnit.MILLISECONDS);
+    if (replyFuture instanceof Executor) {
+      stub = stub.withExecutor((Executor) replyFuture);
+    }
     // create a new grpc message stream pair for each call.
-    final StreamObserver<ContainerCommandRequestProto> requestObserver =
-        dnChannelInfoMap.get(dnId).getStub().withDeadlineAfter(timeout, 
TimeUnit.MILLISECONDS)
-            .send(new StreamObserver<ContainerCommandResponseProto>() {
-              @Override
-              public void onNext(ContainerCommandResponseProto value) {
-                replyFuture.complete(value);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onError(Throwable t) {
-                replyFuture.completeExceptionally(t);
-                decreasePendingMetricsAndReleaseSemaphore();
-              }
-
-              @Override
-              public void onCompleted() {
-                if (!replyFuture.isDone()) {
-                  replyFuture.completeExceptionally(new IOException(
-                      "Stream completed but no reply for request " +
-                          processForDebug(request)));
-                }
-              }
-
-              private void decreasePendingMetricsAndReleaseSemaphore() {
-                metrics.decrPendingContainerOpsMetrics(request.getCmdType());
-                long cost = Time.monotonicNow() - requestTime;
-                metrics.addContainerOpsLatency(request.getCmdType(), cost);
-                if (LOG.isDebugEnabled()) {
-                  LOG.debug("Executed command {} on datanode {}, cost = {}, 
cmdType = {}",
-                      processForDebug(request), dn, cost, 
request.getCmdType());
-                }
-                semaphore.release();
-              }
-            });
+    final StreamObserver<ContainerCommandRequestProto> requestObserver = 
stub.send(reply);
     requestObserver.onNext(request);
     requestObserver.onCompleted();
     return new XceiverClientReply(replyFuture);
   }
 
+  /**
+   * Receives the reply to one request: completes its future, then releases
+   * its semaphore permit and updates the metrics.
+   */
+  @VisibleForTesting
+  final class ReplyObserver implements 
StreamObserver<ContainerCommandResponseProto> {
+    private final ContainerCommandRequestProto request;
+    private final DatanodeDetails dn;
+    private final CompletableFuture<ContainerCommandResponseProto> replyFuture;
+    private final long requestTime;
+
+    ReplyObserver(ContainerCommandRequestProto request, DatanodeDetails dn,
+        CompletableFuture<ContainerCommandResponseProto> replyFuture, long 
requestTime) {
+      this.request = request;
+      this.dn = dn;
+      this.replyFuture = replyFuture;
+      this.requestTime = requestTime;
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      replyFuture.complete(value);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      replyFuture.completeExceptionally(t);
+      decreasePendingMetricsAndReleaseSemaphore();
+    }
+
+    @Override
+    public void onCompleted() {
+      if (!replyFuture.isDone()) {
+        replyFuture.completeExceptionally(new IOException(
+            "Stream completed but no reply for request " +
+                processForDebug(request)));
+      }
+    }
+
+    private void decreasePendingMetricsAndReleaseSemaphore() {
+      metrics.decrPendingContainerOpsMetrics(request.getCmdType());
+      long cost = Time.monotonicNow() - requestTime;
+      metrics.addContainerOpsLatency(request.getCmdType(), cost);
+      if (LOG.isDebugEnabled()) {
+        LOG.debug("Executed command {} on datanode {}, cost = {}, cmdType = 
{}",
+            processForDebug(request), dn, cost, request.getCmdType());
+      }
+      semaphore.release();
+    }
+  }
+
+  /**
+   * A send stream kept open across requests, so a request does not pay for
+   * opening and closing an HTTP/2 stream. It carries one request at a time:
+   * the datanode serves the requests of a stream in order and replies to each
+   * one, so the next reply always belongs to the current request. The gRPC
+   * callbacks run on the current request's {@link CallerRunsReplyFuture}, as
+   * for a stream opened per request.
+   */
+  @VisibleForTesting
+  static final class PooledStream implements Executor,
+      ClientResponseObserver<ContainerCommandRequestProto, 
ContainerCommandResponseProto> {
+    private final ChannelInfo owner;
+    private ClientCallStreamObserver<ContainerCommandRequestProto> 
requestStream;
+    private final AtomicReference<ReplyObserver> current = new 
AtomicReference<>();
+    private volatile CallerRunsReplyFuture currentFuture;
+    private volatile boolean closed;
+    private volatile long idleSinceNanos;
+
+    PooledStream(ChannelInfo owner) {
+      this.owner = owner;
+    }
+
+    @Override
+    public void 
beforeStart(ClientCallStreamObserver<ContainerCommandRequestProto> stream) {
+      requestStream = stream;
+    }
+
+    void send(ContainerCommandRequestProto request, ReplyObserver reply, 
CallerRunsReplyFuture future,
+        long deadlineNanos) {
+      future.setDeadline(deadlineNanos, this::cancel);
+      currentFuture = future;
+      current.set(reply);
+      if (closed && current.compareAndSet(reply, null)) {
+        // the stream ended after it was leased, before the request was 
attached, so its callbacks did not fail it
+        currentFuture = null;
+        reply.onError(Status.UNAVAILABLE.withDescription("pooled stream closed 
before the request was sent")
+            .asRuntimeException());
+        return;
+      }
+      requestStream.onNext(request);
+    }
+
+    @Override
+    public void execute(Runnable task) {
+      final CallerRunsReplyFuture future = current.get() != null ? 
currentFuture : null;
+      if (future != null) {
+        future.execute(task);
+      } else {
+        task.run();
+      }
+    }
+
+    @Override
+    public void onNext(ContainerCommandResponseProto value) {
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      // back in the pool before the permit is released, so the thread woken 
by the permit finds the stream
+      owner.releaseStream(this);
+      reply.onNext(value);
+    }
+
+    @Override
+    public void onError(Throwable t) {
+      closed = true;
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      if (reply != null) {
+        reply.onError(t);
+      }
+    }
+
+    @Override
+    public void onCompleted() {
+      closed = true;
+      final ReplyObserver reply = current.getAndSet(null);
+      currentFuture = null;
+      if (reply != null) {
+        reply.onCompleted();
+      }
+    }
+
+    boolean isClosed() {
+      return closed;
+    }
+
+    void setIdleSince(long nanos) {
+      idleSinceNanos = nanos;
+    }
+
+    long getIdleSince() {
+      return idleSinceNanos;
+    }
+
+    /** Ends an idle stream, the datanode then completes it. */
+    void halfClose() {
+      closed = true;
+      requestStream.onCompleted();
+    }
+
+    /** Called when the request timed out or its waiter gave up; the stream 
fails the request and is dropped. */
+    private void cancel() {
+      closed = true;
+      requestStream.cancel("request abandoned by the client", null);
+    }
+  }
+
+  /**
+   * Reply future that is also the gRPC call executor. The callbacks of the
+   * call are queued and run by the thread waiting in {@link #get()}, so the
+   * response is parsed and the future completed on the caller, without a
+   * hand-off through gRPC's shared executor (as gRPC blocking stubs do).
+   * Once the waiter returns, queued and later callbacks run directly.
+   */
+  @VisibleForTesting
+  static final class CallerRunsReplyFuture

Review Comment:
   [Major] future only completes inside get(); join() deadlocks
   
   ```
   For the pooled path, CallerRunsReplyFuture is both the reply future and the 
call executor. The call's callbacks are
   queued until a thread calls get()/get(timeout), and the request deadline is 
enforced only there.
   
   A caller that uses the future any other way never completes it:
   - join();
   - thenApply/whenComplete with the caller waiting elsewhere;
   - getNow/isDone polling;
   - or simply not waiting.
   
   In those cases there is no gRPC deadline as a fallback, so the semaphore 
permit and the pooled stream stay held for
   good. join() isn't overridden, so it parks forever while the reply sits in 
`tasks`.
   
   sendCommandWithRetry always calls get(), so production is fine today. But 
sendCommandAsync(request, dn) is public and
   returns XceiverClientReply.getResponse(), which looks like an ordinary 
CompletableFuture. Could we:
   (a) make it package-private (it is @VisibleForTesting anyway), and
   (b) override join() (and getNow/isDone, or document them) to run the queued 
tasks like get(), or fail fast?
   At minimum, add a Javadoc warning on CallerRunsReplyFuture.
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
     return !HddsUtils.isReadOnly(request);
   }
 
+  /**
+   * Sends the request to the given datanode for a caller that waits for the
+   * reply with {@link CompletableFuture#get()}: the response is delivered on
+   * the waiting thread, see {@link CallerRunsReplyFuture}.
+   */
   @VisibleForTesting
   public XceiverClientReply sendCommandAsync(
       ContainerCommandRequestProto request, DatanodeDetails dn)
       throws IOException, InterruptedException {
+    return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+  }
+
+  /**
+   * @param replyFuture completed with the reply; if it is an {@link Executor}
+   *        the call's callbacks run on it, otherwise on gRPC's default 
executor
+   */
+  private XceiverClientReply sendCommandAsync(
+      ContainerCommandRequestProto request, DatanodeDetails dn,
+      CompletableFuture<ContainerCommandResponseProto> replyFuture)
+      throws IOException, InterruptedException {
     checkOpen(dn);
     DatanodeID dnId = dn.getID();
     if (LOG.isDebugEnabled()) {
       LOG.debug("Send command {} to datanode {}",
           request.getCmdType(), dn);
     }
-    final CompletableFuture<ContainerCommandResponseProto> replyFuture =
-        new CompletableFuture<>();
     semaphore.acquire();
     long requestTime = Time.monotonicNow();
     metrics.incrPendingContainerOpsMetrics(request.getCmdType());
 
+    final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+    final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture, 
requestTime);
+    if (replyFuture instanceof CallerRunsReplyFuture && 
!Span.current().getSpanContext().isValid()) {
+      // No trace context to send, so a pooled stream carries the same headers 
as a new one would.
+      channelInfo.leaseStream().send(request, reply, (CallerRunsReplyFuture) 
replyFuture,
+          System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout));

Review Comment:
   [Nit] deadline overflow for very large timeouts
   
   ```
   `System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout)` overflows for 
very large timeouts. A typical case is
   Long.MAX_VALUE ms used to mean "no timeout": toNanos saturates at 
Long.MAX_VALUE, and adding nanoTime wraps negative.
   The request then fails with DEADLINE_EXCEEDED right away. 
withDeadlineAfter() on the per-call path handles large values. A saturating 
add, or
   treating timeout <= 0 / >= some cap as "no deadline", would keep the two 
paths consistent.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to