ss77892 commented on code in PR #11874:
URL: https://github.com/apache/ozone/pull/11874#discussion_r4514418096
##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java:
##########
@@ -815,63 +771,337 @@ protected boolean shouldBlockAndWaitAsyncReply(
return !HddsUtils.isReadOnly(request);
}
+ /**
+ * Sends the request to the given datanode for a caller that waits for the
+ * reply with {@link CompletableFuture#get()}: the response is delivered on
+ * the waiting thread, see {@link CallerRunsReplyFuture}.
+ */
@VisibleForTesting
public XceiverClientReply sendCommandAsync(
ContainerCommandRequestProto request, DatanodeDetails dn)
throws IOException, InterruptedException {
+ return sendCommandAsync(request, dn, new CallerRunsReplyFuture());
+ }
+
+ /**
+ * @param replyFuture completed with the reply; if it is an {@link Executor}
+ * the call's callbacks run on it, otherwise on gRPC's default
executor
+ */
+ private XceiverClientReply sendCommandAsync(
+ ContainerCommandRequestProto request, DatanodeDetails dn,
+ CompletableFuture<ContainerCommandResponseProto> replyFuture)
+ throws IOException, InterruptedException {
checkOpen(dn);
DatanodeID dnId = dn.getID();
if (LOG.isDebugEnabled()) {
LOG.debug("Send command {} to datanode {}",
request.getCmdType(), dn);
}
- final CompletableFuture<ContainerCommandResponseProto> replyFuture =
- new CompletableFuture<>();
semaphore.acquire();
long requestTime = Time.monotonicNow();
metrics.incrPendingContainerOpsMetrics(request.getCmdType());
+ final ChannelInfo channelInfo = dnChannelInfoMap.get(dnId);
+ final ReplyObserver reply = new ReplyObserver(request, dn, replyFuture,
requestTime);
+ if (replyFuture instanceof CallerRunsReplyFuture &&
!Span.current().getSpanContext().isValid()) {
Review Comment:
added switch `ozone.client.grpc.pooled.streams.enabled`, default true. When
false, these calls open one stream per call with a gRPC deadline, like traced
calls. Replies are still delivered on the waiting thread, so this turns off the
pool rather than the whole change.
I didn't add the read only limit. Reads and writes take the same path on the
datanode (dispatcher.dispatch followed by one onNext in GrpcXceiverService), so
the limit wouldn't reduce the risk, and the switch covers rollback.
I kept span stuff as is, and documented in `README.gRPC.md`. The trace
header is set once when a stream opens, and GrpcServerInterceptor uses it for
every message on that stream. If we pooled with an application span current,
the first caller's span would stick to all later requests on the stream. It's
not a regression, since those calls behave as they did before this PR.
Having the datanode prefer the per-message traceID is the right way to pool
traced calls, but it's a datanode change, so I'd handle it in a follow-up Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]