yashmayya commented on code in PR #19452:
URL: https://github.com/apache/pinot/pull/19452#discussion_r4364866499


##########
pinot-core/src/main/java/org/apache/pinot/core/transport/QueryRouter.java:
##########
@@ -176,6 +196,101 @@ public boolean connect(ServerInstance serverInstance) {
     }
   }
 
+  /// Pings the server over every channel queries to it use, and returns 
whether all of them answered within
+  /// `timeoutMs`. Returns `false` when there is no such channel, as there is 
nothing to ping over.
+  ///
+  /// A server that is alive answers straight away even when it is busy, 
because it replies from its network thread
+  /// instead of queueing behind queries (see `InstanceRequestHandler`). So a 
channel that stays silent means the server
+  /// cannot answer anything -- its node is gone, its network path is 
blackholed, or its JVM is frozen -- or that the
+  /// channel itself is half-open, left over from a server that went away. 
Either way queries sent over it would time
+  /// out, which is why every channel has to answer, not just one.
+  ///
+  /// Each channel that stays silent is closed so the next attempt over it 
connects afresh -- see
+  /// [ServerChannels#closeChannel]. Pings record no routing stats, so they 
never skew adaptive server selection.
+  public boolean ping(ServerInstance serverInstance, long timeoutMs) {
+    try {
+      List<ServerRoutingInstance> routingInstances = 
channelRoutingInstances(serverInstance);
+      if (routingInstances.isEmpty()) {
+        return false;
+      }
+      long requestId = _pingRequestIdGenerator.decrementAndGet();
+      InstanceRequest pingRequest = newPingRequest(requestId, _brokerId);
+      // Like a hybrid query, the pings share one request id and one response, 
which waits for every channel.
+      AsyncQueryResponse pingResponse =
+          new AsyncQueryResponse(this, requestId, new 
HashSet<>(routingInstances), System.currentTimeMillis(),
+              timeoutMs, _noOpRoutingStatsManager, false);
+      _asyncQueryResponseMap.put(requestId, pingResponse);
+      ServerChannels serverChannels = channels();
+      List<ServerRoutingInstance> pingedRoutingInstances = new 
ArrayList<>(routingInstances.size());
+      for (ServerRoutingInstance routingInstance : routingInstances) {
+        pingedRoutingInstances.add(routingInstance);
+        try {
+          serverChannels.sendRequest(PING_TABLE_NAME, pingResponse, 
routingInstance, pingRequest, timeoutMs);
+        } catch (Exception e) {
+          // Fails the whole ping; do not spend another connect timeout on the 
remaining channels.
+          pingResponse.markQueryFailed(routingInstance, e);
+          break;
+        }
+      }
+      // Removes the ping from _asyncQueryResponseMap on every outcome.
+      // A channel the ping never reached, because an earlier send failed, 
stays open. That failed send already makes
+      // the ping unanswered.
+      Map<ServerRoutingInstance, ServerResponse> responses = 
pingResponse.getFinalResponses();
+      boolean answered = true;
+      for (ServerRoutingInstance routingInstance : pingedRoutingInstances) {
+        if (responses.get(routingInstance).getDataTable() == null) {
+          answered = false;
+          serverChannels.closeChannel(routingInstance);

Review Comment:
   Fixed in 2a219daf20. Each channel now gets its own ping and its own 
response, so a failure on one channel no longer ends the wait for the others. 
Each ping's timeout also starts at its own send. Cleanup closes a channel only 
when its own ping stayed silent.
   
   Regression test: 
`QueryRoutingTest#testChannelFailingAtOnceDoesNotCloseAnotherStillDueToAnswer`. 
OFFLINE answers the ping after 300 ms, and REALTIME fails at once. The OFFLINE 
channel and its in-flight query survive.
   



##########
pinot-core/src/main/java/org/apache/pinot/core/transport/ServerChannels.java:
##########
@@ -169,6 +170,19 @@ public void connect(ServerRoutingInstance 
serverRoutingInstance)
     _serverToChannelMap.computeIfAbsent(serverRoutingInstance, 
ServerChannel::new).connect();
   }
 
+  /// Closes the channel to the server, if one is open, so that the next 
request to it connects afresh.
+  ///
+  /// The broker uses this after a server fails to answer a ping. The channel 
may be half-open -- nothing closes it
+  /// when the peer's node disappears without sending a FIN or an RST -- and 
requests written into it are never read.
+  /// Closing it also ends whatever is still in flight to that server right 
away (see [DataTableHandler]), rather than
+  /// leaving those queries to wait out their own timeouts against a server 
that is not going to answer.
+  void closeChannel(ServerRoutingInstance serverRoutingInstance) {
+    ServerChannel serverChannel = 
_serverToChannelMap.get(serverRoutingInstance);
+    if (serverChannel != null) {
+      serverChannel.closeChannel();

Review Comment:
   Fixed in 2a219daf20. `ServerChannels.sendRequest` now returns the connection 
the request was written to. A silent ping closes only that connection, not 
whichever one the channel holds by then. 
`ServerChannels.closeChannel(ServerRoutingInstance)` is gone, and so is the 
`volatile` I had added to `_channel`, since the ping no longer reads it.
   
   The write-failure listener in `sendRequestWithoutLocking` had the same race 
(it closed `_channel` when it ran), so it now closes the connection it wrote to 
as well.
   
   Regression tests: 
`QueryRoutingTest#testFailedPingClosesOnlyItsOwnConnection` (the ping's 
connection drops, then a query reconnects and sends over a fresh connection 
before the ping's cleanup; the fresh connection and its query survive), and 
`ServerChannelsTest#testWriteFailureClosesChannelAndFailsQuery` now replaces 
the connection before the failure listener runs.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to