xiangfu0 commented on code in PR #20023:
URL: https://github.com/apache/pinot/pull/20023#discussion_r4546613801
##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -850,6 +850,17 @@ public static class Request {
public static class QueryOptionKey {
public static final String TIMEOUT_MS = "timeoutMs";
+ /// Broker-internal: the epoch time in milliseconds at which the
broker stops waiting for this request, i.e.
+ /// the time it set [#TIMEOUT_MS] plus the remaining time it will wait
(for a hybrid table the longer of the
+ /// two table types' remaining times). Set on every single-stage
server request; a user-supplied value is
+ /// overwritten. A request that reaches the server's Netty query
transport (`InstanceRequestHandler`) after
+ /// this time plus the server's clock-skew allowance
+ ///
([CommonConstants.Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS]) is answered
with a timeout error
+ /// instead of being executed: a request can sit for minutes in the
socket buffers of a stalled
+ /// broker-to-server connection, and once the connection recovers the
server would otherwise run it with a
+ /// full timeout budget measured from its arrival, long after the
broker gave up on it. Absent or non-positive
+ /// means no such check, which is what an old broker gets. The gRPC
and multi-stage paths do not check it.
+ public static final String REQUEST_DEADLINE_MS = "requestDeadlineMs";
Review Comment:
Done: the deadline is now an optional `InstanceRequest.deadlineMs` field (id
10, like `ping`), set in `QueryRouter#submitQuery` to the time the broker stops
waiting, so the `requestDeadlineMs` query option and the broker-handler
stamping are gone.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -141,6 +170,18 @@ protected void channelRead0(ChannelHandlerContext ctx,
ByteBuf msg) {
sendPingResponse(ctx, instanceRequest.getRequestId());
return;
}
+ try {
+ if (rejectIfExpired(instanceRequest, ctx, queryArrivalTimeMs)) {
+ return;
+ }
+ } catch (Exception e) {
Review Comment:
Done: with the deadline a Thrift `i64` there is nothing left to parse, so
the try/catch and its error path are gone; a request without the field runs as
before.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -183,6 +224,74 @@ private void sendPingResponse(ChannelHandlerContext ctx,
long requestId) {
}
}
+ /// Answers a request that reached this server after the broker's deadline
for it with a timeout error, without
+ /// executing it, and returns `true`; returns `false` when the request is to
be executed as usual.
+ ///
+ /// The broker sends its deadline as the `requestDeadlineMs` query option,
so a request from an old broker (no
+ /// option) is always executed. The deadline is compared against this
server's clock, which is why it has to be more
+ /// than [Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS] in the past
before the request is dropped. Without this
+ /// check a request that spent minutes in the socket buffers of a stalled
connection, and is delivered when the
+ /// connection recovers, is executed with a full timeout budget measured
from its arrival, for a broker that gave up
+ /// on it long ago. The check reads the option straight from the
deserialized request, before the query is compiled,
+ /// so that such a backlog costs no more than its deserialization on the
event loop. The response is still sent: it
+ /// costs a few bytes and tells the broker side that the channel is alive,
even though the broker no longer has a
+ /// query waiting for it. A malformed option propagates to the caller, which
answers with the parse error under the
+ /// right request id.
+ private boolean rejectIfExpired(InstanceRequest instanceRequest,
ChannelHandlerContext ctx,
+ long queryArrivalTimeMs) {
+ if (_requestDeadlineClockSkewMs < 0) {
+ return false;
+ }
+ Map<String, String> queryOptions = getQueryOptions(instanceRequest);
+ if (queryOptions == null) {
+ return false;
+ }
+ Long requestDeadlineMs =
QueryOptionsUtils.getRequestDeadlineMs(queryOptions);
+ if (requestDeadlineMs == null) {
+ return false;
+ }
+ // Subtract rather than add, so that an absurdly far deadline still runs
instead of overflowing into a rejection.
+ long lateByMs = queryArrivalTimeMs - requestDeadlineMs;
+ if (lateByMs <= _requestDeadlineClockSkewMs) {
+ return false;
+ }
+ long requestId = instanceRequest.getRequestId();
+ String tableNameWithType = getTableNameWithType(instanceRequest);
+
_serverMetrics.addMeteredGlobalValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL, 1,
_queriesExpiredOnArrivalMeter);
+ logExpiredRequest(instanceRequest, tableNameWithType, queryArrivalTimeMs,
requestDeadlineMs, lateByMs);
+ try {
+ DataTable dataTable = DataTableBuilderFactory.getEmptyDataTable();
+ dataTable.getMetadata().put(MetadataKey.REQUEST_ID.getName(),
Long.toString(requestId));
+ dataTable.addException(QueryErrorCode.QUERY_SCHEDULING_TIMEOUT,
+ String.format("%s request reached %s at %d, %dms after the broker
deadline %d, and was not executed",
+ EXPIRED_ON_ARRIVAL_MESSAGE_PREFIX, _instanceName,
queryArrivalTimeMs, lateByMs, requestDeadlineMs));
+ sendResponse(ctx, requestId, tableNameWithType, queryArrivalTimeMs,
dataTable.toDataBuffer());
+ } catch (Exception e) {
+ LOGGER.error("Exception while sending the expired response for request:
{}", requestId, e);
+ }
+ return true;
+ }
+
+ /// Logs at most one line per [#EXPIRED_REQUEST_LOG_INTERVAL_MS] of
wall-clock time, carrying the number of expired
+ /// requests since the previous line and the details of the latest one. The
handler is shared by every channel,
+ /// hence the atomics. The count is approximate: an increment that races the
winning CAS lands in this line or the
+ /// next, never nowhere; the meter is exact.
+ private void logExpiredRequest(InstanceRequest instanceRequest, @Nullable
String tableNameWithType,
Review Comment:
Done: dropped the rate-limited WARN and its atomics; the meter carries the
count and each rejection is logged at debug.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -106,6 +126,15 @@ public InstanceRequestHandler(String instanceName,
PinotConfiguration config, Qu
} else {
_executionContexts = null;
}
+ _requestDeadlineClockSkewMs =
config.getProperty(Server.CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS,
+ Server.DEFAULT_REQUEST_DEADLINE_CLOCK_SKEW_MS);
+ if (_requestDeadlineClockSkewMs >= 0) {
+ LOGGER.info("Requests arriving more than {}ms after the broker deadline
are not executed",
+ _requestDeadlineClockSkewMs);
+ } else {
+ LOGGER.info("Requests arriving after the broker deadline are executed
(check disabled)");
+ }
+ _queriesExpiredOnArrivalMeter =
_serverMetrics.getMeteredValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL);
Review Comment:
Done: no cached meter any more, just
`addMeteredGlobalValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL, 1)`.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1839,6 +1850,24 @@ public enum SortedSelectionMergeMode {
public static final String DEFAULT_SERVER_QUERY_REGEX_CLASS = "JAVA_UTIL";
public static final String CONFIG_OF_ENABLE_QUERY_CANCELLATION =
"pinot.server.enable.query.cancellation";
public static final boolean DEFAULT_ENABLE_QUERY_CANCELLATION = true;
+ /// Allowance for clock skew between a broker and this server when judging
whether a request arrived after the
+ /// broker's deadline for it
([Broker.Request.QueryOptionKey#REQUEST_DEADLINE_MS]). Such a request is
answered
+ /// with a timeout error instead of being executed, but only when it
arrives more than this many milliseconds
+ /// after the deadline. The broker stamps the deadline a few statements
before it starts its own wait for the
+ /// servers, and network or channel-lock delays come after that wait has
started, so with synchronized clocks a
+ /// request is rejected only once the broker has stopped waiting for it
for longer than this allowance; neither a
+ /// slow hop nor a short `timeoutMs` changes that, since the query's
timeout is not part of the comparison. The one
+ /// exposure is a server clock running ahead of the broker's by more than
this allowance: it rejects requests that
+ /// arrive within the excess of their deadline, i.e. every request of a
query whose remaining timeout is shorter
+ /// than the excess. A server clock behind the broker's only makes the
check more lenient. The default, 10
+ /// seconds, is far above the skew between hosts that synchronize their
clocks and far below the minutes a backlog
+ /// on a stalled connection can be old. Rejections show up as
`QuerySchedulingTimeoutError` responses (error code
+ /// 240, the code of a scheduler queue timeout as well) whose message
starts with `Expired on arrival:` and
+ /// carries the arrival time and the deadline, and in the server-wide
`QUERIES_EXPIRED_ON_ARRIVAL` meter. A
+ /// negative value disables the check.
+ public static final String CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS =
Review Comment:
Done: the clock-skew config Javadoc is two lines now (the meter's too),
`REQUEST_DEADLINE_MS` went away with the query option, and the reasoning is in
the PR description.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/ServerChannels.java:
##########
@@ -329,6 +329,19 @@ public UnwritableChannelException(String message) {
}
}
+ /// The channel to one server, with the lock that serializes the writes into
it.
+ ///
+ /// A request is handed to Netty with `writeAndFlush`. Once the flush has
run on the event loop, which happens
+ /// right after the write task, the message sits in the channel's outbound
buffer until the kernel accepts it, and
+ /// Netty offers no way to take a single message back out of that buffer:
`ChannelOutboundBuffer` only discards an
+ /// entry whose promise was cancelled before the flush marked it
uncancellable. So when the connection stalls, the
+ /// requests queued on it are delivered as soon as it recovers, including
those whose query has long timed out at
+ /// the broker. Pulling them back here is not possible; what keeps them
harmless is the server, which answers a
+ /// request that arrives after the broker's deadline (the
`requestDeadlineMs` query option) with a timeout error
+ /// instead of executing it. On the broker side, the request write timeout
([RequestWriteTimeoutHandler]) and the
+ /// write-buffer high watermark
([NettyConfig#WRITE_BUFFER_HIGH_WATER_MARK_BYTES]) bound how much can queue up
on a
+ /// stalled channel, and closing the channel, which that write timeout does,
is the only way to drop what is
+ /// already queued.
@ThreadSafe
class ServerChannel {
Review Comment:
Done: removed the `ServerChannels` comment; the PR description keeps the
explanation.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -183,6 +224,74 @@ private void sendPingResponse(ChannelHandlerContext ctx,
long requestId) {
}
}
+ /// Answers a request that reached this server after the broker's deadline
for it with a timeout error, without
+ /// executing it, and returns `true`; returns `false` when the request is to
be executed as usual.
+ ///
+ /// The broker sends its deadline as the `requestDeadlineMs` query option,
so a request from an old broker (no
+ /// option) is always executed. The deadline is compared against this
server's clock, which is why it has to be more
+ /// than [Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS] in the past
before the request is dropped. Without this
+ /// check a request that spent minutes in the socket buffers of a stalled
connection, and is delivered when the
+ /// connection recovers, is executed with a full timeout budget measured
from its arrival, for a broker that gave up
+ /// on it long ago. The check reads the option straight from the
deserialized request, before the query is compiled,
+ /// so that such a backlog costs no more than its deserialization on the
event loop. The response is still sent: it
+ /// costs a few bytes and tells the broker side that the channel is alive,
even though the broker no longer has a
+ /// query waiting for it. A malformed option propagates to the caller, which
answers with the parse error under the
+ /// right request id.
+ private boolean rejectIfExpired(InstanceRequest instanceRequest,
ChannelHandlerContext ctx,
+ long queryArrivalTimeMs) {
+ if (_requestDeadlineClockSkewMs < 0) {
+ return false;
+ }
+ Map<String, String> queryOptions = getQueryOptions(instanceRequest);
+ if (queryOptions == null) {
+ return false;
+ }
+ Long requestDeadlineMs =
QueryOptionsUtils.getRequestDeadlineMs(queryOptions);
+ if (requestDeadlineMs == null) {
+ return false;
+ }
+ // Subtract rather than add, so that an absurdly far deadline still runs
instead of overflowing into a rejection.
+ long lateByMs = queryArrivalTimeMs - requestDeadlineMs;
+ if (lateByMs <= _requestDeadlineClockSkewMs) {
Review Comment:
Done: the server now caps the execution deadline at `min(arrival +
timeoutMs, deadlineMs + allowance)` via `ServerQueryRequest#setMaxDeadlineMs`,
so a request arriving inside the allowance only gets what is left of it; a
negative allowance turns off both the check and the cap.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]