Jackie-Jiang commented on code in PR #19982:
URL: https://github.com/apache/pinot/pull/19982#discussion_r4623032817
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/QueryRouter.java:
##########
@@ -460,6 +460,22 @@ void receiveDataTable(ServerRoutingInstance
serverRoutingInstance, DataTable dat
}
}
+ // Unlike receiveDataTable, a deserialization failure cannot be tied to a
single requestId (the bytes didn't
+ // parse), so this method must broadcast to every in-flight query on the
server channel. This is consistent with
+ // markServerUnavailable and safe because the channel is closed immediately
after.
+ void markServerDeserializationError(ServerRoutingInstance
serverRoutingInstance, Exception exception,
+ int responseSize, int deserializationTimeMs) {
+ markServerDeserializationError(serverRoutingInstance, null, exception,
responseSize, deserializationTimeMs);
Review Comment:
Cleanup: require channel identity for this new failure path.
This no-channel overload has no callers, and passing null bypasses channel
isolation. The corresponding no-channel overload in `AsyncQueryResponse` is
only used by tests. Remove both shortcuts, make the channel required, and have
tests supply a channel. Channel isolation is an essential invariant here; the
new API can enforce it directly.
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java:
##########
@@ -214,8 +216,36 @@ void markRequestSent(ServerRoutingInstance
serverRoutingInstance, int requestSen
_responseMap.get(serverRoutingInstance).markRequestSent(requestSentLatencyMs);
}
+ // Handle server DataTable deserialization errors without waiting for query
timeout
+ void receiveDeserializationError(ServerRoutingInstance
serverRoutingInstance, Exception exception,
+ int responseSize, int deserializationTimeMs) {
+ receiveDeserializationError(serverRoutingInstance, null, exception,
responseSize, deserializationTimeMs);
+ }
+
+ void receiveDeserializationError(ServerRoutingInstance
serverRoutingInstance, @Nullable Channel channel,
+ Exception exception, int responseSize, int deserializationTimeMs) {
+ if (!shouldActOnServerDown(serverRoutingInstance, channel)) {
+ return;
+ }
+ if (!_countedDownServers.add(serverRoutingInstance)) {
+ return;
+ }
+ _failedServer = serverRoutingInstance;
+ DataTable dataTable = DataTableBuilderFactory.getEmptyDataTable();
+ dataTable.getMetadata().put(DataTable.MetadataKey.REQUEST_ID.getName(),
Long.toString(_requestId));
+ dataTable.addException(QueryErrorCode.DESERIALIZATION_ERROR.getId(),
+ exception.getMessage() != null ? exception.getMessage() :
exception.toString());
+ ServerResponse response = _responseMap.get(serverRoutingInstance);
+ response.receiveDeserializationError(dataTable, responseSize,
deserializationTimeMs);
+ _numServersResponded.getAndIncrement();
+ _countDownLatch.countDown();
Review Comment:
[P2] Reject synthetic deserialization failures in ping health checks.
Pings also use `AsyncQueryResponse` and are registered in
`_asyncQueryResponseMap`. A malformed response on the ping's channel now
installs a non-null synthetic DataTable and releases its latch, but
`QueryRouter.ping()` only checks whether `getDataTable()` is null. With
`enablePingOnTimeout=true`, `retryUnhealthyServer()` consequently returns
`HEALTHY` and can readmit the failing server; `checkServerNotResponded()`
likewise treats the probe as healthy.
Make ping validation reject this synthetic transport failure, while
preserving compatibility with decoded server-error responses, and add a
malformed-ping regression test.
More generally, the synthetic table makes `getDataTable() != null`
insufficient evidence that a real response arrived. Consider a clearly named
transport-failure predicate and audit consumers such as the materialized-view
path's `countSuccessfulDataTables()`. That path now counts synthetic errors as
successful tables and bypasses its missing-response fallback; the 435 is still
propagated, so this is a contract to clarify rather than a separate
silent-wrong-results finding.
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/QueryRouter.java:
##########
@@ -460,6 +460,22 @@ void receiveDataTable(ServerRoutingInstance
serverRoutingInstance, DataTable dat
}
}
+ // Unlike receiveDataTable, a deserialization failure cannot be tied to a
single requestId (the bytes didn't
+ // parse), so this method must broadcast to every in-flight query on the
server channel. This is consistent with
+ // markServerUnavailable and safe because the channel is closed immediately
after.
+ void markServerDeserializationError(ServerRoutingInstance
serverRoutingInstance, Exception exception,
+ int responseSize, int deserializationTimeMs) {
+ markServerDeserializationError(serverRoutingInstance, null, exception,
responseSize, deserializationTimeMs);
+ }
+
+ void markServerDeserializationError(ServerRoutingInstance
serverRoutingInstance, @Nullable Channel channel,
+ Exception exception, int responseSize, int deserializationTimeMs) {
+ for (AsyncQueryResponse asyncQueryResponse :
_asyncQueryResponseMap.values()) {
+ asyncQueryResponse.receiveDeserializationError(serverRoutingInstance,
channel, exception, responseSize,
+ deserializationTimeMs);
Review Comment:
Cleanup: avoid attributing one malformed frame's cost to every pending query.
The broadcast copies the same `responseSize` and `deserializationTimeMs`
into every affected query's synthetic response. Broker scatter then includes
those sizes in each query's total, so one frame can be charged N times. The
wire-byte meter in `DataTableHandler` already counts the frame once. Since its
request ID is unknown, use zero/unknown per-query attribution for synthetic
failures, or explicitly document that these fields represent shared failure
costs rather than bytes/work attributable to that query.
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/DataTableHandler.java:
##########
@@ -89,33 +89,45 @@ protected void channelRead0(ChannelHandlerContext ctx,
ByteBuf msg) {
_queryRouter.recordResponse(_serverRoutingInstance);
int responseSize = msg.readableBytes();
_brokerMetrics.addMeteredGlobalValue(BrokerMeter.NETTY_CONNECTION_BYTES_RECEIVED,
responseSize);
+ long deserializationStartTimeMs = System.currentTimeMillis();
+ ThreadResourceSnapshot resourceSnapshot = new ThreadResourceSnapshot();
+ DataTable dataTable;
try {
- long deserializationStartTimeMs = System.currentTimeMillis();
- ThreadResourceSnapshot resourceSnapshot = new ThreadResourceSnapshot();
- DataTable dataTable = DataTableFactory.getDataTable(msg.nioBuffer());
- _queryRouter.receiveDataTable(_serverRoutingInstance, dataTable,
responseSize,
- (int) (System.currentTimeMillis() - deserializationStartTimeMs));
- long cpuTimeNs = resourceSnapshot.getCpuTimeNs();
- long allocatedBytes = resourceSnapshot.getAllocatedBytes();
- // QUERY scope - keyed by query id
- Map<String, String> metadata = dataTable.getMetadata();
- String queryId = metadata.get(DataTable.MetadataKey.QUERY_ID.getName());
- // NOTE: Fallback to REQUEST_ID for backwards compatibility. When query
id is explicitly provided, and is
- // different from request id, this resource usage won't be tracked.
- if (queryId == null) {
- queryId = metadata.get(DataTable.MetadataKey.REQUEST_ID.getName());
- }
- _threadAccountant.updateUntrackedResourceUsage(queryId, cpuTimeNs,
allocatedBytes, TrackingScope.QUERY);
- // WORKLOAD scope - keyed by workload name
- // Use default workload name if not present
- String workloadName =
metadata.getOrDefault(DataTable.MetadataKey.WORKLOAD_NAME.getName(),
- CommonConstants.Accounting.DEFAULT_WORKLOAD_NAME);
- _threadAccountant.updateUntrackedResourceUsage(workloadName, cpuTimeNs,
allocatedBytes, TrackingScope.WORKLOAD);
+ dataTable = DataTableFactory.getDataTable(msg.nioBuffer());
} catch (Exception e) {
LOGGER.error("Caught exception while deserializing data table of size:
{} from server: {}", responseSize,
_serverRoutingInstance, e);
_brokerMetrics.addMeteredGlobalValue(BrokerMeter.DATA_TABLE_DESERIALIZATION_EXCEPTIONS,
1);
+ // Closing the channel is necessary because broadcasting the
deserialization error releases the latch slots of
+ // all in-flight queries on this server channel, preventing future
responses on this connection from interfering
+ // with those queries or subsequent requests.
Review Comment:
Cleanup: describe channel closure as a conservative recovery policy.
`LengthFieldBasedFrameDecoder` has already isolated this frame before
DataTable parsing. A deserialization exception does not itself establish broken
TCP framing, and the completion guard handles late responses for requests
already settled. Closing the connection after abandoning its outstanding
requests is defensible, but the comment and PR description should explain that
policy explicitly: the failed response cannot be associated with a request, so
pending requests on this channel are failed and the connection is retired.
Avoid claiming that deserialization necessarily corrupts framing or makes
later-response interference unavoidable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]