[
https://issues.apache.org/jira/browse/IMPALA-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074975#comment-18074975
]
Joe McDonnell edited comment on IMPALA-14847 at 4/29/26 11:11 PM:
------------------------------------------------------------------
-Took a quick look at this. For ExecQueryFInstances, there is a shared sidecar
that it sent to all executors plus an executor specific payload. The shared
sidecar is easy to log, because we serialize it ourselves. The per-executor
payload is serialized inside KRPC code (Proxy::AsyncRequest()'s call to
RequestPayload::CreateRequestPayload()), so Impala doesn't see the final
serialized size. One option is to serialize the payload ourselves and pass it
as a sidecar. This would let us see the size for logging. It also gives us the
option of compressing it.-
Update: There are three pieces. There is a shared TQueryCtx sidecar that is
sent to all executors. There is a TExecPlanFragmentInfo that is specific to
this executor. There are also fields on the ExecQueryFInstancesRequestPB as
well (including the scan ranges). We have the first two sizes from serializing
them. For the last one, protobuf messages have a ByteSizeLong() method that can
tell us the size of the message. So, this is doable without changing any of our
structures.
was (Author: joemcdonnell):
Took a quick look at this. For ExecQueryFInstances, there is a shared sidecar
that it sent to all executors plus an executor specific payload. The shared
sidecar is easy to log, because we serialize it ourselves. The per-executor
payload is serialized inside KRPC code (Proxy::AsyncRequest()'s call to
RequestPayload::CreateRequestPayload()), so Impala doesn't see the final
serialized size. One option is to serialize the payload ourselves and pass it
as a sidecar. This would let us see the size for logging. It also gives us the
option of compressing it.
> Add warning logs for slow or large ExecQueryFInstances RPCs
> -----------------------------------------------------------
>
> Key: IMPALA-14847
> URL: https://issues.apache.org/jira/browse/IMPALA-14847
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Quanlong Huang
> Priority: Critical
>
> Similar to IMPALA-10702, it'd be helpful to log warnings for slow or large
> ExecQueryFInstances RPCs from coordinators. The request could be large since
> it contains the file paths and partition paths. Logging the warnings help to
> find the problematic queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
