[
https://issues.apache.org/jira/browse/IMPALA-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18073520#comment-18073520
]
ASF subversion and git services commented on IMPALA-13164:
----------------------------------------------------------
Commit 49ddba704f1eec308d276f9568e7564918677258 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=49ddba704 ]
IMPALA-13164: Expose more KRPC metrics
Exposes Kudu RPC metrics that are already collected through the rpcz
JSON page: reactor_active_latency, reactor_load_percent, and
rpcs_timed_out_in_queue.
Reactor activity and load can help identify when reactor threads are a
bottleneck on RPC performance. Timed out RPCs could happen after
IMPALA-13130; a non-zero value would signify lost EndDataStream messages
reflecting significant problems in RPC flow in the environment.
Adds deserialization-queue-size and deserialization-queue-wait-time to
datastream-manager metrics to help identify when RPC deserialization is
slow and could benefit from increasing
datastream_service_num_deserialization_threads or the queue size.
Implements FunctionGauge to provide access to a thread-safe getter
providing the relevant metric.
Generated-by: Github Copilot (Claude Sonnet 4.6, GPT-5.3-Codex)
Change-Id: I62106bbab2b9290f0d5f456d4bb30005a5261524
Reviewed-on: http://gerrit.cloudera.org:8080/24179
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Add metrics for RPC reactor and deserialization threads
> -------------------------------------------------------
>
> Key: IMPALA-13164
> URL: https://issues.apache.org/jira/browse/IMPALA-13164
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Distributed Exec
> Affects Versions: Impala 4.4.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Critical
> Fix For: Impala 5.0.0
>
>
> When Impala starts to build up a queue in an RPC service - usually
> DataStreamService - they become a bottleneck that slows down all exchanges
> going through that node. To identify what's happening we generally need to
> collect pstacks to identify why the queue is backed up.
> If all KrpcDataStreamMgr deserialization threads are in use, it can help to
> increase the number of threads available via
> {{{}datastream_service_num_deserialization_threads{}}}. This is the most
> common bottleneck on larger machines; DataStream packets can be large and
> take time to deserialize.
> Less frequently the number of reactor threads in RpcMgr sending requests and
> receiving responses (so handling the actual network I/O) might be full; this
> is much less common because {{num_reactor_threads}} defaults to the number of
> CPU cores.
> ControlService uses a single thread pool for network I/O and deserialization
> because the packets tend to be much smaller; this is controlled by
> {{{}control_service_num_svc_threads{}}}, which defaults to the number of CPU
> cores as well.
> It would help to have metrics for RPC reactor and deserialization threads to
> tell when they're fully loaded and building up a queue; this could identify
> when we need to increase {{datastream_service_num_deserialization_threads}}
> or {{{}num_reactor_threads{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
