[
https://issues.apache.org/jira/browse/IMPALA-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-13164:
-----------------------------------
Description:
When Impala starts to build up a queue in an RPC service - usually
DataStreamService, but could also be ControlService - they become a bottleneck
that slows down all exchanges going through that node. To identify what's
happening we generally need to collect pstacks to identify why the queue is
backed up.
If all KrpcDataStreamMgr deserialization threads are in use, it can help to
increase the number of threads available via
{{datastream_service_num_deserialization_threads}}. This is the most common
bottleneck on larger machines; DataStream packets can be large and take time to
deserialize.
Less frequently the number of reactor threads in RpcMgr sending requests and
receiving responses (so handling the actual network I/O) might be full; this is
much less common because {{num_reactor_threads}} defaults to the number of CPU
cores. ControlService uses a single thread pool for network I/O and
deserialization because the packets tend to be much smaller; this is controlled
by {{control_service_num_svc_threads}}, which defaults to the number of CPU
cores as well.
It would help to have metrics for RPC reactor and deserialization threads to
tell when they're fully loaded and building up a queue; this could identify
when we need to increase {{datastream_service_num_deserialization_threads}} or
{{num_reactor_threads}}.
was:When Impala starts to build up a queue in an RPC service, we generally
need to start collecting pstacks to identify why the queue is backed up. It
would help to have metrics for RPC reactor and deserialization threads to tell
when they're fully loaded and building up a queue; this could identify when we
need to increase {{datastream_service_num_deserialization_threads}} or
{{num_reactor_threads}}.
> Add metrics for RPC reactor and deserialization threads
> -------------------------------------------------------
>
> Key: IMPALA-13164
> URL: https://issues.apache.org/jira/browse/IMPALA-13164
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Distributed Exec
> Affects Versions: Impala 4.4.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Critical
>
> When Impala starts to build up a queue in an RPC service - usually
> DataStreamService, but could also be ControlService - they become a
> bottleneck that slows down all exchanges going through that node. To identify
> what's happening we generally need to collect pstacks to identify why the
> queue is backed up.
> If all KrpcDataStreamMgr deserialization threads are in use, it can help to
> increase the number of threads available via
> {{datastream_service_num_deserialization_threads}}. This is the most common
> bottleneck on larger machines; DataStream packets can be large and take time
> to deserialize.
> Less frequently the number of reactor threads in RpcMgr sending requests and
> receiving responses (so handling the actual network I/O) might be full; this
> is much less common because {{num_reactor_threads}} defaults to the number of
> CPU cores. ControlService uses a single thread pool for network I/O and
> deserialization because the packets tend to be much smaller; this is
> controlled by {{control_service_num_svc_threads}}, which defaults to the
> number of CPU cores as well.
> It would help to have metrics for RPC reactor and deserialization threads to
> tell when they're fully loaded and building up a queue; this could identify
> when we need to increase {{datastream_service_num_deserialization_threads}}
> or {{num_reactor_threads}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
