[
https://issues.apache.org/jira/browse/IMPALA-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472883#comment-16472883
]
ASF subversion and git services commented on IMPALA-6957:
---------------------------------------------------------
Commit e12ee485cf4c77203b144c053ee167509cc39374 in impala's branch
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=e12ee48 ]
IMPALA-6957: calc thread resource requirement in planner
This only factors in fragment execution threads. E.g. this does *not*
try to account for the number of threads on the old Thrift RPC
code path if that is enabled.
This is loosely related to the old VCores estimate, but is different in
that it:
* Directly ties into the notion of required threads in
ThreadResourceMgr.
* Is a strict upper bound on the number of such threads, rather than
an estimate.
Does not include "optional" threads. ThreadResourceMgr in the backend
bounds the number of "optional" threads per impalad, so the number of
execution threads on a backend is limited by
sum(required threads per query) +
CpuInfo::num_cores() * FLAGS_num_threads_per_core
DCHECKS in the backend enforce that the calculation is correct. They
were actually hit in KuduScanNode because of some races in thread
management leading to multiple "required" threads running. Now the
first thread in the multithreaded scans never exits, which means
that it's always safe for any of the other threads to exit early,
which simplifies the logic a lot.
Testing:
Updated planner tests.
Ran core tests.
Change-Id: I982837ef883457fa4d2adc3bdbdc727353469140
Reviewed-on: http://gerrit.cloudera.org:8080/10256
Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Include number of required threads in explain plan
> --------------------------------------------------
>
> Key: IMPALA-6957
> URL: https://issues.apache.org/jira/browse/IMPALA-6957
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Not Applicable
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
>
> Impala has an internal notion of "required threads" to execute a fragment,
> e.g. the fragment execution thread and the first scanner thread. It's
> possible to compute the number of required threads per fragment instance
> based on the plan.
> We should include this in the resource profile and expose it in the explain
> plan. This could then be a step toward implementing something like
> IMPALA-6035.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
