[
https://issues.apache.org/jira/browse/IMPALA-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147736#comment-17147736
]
Daniel Becker commented on IMPALA-5444:
---------------------------------------
[https://gerrit.cloudera.org/#/c/15105/]
> Asynchronous code generation
> ----------------------------
>
> Key: IMPALA-5444
> URL: https://issues.apache.org/jira/browse/IMPALA-5444
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Michael Ho
> Assignee: Daniel Becker
> Priority: Minor
> Labels: codegen
>
> Currently, codegen happens during the preparation phase of a query fragment.
> In other words, the query fragment cannot start running until the code
> generation is complete. There are queries in which the code generation time
> is taking a huge amount of time. While we should disable codegen in some exec
> nodes if we can accurately estimate in the planner that running without
> codegen will be better off (e.g. number of rows to process is relatively
> small), we will still pay the price if say the stats is stale or the
> estimation is off.
> With async codegen, the idea is that we should run the code generation in a
> separate thread so that codegen is not on the critical path of the query
> execution. Once codegen completes for a fragment, we can atomically swap the
> function pointers of compiled functions embedded in the exec nodes. The exec
> nodes all currently support falling back to interpretation if the codegend
> functions don't exist anyway (i.e. the pointer to the compiled function is
> NULL). In some cases, it can occur that the query may run to completion
> before codegen completes. Once IMPALA-3259 is fixed (if feasible), we should
> be able to cancel the codegen execution.
> Another thing to note is that we should be able to bound the codegen work to
> a set of threads in thread pool so as to control the CPU and memory resources
> consumed by codegen.
> Another potential extension of this decoupling is IMPALA-9660.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
