[
https://issues.apache.org/jira/browse/PIG-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-2784:
----------------------------
Description:
We need a framework to implement dynamic query optimization, i.e. changing the
query plan at runtime. Currently we support estimating the number of reducers
dynamically, which works well as the first step but was not perfectly
implemented. In near future, we'll support more dynamic optimization, like
[removing sample job for
order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit
job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting skew
and using skew-join, etc.
Currently estimating #reducer is implemented in JobControlCompiler after
MRCompiler compiles all the MapReduceOperators and generate the complete
MRPlan. One place (discussed with Thejas) to implement the framework is at the
MRCompiler, where the MRPlan'll be generated at batches and adjusted
dynamically.
Any comment?
This is a candidate project for Google summer of code 2014. More information
about the program can be found at
https://cwiki.apache.org/confluence/display/PIG/GSoc2014
was:
We need a framework to implement dynamic query optimization, i.e. changing the
query plan at runtime. Currently we support estimating the number of reducers
dynamically, which works well as the first step but was not perfectly
implemented. In near future, we'll support more dynamic optimization, like
[removing sample job for
order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit
job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting skew
and using skew-join, etc.
Currently estimating #reducer is implemented in JobControlCompiler after
MRCompiler compiles all the MapReduceOperators and generate the complete
MRPlan. One place (discussed with Thejas) to implement the framework is at the
MRCompiler, where the MRPlan'll be generated at batches and adjusted
dynamically.
Any comment?
> Framework for dynamic query optimization
> ----------------------------------------
>
> Key: PIG-2784
> URL: https://issues.apache.org/jira/browse/PIG-2784
> Project: Pig
> Issue Type: New Feature
> Reporter: Jie Li
> Assignee: Aniket Mokashi
> Labels: GSOC2014
>
> We need a framework to implement dynamic query optimization, i.e. changing
> the query plan at runtime. Currently we support estimating the number of
> reducers dynamically, which works well as the first step but was not
> perfectly implemented. In near future, we'll support more dynamic
> optimization, like [removing sample job for
> order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit
> job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting
> skew and using skew-join, etc.
> Currently estimating #reducer is implemented in JobControlCompiler after
> MRCompiler compiles all the MapReduceOperators and generate the complete
> MRPlan. One place (discussed with Thejas) to implement the framework is at
> the MRCompiler, where the MRPlan'll be generated at batches and adjusted
> dynamically.
> Any comment?
> This is a candidate project for Google summer of code 2014. More information
> about the program can be found at
> https://cwiki.apache.org/confluence/display/PIG/GSoc2014
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
