[
https://issues.apache.org/jira/browse/PIG-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aniket Mokashi updated PIG-2784:
--------------------------------
Labels: GSOC2014 (was: )
> Framework for dynamic query optimization
> ----------------------------------------
>
> Key: PIG-2784
> URL: https://issues.apache.org/jira/browse/PIG-2784
> Project: Pig
> Issue Type: New Feature
> Reporter: Jie Li
> Assignee: Aniket Mokashi
> Labels: GSOC2014
>
> We need a framework to implement dynamic query optimization, i.e. changing
> the query plan at runtime. Currently we support estimating the number of
> reducers dynamically, which works well as the first step but was not
> perfectly implemented. In near future, we'll support more dynamic
> optimization, like [removing sample job for
> order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit
> job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting
> skew and using skew-join, etc.
> Currently estimating #reducer is implemented in JobControlCompiler after
> MRCompiler compiles all the MapReduceOperators and generate the complete
> MRPlan. One place (discussed with Thejas) to implement the framework is at
> the MRCompiler, where the MRPlan'll be generated at batches and adjusted
> dynamically.
> Any comment?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
