Jie Li created PIG-2784:
---------------------------
Summary: Framework for dynamic query optimization
Key: PIG-2784
URL: https://issues.apache.org/jira/browse/PIG-2784
Project: Pig
Issue Type: New Feature
Reporter: Jie Li
We need a framework to implement dynamic query optimization, i.e. changing the
query plan at runtime. Currently we support estimating the number of reducers
dynamically, which works well as the first step but was not perfectly
implemented. In near future, we'll support more dynamic optimization, like
[removing sample job for
order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit
job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting skew
and using skew-join, etc.
Currently estimating #reducer is implemented in JobControlCompiler after
MRCompiler compiles all the MapReduceOperators and generate the complete
MRPlan. One place (discussed with Thejas) to implement the framework is at the
MRCompiler, where the MRPlan'll be generated at batches and adjusted
dynamically.
Any comment?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
