Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- Review request for hive. Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/#review34599 --- ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java https://reviews.apache.org/r/18172/#comment64759 indentation needs fixing (2 spaces) ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java https://reviews.apache.org/r/18172/#comment64758 The coding conventions followed by hive require braces with if statements. As hive follows Sun/Java code conventions (except for indentation of 2 chars, and line limit of 100 chars), you can select the java code convention under eclipse formatter and select the java profile, edit it for these two settings and save it as hive profile. Then highlight your section of new/edited code, right-click source-format . I will add these instructions to HowToContribute once the wiki is working again. ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java https://reviews.apache.org/r/18172/#comment64760 indentation issues ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java https://reviews.apache.org/r/18172/#comment64761 braces needed for if - Thejas Nair On Feb. 16, 2014, 8:39 p.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 16, 2014, 8:39 p.m.) Review request for hive. Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/#review34602 --- conf/hive-default.xml.template https://reviews.apache.org/r/18172/#comment64764 max joins is missing. ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java https://reviews.apache.org/r/18172/#comment64765 if this is meant to be a generic contract this shouldn't be in the optiq package. ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java https://reviews.apache.org/r/18172/#comment64763 Why can't we use a generic type here? - Gunther Hagleitner On Feb. 16, 2014, 8:39 p.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 16, 2014, 8:39 p.m.) Review request for hive. Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/#review34603 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/18172/#comment64766 this shouldn't be part of the contract should it? - Gunther Hagleitner On Feb. 16, 2014, 8:39 p.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 16, 2014, 8:39 p.m.) Review request for hive. Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 17, 2014, 6:49 a.m.) Review request for hive. Changes --- John, I agree with Gunther's points, I propose: - we take out the max joins parameter from HiveConf for now; at least until it becomes clearer that this is one of the ways we want to control CBO use. - we move CostBasedOptimizer to the ql.optimizer package. Do you agree? Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/#review34624 --- 1. In most query engines 20 way joins and beyond is hard to reorder. So i would imagine that we would need a similar flag(hive.cbo.max.joins.supported) to control the length of join graph that is being considered for reordering. 2. Its reasonable to introduce ql.optimizer.CostBasedOptimizer which then calls in to Optiq based Optimizer. One thing to keep in mind, Optiq based optimizer would have both rule based and cost based portions. - John Pullokkaran On Feb. 17, 2014, 6:49 a.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 17, 2014, 6:49 a.m.) Review request for hive. Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
Re: Review Request 18172: HIVE-6439 Introduce CBO step in Semantic Analyzer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18172/ --- (Updated Feb. 17, 2014, 7:35 a.m.) Review request for hive. Changes --- ok added max.joins to conf; moved CBO to ql.optimizer Bugs: HIVE-6439 https://issues.apache.org/jira/browse/HIVE-6439 Repository: hive-git Description --- This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: CBO step is controlled by the 'hive.enable.cbo.flag'. When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 conf/hive-default.xml.template 0d08aa2 ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 1ba5654 ql/src/java/org/apache/hadoop/hive/ql/optimizer/CostBasedOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/PreCBOOptimizer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 52c39c0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77388dd Diff: https://reviews.apache.org/r/18172/diff/ Testing --- Thanks, Harish Butani
