[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909040#comment-13909040 ] Ashutosh Chauhan commented on HIVE-6439: Is plan to get CBO optimizer work in 0.13 timeframe? If no, I think we may want to delay this till branching 0.13 Otherwise, this will go in 0.13 release with a config which doesn't do anything and thus will be confusing for end users. Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch, HIVE-6439.5.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909075#comment-13909075 ] Harish Butani commented on HIVE-6439: - yes agreed, let's add this post hive 0.13 branching. Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch, HIVE-6439.5.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[
https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904073#comment-13904073
]
Hive QA commented on HIVE-6439:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629471/HIVE-6439.5.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1379/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1379/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1379/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1569331.
At revision 1569331.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12629471
Introduce CBO step in Semantic Analyzer
---
Key: HIVE-6439
URL: https://issues.apache.org/jira/browse/HIVE-6439
Project: Hive
Issue Type: Sub-task
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch,
HIVE-6439.5.patch
This patch introduces CBO step in SemanticAnalyzer. For now the
CostBasedOptimizer is an empty shell.
The contract between SemAly and CBO is:
- CBO step is controlled by the 'hive.enable.cbo.flag'.
- When true Hive SemAly will hand CBO a Hive Operator tree (with operators
annotated with stats). If it can CBO will return a better plan in Hive AST
form.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904970#comment-13904970 ] Thejas M Nair commented on HIVE-6439: - +1 LGTM Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch, HIVE-6439.5.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[
https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13903367#comment-13903367
]
Hive QA commented on HIVE-6439:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629326/HIVE-6439.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5127 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1362/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1362/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12629326
Introduce CBO step in Semantic Analyzer
---
Key: HIVE-6439
URL: https://issues.apache.org/jira/browse/HIVE-6439
Project: Hive
Issue Type: Sub-task
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch
This patch introduces CBO step in SemanticAnalyzer. For now the
CostBasedOptimizer is an empty shell.
The contract between SemAly and CBO is:
- CBO step is controlled by the 'hive.enable.cbo.flag'.
- When true Hive SemAly will hand CBO a Hive Operator tree (with operators
annotated with stats). If it can CBO will return a better plan in Hive AST
form.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13903762#comment-13903762 ] Harish Butani commented on HIVE-6439: - AS requested by John: - raise default for 'hive.cbo.max.joins.supported' to 10 - fix name in hive-default.xml Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch, HIVE-6439.4.patch, HIVE-6439.5.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902828#comment-13902828 ] Harish Butani commented on HIVE-6439: - Review at https://reviews.apache.org/r/18172/ Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[
https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902860#comment-13902860
]
Hive QA commented on HIVE-6439:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629280/HIVE-6439.2.patch
{color:green}SUCCESS:{color} +1 5120 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1348/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1348/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12629280
Introduce CBO step in Semantic Analyzer
---
Key: HIVE-6439
URL: https://issues.apache.org/jira/browse/HIVE-6439
Project: Hive
Issue Type: Sub-task
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch
This patch introduces CBO step in SemanticAnalyzer. For now the
CostBasedOptimizer is an empty shell.
The contract between SemAly and CBO is:
- CBO step is controlled by the 'hive.enable.cbo.flag'.
- When true Hive SemAly will hand CBO a Hive Operator tree (with operators
annotated with stats). If it can CBO will return a better plan in Hive AST
form.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902903#comment-13902903 ] Brock Noland commented on HIVE-6439: Hi Laljo, Essentially that is correct. The problem with that code is that it catches Throwable and logs at debug. This means nasty errors such as OOM or internal JVM errors which occur and will not be logged since most users do not log at the debug level in production. I'd suggest not catching throwable and instead catching exception. Catching exception is still a poor coding practice as you will catch all kinds of runtime errors which you do not expect. I am guessing the author is looking to try CBO and fail back if a bug is hit. Even in that case, catching exception, we should be logging at a ERROR level unless there is a very good reason not to. Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch, HIVE-6439.2.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902553#comment-13902553 ] Thejas M Nair commented on HIVE-6439: - Can you please create a reviewboard link ? Can you also add the config parameters to hive-default.xml.template and give a description there ? Indentation seems to be off at some places, can you also please fix that ? Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[
https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902557#comment-13902557
]
Brock Noland commented on HIVE-6439:
1) There are many tabs and spacing issues in this patch
2) We are catching throwable and logging at debug. We should *never* do this.
{noformat}
} catch (Throwable t) {
LOG.debug(CBO failed, skipping CBO, t);
{noformat}
Introduce CBO step in Semantic Analyzer
---
Key: HIVE-6439
URL: https://issues.apache.org/jira/browse/HIVE-6439
Project: Hive
Issue Type: Sub-task
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6439.1.patch
This patch introduces CBO step in SemanticAnalyzer. For now the
CostBasedOptimizer is an empty shell.
The contract between SemAly and CBO is:
- CBO step is controlled by the 'hive.enable.cbo.flag'.
- When true Hive SemAly will hand CBO a Hive Operator tree (with operators
annotated with stats). If it can CBO will return a better plan in Hive AST
form.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6439) Introduce CBO step in Semantic Analyzer
[ https://issues.apache.org/jira/browse/HIVE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902603#comment-13902603 ] Laljo John Pullokkaran commented on HIVE-6439: -- @Brock #2 Is the concern that we are swallowing the exception or is it that the log level should be info or warning instead of debug? Introduce CBO step in Semantic Analyzer --- Key: HIVE-6439 URL: https://issues.apache.org/jira/browse/HIVE-6439 Project: Hive Issue Type: Sub-task Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6439.1.patch This patch introduces CBO step in SemanticAnalyzer. For now the CostBasedOptimizer is an empty shell. The contract between SemAly and CBO is: - CBO step is controlled by the 'hive.enable.cbo.flag'. - When true Hive SemAly will hand CBO a Hive Operator tree (with operators annotated with stats). If it can CBO will return a better plan in Hive AST form. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
