[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234992#comment-14234992 ] Xuefu Zhang commented on HIVE-5771: --- This optimization might have caused the problem in HIVE-9025. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, > HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090265#comment-14090265 ] Lefty Leverenz commented on HIVE-5771: -- Okay, thanks Hari. In Configuration Properties the description of *hive.optimize.constant.propagation* can include that Wikipedia link. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, > HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090221#comment-14090221 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5771: - [~leftylev] constant propagation is not vectorization specific. IMO, it is a general concept used in many commercial databases and even in compilers : http://en.wikipedia.org/wiki/Constant_folding > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, > HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075542#comment-14075542 ] Lefty Leverenz commented on HIVE-5771: -- This adds configuration parameter *hive.optimize.constant.propagation* so it needs to be documented in the wiki for the 0.14.0 release. * [Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] Questions: Should general concepts about the constant propagation optimizer be documented, or is it sufficient to have the *hive.optimize.constant.propagation* doc point back to this JIRA ticket? If additional doc is needed, should it go in the Vectorization design doc or elsewhere? * [Design Doc: Vectorized Query Execution | https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution] > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, > HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075313#comment-14075313
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12657932/HIVE-5771.17.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5785 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_cluster_tasklog_retrieval
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/64/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/64/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-64/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12657932
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch,
> HIVE-5771.16.patch, HIVE-5771.17.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch,
> HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074833#comment-14074833 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5771: - The restriction related to vectorization in ConstantPropagate::transform() can be lifted once HIVE-7514 is committed. Vectorization will benefit from this fix, so it is better to make this change. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.16.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, > HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, > HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074132#comment-14074132
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12657732/HIVE-5771.16.patch
{color:red}ERROR:{color} -1 due to 100 failed/errored test(s), 5760 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_file_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_set_processor_namespaces
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_set_variable_sub
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2
org.apache.hadoop.hive.cli.TestCliDriver.testCl
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073620#comment-14073620
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12657603/HIVE-5771.14.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/45/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/45/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-45/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-45/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'conf/hive-default.xml.template'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
hwi/target common/target common/src/gen contrib/target service/target
serde/target beeline/target odbc/target cli/target
ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1613281.
At revision 1613281.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12657603
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch,
> HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch,
> HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch,
> HIVE-5771.patch, HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelera
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073573#comment-14073573 ] Ashutosh Chauhan commented on HIVE-5771: Good work, Ted! +1 > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.14.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047377#comment-14047377
]
Ashutosh Chauhan commented on HIVE-5771:
Yeah, turned out HIVE-7232 was unrelated. Sorry for the confusion.
Yes, extra {{is null}} predicate is introduced by query rewrite involving exist
in clause. You can found more info about this rewrite in design doc uploaded on
HIVE-784
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch,
> HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047086#comment-14047086
]
Ted Xu commented on HIVE-5771:
--
Hi [~ashutoshc], thanks for looking into that issue. I don't have enough
context on HIVE-7232, but the issue is still there after HIVE-7232 is patched.
I looked into subquery_views.q, it seems there is an extra filter predicate
which breaks the query. Notice the following hive.log segment:
{code}
2014-06-29 01:20:00,662 INFO ppd.OpProcFactory
(OpProcFactory.java:process(209)) - Processing for FIL(37)
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(601)) - Pushdown Predicates of FIL For Alias :
sq_1_notin_nullcheck
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(604)) - (_col0 = 0)
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:process(549)) - Processing for SEL(36)
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(601)) - Pushdown Predicates of SEL For Alias :
sq_1_notin_nullcheck
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(604)) - (_col0 = 0)
2014-06-29 01:20:00,663 INFO ppd.OpProcFactory
(OpProcFactory.java:process(549)) - Processing for GBY(35)
2014-06-29 01:20:00,664 INFO ppd.OpProcFactory
(OpProcFactory.java:process(549)) - Processing for RS(34)
2014-06-29 01:20:00,666 INFO ppd.OpProcFactory
(OpProcFactory.java:process(549)) - Processing for GBY(33)
2014-06-29 01:20:00,667 INFO ppd.OpProcFactory
(OpProcFactory.java:process(549)) - Processing for SEL(32)
2014-06-29 01:20:00,667 INFO ppd.OpProcFactory
(OpProcFactory.java:process(209)) - Processing for FIL(31)
2014-06-29 01:20:00,667 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(601)) - Pushdown Predicates of FIL For Alias : sq_1
2014-06-29 01:20:00,667 INFO ppd.OpProcFactory
(OpProcFactory.java:logExpr(604)) - ((_col0 is null or _col1 is null) or
_col2 is null)
{code}
FIL37 has a constant predicate (_col0 = 0), which is supposed to be predicated
to following operators, which breaks FIL31. The query don't contain such
predicate, I'm not sure if it is introduced by exist in clause.
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch,
> HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031827#comment-14031827
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650333/HIVE-5771.12.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 5615 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/470/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/470/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-470/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650333
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.12.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch,
> HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030841#comment-14030841
]
Ashutosh Chauhan commented on HIVE-5771:
Test subquery_in.q failed with exception:
{code}
java.lang.Exception: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to
deserialize reduce input key from x1x128x0x0x1 with properties
{columns=reducesinkkey0,reducesinkkey1,
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=++, columns.types=int,int}
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to
deserialize reduce input key from x1x128x0x0x1 with properties
{columns=reducesinkkey0,reducesinkkey1,
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=++, columns.types=int,int}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error: Unable to deserialize reduce input key from x1x128x0x0x1 with properties
{columns=reducesinkkey0,reducesinkkey1,
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=++, columns.types=int,int}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:222)
... 9 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:191)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:220)
... 9 more
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:201)
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:187)
... 10 more
{code}
subquery_views.q is failing with following exception
{code}
java.lang.Exception: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to
deserialize reduce input key from with properties {columns=reducesinkkey0,
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=+, columns.types=string}
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to
deserialize reduce input key from with properties {columns=reducesinkkey0,
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=+, columns.types=string}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error: Unable to deseriali
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028761#comment-14028761
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12649760/HIVE-5771.11.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 5539 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_second
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union27
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/441/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/441/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12649760
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch,
> HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch,
> HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028582#comment-14028582 ] Ashutosh Chauhan commented on HIVE-5771: Looked further into it. * Optimization HIVE_MAP_GROUPBY_SORT is no longer being effective , thus plans for groupby_sort_1.q,groupby_sort_skew_1.q are changed. * Queries which have not in where clause like select * from T where c1 not in (select c2 from T) queries are resulting in wrong results. So, results of subquery_multiinsert.q,subquery_notin.q looks incorrect. * Changes for annotate_stats_filter.q looks correct since after optimization of where !false we get optimization of select * from T which results in FetchTask only plan. * Warning for cross-product for join cases is fine and can be ignored. * smb_mapjoin_25.q needs to be looked into. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.11.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, > HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, > HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028377#comment-14028377
]
Ashutosh Chauhan commented on HIVE-5771:
[~tedxu] Can you create a Review Board request for your latest patch. I took a
cursorily look and have following observations:
* In few tests an extra (or in some cases 2) MR stages got added in the plan.
These tests were testing specific optimizations, so seems like those
optimizations got disabled now. Tests are :
groupby_sort_1.q,groupby_sort_skew_1.q
* Tests subquery_multiinsert.q,subquery_notin.q are generating in wrong results
* For test annotate_stats_filter.q plan changed from MR to fetch-only, which
seems like an improvement. But, not sure how plan got changed.
* Some join tests now print a warning about being getting converted into
cross-join, which will be performance degradation.
cluster.q,join38.q,join_literals.q,join_nullsafe.q,ppd2.q,ppd_clusterby.q,ppd_join4.q,ppd_outer_join5.q
* Test smb_mapjoin_25.q is failing with following stack trace:
{code}
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.getValueObjectInspectors(MapJoinOperator.java:135)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:167)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:310)
at
org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:72)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator.initializeOp(HashTableDummyOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:145)
{code}
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch,
> HIVE-5771.11.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch,
> HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch,
> HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014799#comment-14014799 ] Ashutosh Chauhan commented on HIVE-5771: [~tedxu] I think for now we can disable this optimization if hive.optimize.index.filter is on. We can take that on in a follow on jira. Are there any other bugs you are running into? Also, you may want to checkout HIVE-4867 changes getting made there may be useful for this patch. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008381#comment-14008381 ] Ashutosh Chauhan commented on HIVE-5771: No worries. Let me know, if I can be of any help. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008364#comment-14008364 ] Ted Xu commented on HIVE-5771: -- Hi [~ashutoshc], I may need more time to fix some bugs. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003663#comment-14003663 ] Ashutosh Chauhan commented on HIVE-5771: Also, I noticed constant folding didnt happen for constprog1.q and constprog_type.q in latest patch, whereas it use to happen in earlier patches. Is that intentional with latest patch? > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003546#comment-14003546 ] Ted Xu commented on HIVE-5771: -- Sure. Thanks [~ashutoshc] and [~rusanu]. As for the orc_ppd_date.q issue, it seems the current optimizer conflict with "hive.optimize.index.filter". I'm disabling constant propagation if hive.optimize.index.filter=true. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002702#comment-14002702 ] Ashutosh Chauhan commented on HIVE-5771: [~tedxu] Lets see where we stand with your latest patch. Would you like to upload a full patch and have Hive QA go at it? > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001397#comment-14001397 ] Ashutosh Chauhan commented on HIVE-5771: Thanks [~rusanu] for helping out on this one! > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001395#comment-14001395 ] Remus Rusanu commented on HIVE-5771: I have opened a separate issue to track the failure for vectorization_parquet, HIVE-7082 and I uploaded a patch. I think these changes in HIVE-5771 just exposed the problem so I though it warrants its own tracking. I've tested the HIVE-7082.1.patch both on trunk and on HIVE-5771.10.patch and it fixes the problem. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001349#comment-14001349 ] Remus Rusanu commented on HIVE-5771: I'm looking at the Parquet failure. Looks like the first row returned by parquet reader has a different number of fields than expected. Truth is that the building of the vectorizatized batch based on the first Parquet returned object was a hack to work around HIVE-6414. Now that it's fixed, perhaps I should also fix the hack and properly build the factorized batch out of the object inspectors, not out of the first row. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, > HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, > HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, > HIVE-5771.patch, HIVE-5771.patch.javaonly > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000667#comment-14000667
]
Ashutosh Chauhan commented on HIVE-5771:
Thanks, [~tedxu] for update. I tried your latest patch.
cluster.q,ppd_clusterby.q are now fixed with your latest patch. But,
orc_ppd_date.q & vectorized_parquet.q still failed with following stack traces
respectively.
{code}
java.lang.Exception: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 11 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 19 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
... 24 more
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
java.sql.Date
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveWritableObject(JavaDateObjectInspector.java:37)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveWritableObject(JavaDateObjectInspector.java:28)
at
org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:67)
at
org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:36)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:44)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:57)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at org.apache.hadoop.hive.ql.exec.Operator.initiali
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999738#comment-13999738
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645158/HIVE-5771.9.patch
{color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 5528 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/211/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/211/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exite
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977566#comment-13977566
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5771:
-
Hi [~tedxu] I have a minor comment on the implementation.
Is there a reason for these rules sharing same val.
+opRules.put(new RuleRegExp("R5", "UNION%"), ConstantPropagateProcFactory
+.getUnionProc());
+opRules.put(new RuleRegExp("R5", "RS%"), ConstantPropagateProcFactory
+.getReduceSinkProc());
Thanks
Hari
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977563#comment-13977563 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5771: - [~ashutoshc] Created HIVE-6959 to remove vectorization related constant expression folding code once HIVE-5771 is committed. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977120#comment-13977120 ] Ashutosh Chauhan commented on HIVE-5771: Cool. As you can see from last test run on .8 patch, only handful of failures are remaining which needs to be looked into it. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976410#comment-13976410 ] Ted Xu commented on HIVE-5771: -- Hi [~ashutoshc], thanks for the patch, I will look into this. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, > HIVE-5771.8.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975038#comment-13975038
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12640969/HIVE-5771.8.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 5410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnectionSchemaAPIs
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/14/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/14/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12640969
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch,
> HIVE-5771.8.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13974682#comment-13974682
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12640915/HIVE-5771.7.patch
{color:red}ERROR:{color} -1 due to 106 failed/errored test(s), 5410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876529#comment-13876529 ] Ted Xu commented on HIVE-5771: -- Hi [~ashutoshgupt...@gmail.com], Your points are valid, thanks! Here is my thinking of those issues: * smb_mapjoin_18.q & smb_mapjoin_25.q: those problems are introduced by constant propagate optimizer (CPO) conflicting with *Bucketing Sorting ReduceSink Optimizer (BSRO)*. I tried apply BSRO before CPO and the issue seems fixed. * groupby_sort_1.q & groupby_sort_skew_1.q: those are because of CPO conflicting with *Groupby Optimizer (GO)*, apply it before CPO also fixes issue. In fact I'm wondering if it is safe to reorder those optimizers, making it GO->BSRO->CPO. * decimal.q & pcr.q: I disabled these two cases because of some issue I still not figured out. My local machine told me to patch a piece of output data like '0.0040' to '0,004', but it is still '0.0040' in hudson server. I guess it is an environment issue. I will update the patch as soon as I validated the above fixes. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875991#comment-13875991 ] Ashutosh Chauhan commented on HIVE-5771: Pretty good work, Ted. Hive is in need of this optimization for long time. Thanks for taking it up. I scanned the patch. Mostly looking at .q.out changes. Most of them look are correct, except following : * smb_mapjoin_18.q : Seems like a Map only job has turned into MR job. * smb_mapjoin_25.q : extra MR stage got introduced groupby_sort_1.q --> extra MR stage got introduced groupby_sort_skew_1.q --> extra MR stage got introduced udf_between.q --> betweeen 2 and '3' got optimized away. Here types don't match, shouldn't this instead have optimized into always false filter? decimal.q - optimization is turned off. Any particular reason? pcr.q - optimization is turned off. Any particular reason? I haven't looked at code changes yet. Will be looking at those soon. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875669#comment-13875669 ] Ted Xu commented on HIVE-5771: -- Eric, I've updated the diff file on https://reviews.apache.org/r/15561/. Thanks. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875488#comment-13875488 ] Eric Hanson commented on HIVE-5771: --- Ted, can you please post the latest patch to Review Board? > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873304#comment-13873304
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12623285/HIVE-5771.6.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4931 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/930/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/930/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12623285
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867338#comment-13867338 ] Eric Hanson commented on HIVE-5771: --- For vectorization_div0, there is the following diff here: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/843/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vectorization_div0/ < expr: ((cdouble >= -500) and (cdouble < -199)) --- > expr: ((cdouble >= (- 500)) and (cdouble < (- 199))) The difference is that the unary-minus was constant-folded so one blank is missing. This is good and not a problem. The query still vectorizes. The vectorization code has a special-purpose constant folder in it now to do this. Your code will make that unnecessary. You can safely regenerate the .q.out file for this test. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867292#comment-13867292
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12622188/HIVE-5771.5.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 4908 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notexists_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_div0
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/843/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/843/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12622188
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864422#comment-13864422 ] Eric Hanson commented on HIVE-5771: --- Glad to hear it! > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864352#comment-13864352 ] Ted Xu commented on HIVE-5771: -- Hi Eric, Thank you for your concern. I'm sorry the work have been delayed by some other stuff. I will move it forward ASAP. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13862064#comment-13862064 ] Eric Hanson commented on HIVE-5771: --- Where does this patch stand? Ted, are you going to move it forward? > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, > HIVE-5771.4.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827671#comment-13827671
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614749/HIVE-5771.4.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/371/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/371/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-371/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target
shims/common-secure/target packaging/target hbase-handler/target
testutils/target jdbc/target metastore/target itests/target
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/storage-handlers/hbase/target
hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen
service/target contrib/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1543817.
At revision 1543814.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12614749
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.4.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827118#comment-13827118
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614632/HIVE-5771.3.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/367/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/367/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-367/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target
shims/common-secure/target packaging/target hbase-handler/target
testutils/target jdbc/target metastore/target itests/target
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/storage-handlers/hbase/target
hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen
contrib/target service/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1543651.
At revision 1543651.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12614632
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch,
> HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825543#comment-13825543
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614382/HIVE-5771.2.patch
{color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 4613 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_between
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_find_in_set
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reverse
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_casts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_string_funcs
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby6
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/342/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/342/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 49 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12614382
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.patch
>
>
> Currently there is no constant foldin
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824885#comment-13824885
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614281/HIVE-5771.1.patch
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 4612 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_in_file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby6
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/334/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/334/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12614281
> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions
> are evaluated at runtime.
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase,
> however, it is still a runtime evaluation and it doesn't propagate constants
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824834#comment-13824834 ] Ted Xu commented on HIVE-5771: -- Hi Eric, Thank you for your reviewing. I already updated the diff file according to your suggestions. With regard to timestamp literal and cast constants, this optimizer will fold them like other primitive types. I tested timestamp and date, it look good to me. I'm sorry I haven't got chance to merge the tests into HIVE-5771.1.patch. I will add them later. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.1.patch, HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824310#comment-13824310
]
Hive QA commented on HIVE-5771:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12613993/HIVE-5771.patch
{color:red}ERROR:{color} -1 due to 92 failed/errored test(s), 4616 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_cast
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_comparison
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_dp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_file_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_set_processor_namespaces
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subq
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824275#comment-13824275 ] Eric Hanson commented on HIVE-5771: --- Ted, This looks excellent. Please address my comments on ReviewBoard. They are mostly suggestions about code comments and style. This patch will definitely help more queries vectorize and reduce some work needed for the translation from a standard expression to a VectorExpression. One thing you may want to think about is that right now there is no such thing as a Timestamp literal, I don't think. But you will create Timestamp constants in your folding. I don't think that will cause any problems, but it might. It is something to think about. E.g. do you need an test that casts a string to a timestamp? Eric > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824147#comment-13824147 ] Eric Hanson commented on HIVE-5771: --- Right now, a query like this: select ... from T where a = 2 * 2; will not vectorize. This is because the "=(column,scalar)" operator looks in the syntax tree to find variable expression result or column on the left and a single scalar on the right. If the constant folder can turn 2 * 2 into the integer literal 4, then the vectorization code can process the "a = 4" filter. Being able to fold CAST(constant-expression AS type) will also help, because people use this, especially for casting a string literal to a timestamp. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823214#comment-13823214 ] Ted Xu commented on HIVE-5771: -- Eric, thank you for your comment. Yes, the constant propagate is an logical plan optimizer so it will process before vectorization process. But I'm not sure how constant folding can benefit vectorized execution. If there is anything I can do to improve it please let me know, thanks! I created a review request here: https://reviews.apache.org/r/15561/ > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.patch, constprog_default_true.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823211#comment-13823211 ] Ted Xu commented on HIVE-5771: -- Thanks Xuefu! I already renamed it. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: HIVE-5771.patch, constprog_default_true.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823000#comment-13823000 ] Eric Hanson commented on HIVE-5771: --- Does this do constant folding before the vectorization process? If it can do that, then more queries will vectorize. Please put up a ReviewBoard entry for your patch to make it easier to review. Thanks, Eric > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: constprog_default_true.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822655#comment-13822655 ] Xuefu Zhang commented on HIVE-5771: --- [~tedxu] Thank you for your patch. Please rename the patch according to the convention so that test can run, something like HIVE-5771.patch, HIVE-5771.1.patch, and so on. > Constant propagation optimizer for Hive > --- > > Key: HIVE-5771 > URL: https://issues.apache.org/jira/browse/HIVE-5771 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ted Xu >Assignee: Ted Xu > Attachments: constprog_default_true.patch > > > Currently there is no constant folding/propagation optimizer, all expressions > are evaluated at runtime. > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, > however, it is still a runtime evaluation and it doesn't propagate constants > from a subquery to outside. > It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.1#6144)
