[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056161#comment-16056161
]
Pengcheng Xiong commented on HIVE-16797:
yes, i did update the golden file.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 3.0.0
>
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056119#comment-16056119
]
Ashutosh Chauhan commented on HIVE-16797:
-
Failure of filter_union.q is on TestMiniLlapLocalCliDriver. On CliDriver it
works fine. So, golden file for llap driver needs to be updated.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 3.0.0
>
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056098#comment-16056098
]
Pengcheng Xiong commented on HIVE-16797:
pushed to master. thanks [~ashutoshc] for the review!
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056097#comment-16056097
]
Pengcheng Xiong commented on HIVE-16797:
explainanalyze_2.q,columnstats_part_coltype.q,filter_union.q can not be
reproduced locally.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054668#comment-16054668
]
Ashutosh Chauhan commented on HIVE-16797:
-
+1
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054660#comment-16054660
]
Hive QA commented on HIVE-16797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12873534/HIVE-16797.05.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10814 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[filter_union]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
(batchId=232)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=239)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=216)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5678/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5678/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5678/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12873534 - PreCommit-HIVE-Build
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch, HIVE-16797.05.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053117#comment-16053117
]
Hive QA commented on HIVE-16797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12873401/HIVE-16797.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
(batchId=232)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=103)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=216)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12873401 - PreCommit-HIVE-Build
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch, HIVE-16797.04.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047170#comment-16047170
]
Ashutosh Chauhan commented on HIVE-16797:
-
Some comments on RB. For 4 points listed above: few clarifications: for (2)
That's an enhancement for RexSimplify which will help all usages of
RexSimplify. Lets open a jira on Calcite and pursue it there. For (3) How does
Project help ?
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045723#comment-16045723
]
Pengcheng Xiong commented on HIVE-16797:
[~ashutoshc], the failed test are not reproducible locally.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045482#comment-16045482
]
Hive QA commented on HIVE-16797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12872413/HIVE-16797.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10831 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
(batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5616/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5616/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5616/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12872413 - PreCommit-HIVE-Build
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch,
> HIVE-16797.03.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037454#comment-16037454
]
Hive QA commented on HIVE-16797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12871266/HIVE-16797.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10822 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=237)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_aggr] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union24] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union30] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union34] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unionall_unbalancedppd]
(batchId=2)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_multiinsert]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query33]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query56]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query5] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query60]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query71]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query76]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query77]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query78]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query80]
(batchId=232)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union30]
(batchId=134)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5534/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5534/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12871266 - PreCommit-HIVE-Build
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037265#comment-16037265
]
Pengcheng Xiong commented on HIVE-16797:
I use pull-up-constant to pull the constant out of union. Then use RexSimplify
to simplify that to see if it can be reduced to always false. However, there
are still several comments regarding patch 02: (1) it sounds like that I was
not able to simplify (($2=1 OR $2=2) AND $2=3) to a false. Here ($2=1 OR $2=2)
comes from two branches of union. Thus, I introduce a hive union merge rule
(btw, calcite union merge rule does not fire well in current hive master) so
that we can check ($2=1 AND $2=3) and ($2=2 AND $2=3), repectively, which
works with RexSimplify. (2) it sounds like RexSimplify also can not reduce
($2>2 AND $2=3) to false. There is a test case in filter_union.q for that and
you will see. (3) if we can assume that it is always a project under union, we
may have better options.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch, HIVE-16797.02.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036088#comment-16036088
]
Pengcheng Xiong commented on HIVE-16797:
yes, I will do that accordingly. btw, unionall_unbalancedppd is actually an
improvement
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036078#comment-16036078
]
Ashutosh Chauhan commented on HIVE-16797:
-
[~pxiong] Can you rebase now that HIVE-16775 is in? You may have to update
golden files for new tests there and a failure reported here on QA run. Can you
also create a RB for it?
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035530#comment-16035530
]
Pengcheng Xiong commented on HIVE-16797:
[~ashutoshc], that is true. And also query74, query11 as well.
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034903#comment-16034903
]
Ashutosh Chauhan commented on HIVE-16797:
-
This seems to be working. On query 4, instead of 67 vertex Tez Dag, after this
patch we are only getting 33 vertex dag. cc: [~gopalv] [~ndembla]
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034092#comment-16034092
]
Hive QA commented on HIVE-16797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12870871/HIVE-16797.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10813 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unionall_unbalancedppd]
(batchId=2)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12870871 - PreCommit-HIVE-Build
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16797) Enhance HiveFilterSetOpTransposeRule to remove union branches
[
https://issues.apache.org/jira/browse/HIVE-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033810#comment-16033810
]
Pengcheng Xiong commented on HIVE-16797:
temporary testing patch
> Enhance HiveFilterSetOpTransposeRule to remove union branches
> -
>
> Key: HIVE-16797
> URL: https://issues.apache.org/jira/browse/HIVE-16797
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16797.01.patch
>
>
> in query4.q, we can see that it creates a CTE with union all of 3 branches.
> Then it is going to do a 3 way self-join of the CTE with predicates. The
> predicates actually specifies only one of the branch in CTE to participate in
> the join. Thus, in some cases, e.g.,
> {code}
>/- filter(false) -TS0
> union all - filter(false) -TS1
>\-TS2
> {code}
> we can cut the branches of TS0 and TS1. The union becomes only TS2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
