[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=826388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-826388
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 16/Nov/22 04:16
Start Date: 16/Nov/22 04:16
Worklog Time Spent: 10m
Work Description: kasakrisz merged PR #3753:
URL: https://github.com/apache/hive/pull/3753
Issue Time Tracking
---
Worklog Id: (was: 826388)
Time Spent: 7h 40m (was: 7.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=826387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-826387
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 16/Nov/22 04:15
Start Date: 16/Nov/22 04:15
Worklog Time Spent: 10m
Work Description: kasakrisz commented on PR #3753:
URL: https://github.com/apache/hive/pull/3753#issuecomment-1316304825
The `implement` method is never implemented in any of the operators. I
submitted an experimental patch to remove it completely and all PTests passed
#3766
Issue Time Tracking
---
Worklog Id: (was: 826387)
Time Spent: 7.5h (was: 7h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=825669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-825669
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 14/Nov/22 08:52
Start Date: 14/Nov/22 08:52
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3753:
URL: https://github.com/apache/hive/pull/3753#issuecomment-1313304664
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3753)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3753=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3753=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 825669)
Time Spent: 7h 20m (was: 7h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=825227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-825227
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 11/Nov/22 10:15
Start Date: 11/Nov/22 10:15
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3753:
URL: https://github.com/apache/hive/pull/3753#issuecomment-1311501027
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3753)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3753=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3753=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3753=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3753=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 825227)
Time Spent: 7h 10m (was: 7h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=825210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-825210
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 11/Nov/22 09:30
Start Date: 11/Nov/22 09:30
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request, #3753:
URL: https://github.com/apache/hive/pull/3753
### What changes were proposed in this pull request?
Add missing `HiveRelNode` interface implementation to `HiveValues`
### Why are the changes needed?
All Hive operators must implement it to ensure Hive calling convetion.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing ptests.
Issue Time Tracking
---
Worklog Id: (was: 825210)
Time Spent: 7h (was: 6h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814525
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 07/Oct/22 03:43
Start Date: 07/Oct/22 03:43
Worklog Time Spent: 10m
Work Description: kasakrisz merged PR #3588:
URL: https://github.com/apache/hive/pull/3588
Issue Time Tracking
---
Worklog Id: (was: 814525)
Time Spent: 6h 50m (was: 6h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814413
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 06/Oct/22 15:28
Start Date: 06/Oct/22 15:28
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1270269524
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[6
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[1 Security
Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[64 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 814413)
Time Spent: 6h 40m (was: 6.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814086
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:05
Start Date: 05/Oct/22 20:05
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1265553074
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[5
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[17 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 814086)
Time Spent: 6.5h (was: 6h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814068
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:05
Start Date: 05/Oct/22 20:05
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985791359
##
ql/src/test/results/clientpositive/llap/auto_join21.q.out:
##
@@ -43,37 +45,38 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3
input vertices:
1 Map 2
-Statistics: Num rows: 1 Data size: 356 Basic stats:
COMPLETE Column stats: COMPLETE
+Statistics: Num rows: 55 Data size: 19580 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col2 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 1 Data size: 356 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 55 Data size: 19580 Basic
stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string), _col1
(type: string), _col3 (type: string)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map 2
Map Operator Tree:
TableScan
alias: src2
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key)
< 11.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
- Limit
-Number of rows: 0
-Statistics: Num rows: 1 Data size: 178 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+predicate: ((UDFToDouble(key) > 9.0D) and
(UDFToDouble(key) < 11.0D)) (type: boolean)
+Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
Review Comment:
Added comments to the jira.
##
ql/src/test/queries/clientpositive/antijoin.q:
##
@@ -44,7 +44,7 @@ explain cbo select a.key, a.value from t1_n55 a left join
t2_n33 b on a.key=b.ke
select a.key, a.value from t1_n55 a left join t2_n33 b on a.key=b.key join
t3_n12 c on a.key=c.key where b.key is null sort by a.key, a.value;
Issue Time Tracking
---
Worklog Id: (was: 814068)
Time Spent: 6h 20m (was: 6h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814049
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:04
Start Date: 05/Oct/22 20:04
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985743640
##
ql/src/test/results/clientpositive/llap/subquery_ALL.q.out:
##
@@ -413,8 +413,7 @@ POSTHOOK: Input: default@part
POSTHOOK: Input: default@part_null_n0
A masked pattern was here
26
-Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_1, $hdt$_2]] in Stage
'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[38][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in
Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[22][tables = [$hdt$_0, $hdt$_1]] in Stage
'Reducer 2' is a cross product
Review Comment:
> Is there a branch here that is simplified to empty? Why?
This is the plan after subquery rewrite and decorrelation:
```
HiveProject(_o__c0=[$0])
HiveAggregate(group=[{}], agg#0=[count()])
HiveProject($f0=[$0])
HiveProject(p_partkey=[$0], p_name=[$1], p_mfgr=[$2], p_brand=[$3],
p_type=[$4], p_size=[$5], p_container=[$6], p_retailprice=[$7], p_comment=[$8],
BLOCK__OFFSET__INSIDE__FILE=[$9], INPUT__FILE__NAME=[$10], ROW__ID=[$11],
ROW__IS__DELETED=[$12])
HiveFilter(condition=[IS NULL(OR(AND(IS NOT NULL($16), <>($13, 0)),
AND(OR(IS NULL($0), <($14, $13)), null, <>($13, 0), IS NULL($16])
HiveJoin(condition=[=($0, $15)], joinType=[left],
algorithm=[none], cost=[not available])
HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
cost=[not available])
HiveTableScan(table=[[default, part]], table:alias=[part])
HiveAggregate(group=[{}], c=[COUNT()], ck=[COUNT($0)])
HiveProject(p_partkey=[$0])
HiveFilter(condition=[IS NULL($0)])
HiveTableScan(table=[[default, part_null_n0]],
table:alias=[part_null_n0])
HiveAggregate(group=[{0, 1}])
HiveProject(p_partkey=[$0], literalTrue=[true])
HiveFilter(condition=[IS NULL($0)])
HiveTableScan(table=[[default, part_null_n0]],
table:alias=[part_null_n0])
```
`HiveJoin(condition=[=($0, $15)], joinType=[left]` is removed because
ReduceExpressionRule transforms the join condition always `false`.
It does because `$15` coming from the right branch is always `null` due to
`HiveFilter(condition=[IS NULL($0)])`
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveValues.java:
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.reloperators;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+
+import java.util.List;
+
+/**
+ * Subclass of {@link org.apache.calcite.rel.core.Values}.
+ * Targeting Hive engine.
+ */
+public class HiveValues extends Values {
+
+ public HiveValues(
+ RelOptCluster cluster,
+ RelDataType rowType,
+ ImmutableList> tuples,
+ RelTraitSet traits) {
+super(cluster, rowType, tuples, traits);
+ }
+
+ @Override
+ public RelNode copy(RelTraitSet traitSet, List inputs) {
+if (getInputs().equals(inputs) && traitSet.equals(getTraitSet())) {
+ return this;
+}
+
Review Comment:
Changed this to always create new instance.
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java:
##
@@ -74,4 +74,14 @@ public final class Bug {
* Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-4704
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814035=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814035
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:03
Start Date: 05/Oct/22 20:03
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985692371
##
ql/src/test/results/clientpositive/llap/ppd_udf_col.q.out:
##
@@ -80,22 +80,9 @@ STAGE DEPENDENCIES:
STAGE PLANS:
Stage: Stage-0
Fetch Operator
- limit: -1
+ limit: 0
Processor Tree:
-TableScan
- alias: src
- filterExpr: (UDFToDouble(key) = 100.0D) (type: boolean)
- Filter Operator
-predicate: (UDFToDouble(key) = 100.0D) (type: boolean)
-Limit
- Number of rows: 0
- Select Operator
-expressions: key (type: string)
-outputColumnNames: _col0
-Select Operator
- expressions: _col0 (type: string), rand() (type: double),
'4' (type: string)
- outputColumnNames: _col0, _col1, _col2
- ListSink
+ListSink
Review Comment:
The original empty plan has a TS on the `dummy_table` but this is converted
to a Fetch task.
```
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: 0
Processor Tree:
ListSink
```
Issue Time Tracking
---
Worklog Id: (was: 814035)
Time Spent: 6h (was: 5h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814019
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:02
Start Date: 05/Oct/22 20:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985669610
##
ql/src/test/results/clientpositive/llap/masking_10.q.out:
##
@@ -137,9 +136,7 @@ STAGE PLANS:
Tez
A masked pattern was here
Edges:
-Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 4 (CUSTOM_SIMPLE_EDGE)
-Reducer 3 <- Map 1 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 1 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
Review Comment:
This is the query after applying the masking
```
select `alias01`.`key`, `alias01`.`value`, `alias02`.`a`, `alias02`.`value`,
`alias03`.`key`, `alias03`.`value` from
(SELECT `key`, CAST(reverse(value) AS string) AS `value`,
BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID, ROW__IS__DELETED FROM
`default`.`masking_test` WHERE key % 2 = 0 and key < 10)`alias01`
left join
(
select 2017 as `a`, `value` from (SELECT `key`, CAST(reverse(value) AS
string) AS `value`, BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID,
ROW__IS__DELETED FROM `default`.`masking_test` WHERE key % 2 = 0 and key <
10)`masking_test` group by 1, 2
) `alias02`
on `alias01`.key = `alias02`.`a`
left join
(SELECT `key`, CAST(reverse(value) AS string) AS `value`,
BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID, ROW__IS__DELETED FROM
`default`.`masking_test` WHERE key % 2 = 0 and key < 10)`alias03`
on `alias01`.key = `alias03`.key
```
The first join has a condition: `alias01.key = alias02.a`
In the left branch there is a Filter on `key`: `key % 2 = 0 and key < 10`
In the right branch `a` is constant `2017` so the join condition is going to
be evaluated always `false` and that join is replaced by its left branch
##
ql/src/test/results/clientpositive/llap/ppd_udf_col.q.out:
##
@@ -80,22 +80,9 @@ STAGE DEPENDENCIES:
STAGE PLANS:
Stage: Stage-0
Fetch Operator
- limit: -1
+ limit: 0
Processor Tree:
-TableScan
- alias: src
- filterExpr: (UDFToDouble(key) = 100.0D) (type: boolean)
- Filter Operator
-predicate: (UDFToDouble(key) = 100.0D) (type: boolean)
-Limit
- Number of rows: 0
- Select Operator
-expressions: key (type: string)
-outputColumnNames: _col0
-Select Operator
- expressions: _col0 (type: string), rand() (type: double),
'4' (type: string)
- outputColumnNames: _col0, _col1, _col2
- ListSink
+ListSink
Review Comment:
This is the empty plan
```
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: 0
Processor Tree:
ListSink
```
Issue Time Tracking
---
Worklog Id: (was: 814019)
Time Spent: 5h 40m (was: 5.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814020
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:02
Start Date: 05/Oct/22 20:02
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985676478
##
ql/src/test/results/clientpositive/llap/masking_10.q.out:
##
@@ -137,9 +136,7 @@ STAGE PLANS:
Tez
A masked pattern was here
Edges:
-Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 4 (CUSTOM_SIMPLE_EDGE)
-Reducer 3 <- Map 1 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 1 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
Review Comment:
Thanks for clarifying. I forgot about the query rewrites due to masking.
Issue Time Tracking
---
Worklog Id: (was: 814020)
Time Spent: 5h 50m (was: 5h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=814005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814005
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:01
Start Date: 05/Oct/22 20:01
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r985560069
##
ql/src/test/results/clientpositive/llap/float_equality.q.out:
##
@@ -9,9 +9,7 @@ POSTHOOK: Input: _dummy_database@_dummy_table
1
PREHOOK: query: select 1 where -0.0<0.0
PREHOOK: type: QUERY
-PREHOOK: Input: _dummy_database@_dummy_table
Review Comment:
This was the original plan: empty result query is converted to a sub-query
hence the the limit is not treated as global limit and the query is executed.
```
HiveProject(_o__c0=[1])
HiveSortLimit(fetch=[0])
HiveProject(DUMMY=[0])
HiveTableScan(table=[[_dummy_database, _dummy_table]],
table:alias=[_dummy_table])
```
These is a TS on `_dummy_table`
```
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
Limit
Number of rows: 0
Select Operator
Select Operator
expressions: 1 (type: int)
outputColumnNames: _col0
ListSink
```
The new plan is just an empty values op.
```
HiveValues(tuples=[[]])
```
There is no TS,
```
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: 0
Processor Tree:
ListSink
```
##
ql/src/test/results/clientpositive/llap/fold_case.q.out:
##
@@ -177,7 +180,7 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
-Execution mode: vectorized, llap
+Execution mode: llap
Review Comment:
The vectorization was lost because instead of a real table the
`_dummy_table` with empty schema is scanned. It affects only this mapper. Rest
of the plan (Reducer) is still vectorized.
Issue Time Tracking
---
Worklog Id: (was: 814005)
Time Spent: 5.5h (was: 5h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813808
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 22:56
Start Date: 30/Sep/22 22:56
Worklog Time Spent: 10m
Work Description: difin commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984991177
##
ql/src/test/results/clientpositive/llap/subquery_ALL.q.out:
##
@@ -413,8 +413,7 @@ POSTHOOK: Input: default@part
POSTHOOK: Input: default@part_null_n0
A masked pattern was here
26
-Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_1, $hdt$_2]] in Stage
'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[38][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in
Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[22][tables = [$hdt$_0, $hdt$_1]] in Stage
'Reducer 2' is a cross product
Review Comment:
Will these changes work for insert statements that produce zero records
also, or only for select queries?
I am asking because there is a ticket
https://issues.apache.org/jira/browse/HIVE-24466 - "insert queries should not
launch job when condition in the query would output 0 rows", I was wondering it
these changes are a fix for that ticket too.
Issue Time Tracking
---
Worklog Id: (was: 813808)
Time Spent: 5h 20m (was: 5h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813666
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 11:11
Start Date: 30/Sep/22 11:11
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984481438
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+
+import java.util.Collections;
+import java.util.List;
+
+import static com.google.common.collect.Iterables.concat;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
Review Comment:
One thing to keep in mind is that the rule may turn `ConventionXx` to
`HiveXX` since we use the `HIVE_BUILDER`. If you think that it is safe to do so
here then I am fine with it.
Issue Time Tracking
---
Worklog Id: (was: 813666)
Time Spent: 5h 10m (was: 5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813661
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:57
Start Date: 30/Sep/22 10:57
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984472010
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+
+import java.util.Collections;
+import java.util.List;
+
+import static com.google.common.collect.Iterables.concat;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
Review Comment:
I tried some tests locally and it works with Jdbc operators. I think it
makes sense to match general operators.
Issue Time Tracking
---
Worklog Id: (was: 813661)
Time Spent: 4h 50m (was: 4h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813662
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:57
Start Date: 30/Sep/22 10:57
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984472066
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,87 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ public static ASTNode emptyPlan(RelDataType dataType) {
+if (dataType.getFieldCount() == 0) {
+ throw new IllegalArgumentException("Schema is empty.");
+}
+
+ASTBuilder select = ASTBuilder.construct(HiveParser.TOK_SELECT,
"TOK_SELECT");
+for (int i = 0; i < dataType.getFieldCount(); ++i) {
+ RelDataTypeField fieldType = dataType.getFieldList().get(i);
+ if (fieldType.getValue().getSqlTypeName() == SqlTypeName.NULL) {
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_NULL, "TOK_NULL").node(),
+fieldType.getName()));
+ } else {
+ASTNode typeNode = createCast(fieldType);
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_FUNCTION, "TOK_FUNCTION")
+.add(typeNode)
+.add(ASTBuilder.construct(HiveParser.TOK_NULL,
"TOK_NULL").node()).node(),
+fieldType.getName()));
+ }
+}
+
+ASTNode insert = ASTBuilder.
+construct(HiveParser.TOK_INSERT, "TOK_INSERT").
+add(ASTBuilder.destNode()).
+add(select).
+add(ASTBuilder.limit(0, 0)).
+node();
+
+return ASTBuilder.
+construct(HiveParser.TOK_QUERY, "TOK_QUERY").
+add(insert).
+node();
+ }
+
+ private static ASTNode createCast(RelDataTypeField fieldType) {
+HiveToken ht = TypeConverter.hiveToken(fieldType.getType());
+ASTNode typeNode;
+if (ht == null) {
+ typeNode = ASTBuilder.construct(
+ HiveParser.Identifier,
fieldType.getType().getSqlTypeName().getName().toLowerCase()).node();
+} else {
+ ASTBuilder typeNodeBuilder = ASTBuilder.construct(ht.type, ht.text);
+ if (ht.args != null) {
+for (String castArg : ht.args) {
+ typeNodeBuilder.add(HiveParser.Identifier, castArg);
+}
+ }
+ typeNode = typeNodeBuilder.node();
+}
+return typeNode;
+ }
+
private ASTNode convert() throws CalciteSemanticException {
+if (root instanceof HiveValues) {
+ HiveValues values = (HiveValues) root;
+ if (isEmpty(values)) {
+select = values;
+return emptyPlan(values.getRowType());
+ }
Review Comment:
Exception or assertion both are fine with me; as you prefer as long as it
fails fast :)
Issue Time Tracking
---
Worklog Id: (was: 813662)
Time Spent: 5h (was: 4h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813657
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:46
Start Date: 30/Sep/22 10:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984464007
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,87 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ public static ASTNode emptyPlan(RelDataType dataType) {
+if (dataType.getFieldCount() == 0) {
+ throw new IllegalArgumentException("Schema is empty.");
+}
+
+ASTBuilder select = ASTBuilder.construct(HiveParser.TOK_SELECT,
"TOK_SELECT");
+for (int i = 0; i < dataType.getFieldCount(); ++i) {
+ RelDataTypeField fieldType = dataType.getFieldList().get(i);
+ if (fieldType.getValue().getSqlTypeName() == SqlTypeName.NULL) {
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_NULL, "TOK_NULL").node(),
+fieldType.getName()));
+ } else {
+ASTNode typeNode = createCast(fieldType);
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_FUNCTION, "TOK_FUNCTION")
+.add(typeNode)
+.add(ASTBuilder.construct(HiveParser.TOK_NULL,
"TOK_NULL").node()).node(),
+fieldType.getName()));
+ }
+}
+
+ASTNode insert = ASTBuilder.
+construct(HiveParser.TOK_INSERT, "TOK_INSERT").
+add(ASTBuilder.destNode()).
+add(select).
+add(ASTBuilder.limit(0, 0)).
+node();
+
+return ASTBuilder.
+construct(HiveParser.TOK_QUERY, "TOK_QUERY").
+add(insert).
+node();
+ }
+
+ private static ASTNode createCast(RelDataTypeField fieldType) {
+HiveToken ht = TypeConverter.hiveToken(fieldType.getType());
+ASTNode typeNode;
+if (ht == null) {
+ typeNode = ASTBuilder.construct(
+ HiveParser.Identifier,
fieldType.getType().getSqlTypeName().getName().toLowerCase()).node();
+} else {
+ ASTBuilder typeNodeBuilder = ASTBuilder.construct(ht.type, ht.text);
+ if (ht.args != null) {
+for (String castArg : ht.args) {
+ typeNodeBuilder.add(HiveParser.Identifier, castArg);
+}
+ }
+ typeNode = typeNodeBuilder.node();
+}
+return typeNode;
+ }
+
private ASTNode convert() throws CalciteSemanticException {
+if (root instanceof HiveValues) {
+ HiveValues values = (HiveValues) root;
+ if (isEmpty(values)) {
+select = values;
+return emptyPlan(values.getRowType());
+ }
Review Comment:
No it can't. Theoretically non-empty `HiveValues` are not created currently.
How about throwing an exception? My problem with assertions is that it does
not evaluates in production environments and I had some difficulties to repro
issues locally from q test: so the q test failed with assertion error but
something different exception was thrown from a different place in prod.
Issue Time Tracking
---
Worklog Id: (was: 813657)
Time Spent: 4h 40m (was: 4.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
>
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813623
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 09:11
Start Date: 30/Sep/22 09:11
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984386036
##
ql/src/test/results/clientpositive/llap/float_equality.q.out:
##
@@ -9,9 +9,7 @@ POSTHOOK: Input: _dummy_database@_dummy_table
1
PREHOOK: query: select 1 where -0.0<0.0
PREHOOK: type: QUERY
-PREHOOK: Input: _dummy_database@_dummy_table
Review Comment:
In my understanding the scan is involved in an empty sub-plan that now is
not executed, so the scan does not take place.
Issue Time Tracking
---
Worklog Id: (was: 813623)
Time Spent: 4.5h (was: 4h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813509
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 29/Sep/22 21:49
Start Date: 29/Sep/22 21:49
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r983479717
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java:
##
@@ -74,4 +74,14 @@ public final class Bug {
* Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-4704 is
fixed.
*/
public static final boolean CALCITE_4704_FIXED = false;
+
+ /**
+ * Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-5293 is
fixed.
Review Comment:
nit: Link points to the wrong JIRA.
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java:
##
@@ -74,4 +74,14 @@ public final class Bug {
* Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-4704 is
fixed.
*/
public static final boolean CALCITE_4704_FIXED = false;
+
+ /**
+ * Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-5293 is
fixed.
+ */
+ public static final boolean CALCITE_5293_FIXED = false;
+
+ /**
+ * Whether https://issues.apache.org/jira/browse/CALCITE-4704;>CALCITE-5294 is
fixed.
Review Comment:
nit: Link points to the wrong JIRA.
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,87 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ public static ASTNode emptyPlan(RelDataType dataType) {
+if (dataType.getFieldCount() == 0) {
+ throw new IllegalArgumentException("Schema is empty.");
+}
+
+ASTBuilder select = ASTBuilder.construct(HiveParser.TOK_SELECT,
"TOK_SELECT");
+for (int i = 0; i < dataType.getFieldCount(); ++i) {
+ RelDataTypeField fieldType = dataType.getFieldList().get(i);
+ if (fieldType.getValue().getSqlTypeName() == SqlTypeName.NULL) {
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_NULL, "TOK_NULL").node(),
+fieldType.getName()));
+ } else {
+ASTNode typeNode = createCast(fieldType);
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_FUNCTION, "TOK_FUNCTION")
+.add(typeNode)
+.add(ASTBuilder.construct(HiveParser.TOK_NULL,
"TOK_NULL").node()).node(),
+fieldType.getName()));
+ }
+}
+
+ASTNode insert = ASTBuilder.
+construct(HiveParser.TOK_INSERT, "TOK_INSERT").
+add(ASTBuilder.destNode()).
+add(select).
+add(ASTBuilder.limit(0, 0)).
+node();
+
+return ASTBuilder.
+construct(HiveParser.TOK_QUERY, "TOK_QUERY").
+add(insert).
+node();
+ }
+
+ private static ASTNode createCast(RelDataTypeField fieldType) {
+HiveToken ht = TypeConverter.hiveToken(fieldType.getType());
+ASTNode typeNode;
+if (ht == null) {
+ typeNode = ASTBuilder.construct(
+ HiveParser.Identifier,
fieldType.getType().getSqlTypeName().getName().toLowerCase()).node();
+} else {
+ ASTBuilder typeNodeBuilder = ASTBuilder.construct(ht.type, ht.text);
+ if (ht.args != null) {
+for (String castArg : ht.args) {
+ typeNodeBuilder.add(HiveParser.Identifier, castArg);
+}
+ }
+ typeNode = typeNodeBuilder.node();
+}
+return typeNode;
+ }
+
private ASTNode convert() throws CalciteSemanticException {
+if (root instanceof HiveValues) {
+ HiveValues values = (HiveValues) root;
+ if (isEmpty(values)) {
+select = values;
+return emptyPlan(values.getRowType());
+ }
Review Comment:
If we have a `HiveValues` that it is not empty can the code below handle it?
If not, I would put an assertion with a proper comment.
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveValues.java:
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=812132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812132
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 26/Sep/22 14:37
Start Date: 26/Sep/22 14:37
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1258140223
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[11 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 812132)
Time Spent: 4h 10m (was: 4h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=812031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812031
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 26/Sep/22 08:58
Start Date: 26/Sep/22 08:58
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1257715251
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[12 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 812031)
Time Spent: 4h (was: 3h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811996
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 26/Sep/22 05:35
Start Date: 26/Sep/22 05:35
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1257504403
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[11 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 811996)
Time Spent: 3h 50m (was: 3h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811538
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 23/Sep/22 09:01
Start Date: 23/Sep/22 09:01
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1255962118
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[8 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 811538)
Time Spent: 3h 40m (was: 3.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811510
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 23/Sep/22 07:16
Start Date: 23/Sep/22 07:16
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r978329079
##
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TestASTConverter.java:
##
@@ -0,0 +1,102 @@
+package org.apache.hadoop.hive.ql.optimizer.calcite.translator;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.sql.type.ArraySqlType;
+import org.apache.calcite.sql.type.BasicSqlType;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveTypeSystemImpl;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.emptyPlan;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class TestASTConverter {
+ @Test
+ public void testEmptyPlanWhenInputSchemaIsEmpty() {
+IllegalArgumentException thrown =
Assertions.assertThrows(IllegalArgumentException.class, () -> {
+ emptyPlan(new RelRecordType(Collections.emptyList()));
+});
+
+Assertions.assertTrue(thrown.getMessage().contains("Schema is empty"));
+ }
+
+ @Test
+ public void testEmptyPlan() {
+List fields = asList(
+new RelDataTypeFieldImpl("a", 0, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.INTEGER)),
+new RelDataTypeFieldImpl("b", 1, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.CHAR, 30)),
+new RelDataTypeFieldImpl("c", 2, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.NULL)),
+new RelDataTypeFieldImpl("d", 3, new ArraySqlType(new
BasicSqlType(new HiveTypeSystemImpl(), SqlTypeName.INTEGER), false)),
+new RelDataTypeFieldImpl("e", 4, new ArraySqlType(new
BasicSqlType(new HiveTypeSystemImpl(), SqlTypeName.INTEGER), true)));
+RelDataType dataType = new RelRecordType(fields);
+
+ASTNode tree = emptyPlan(dataType);
+
+// TOK_QUERY -> TOK_INSERT -> TOK_SELECT
+assertThat(tree.getChild(0).getChild(1).getChildCount(),
is(fields.size()));
+
+assertThat(tree.dump(), is(EXPECTED_TREE));
+ }
+
+ private static final String EXPECTED_TREE = "\n" +
+ "TOK_QUERY\n" +
+ " TOK_INSERT\n" +
+ " TOK_DESTINATION\n" +
+ " TOK_DIR\n" +
+ "TOK_TMP_FILE\n" +
+ " TOK_SELECT\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " TOK_INT\n" +
+ " TOK_NULL\n" +
+ "a\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " TOK_CHAR\n" +
+ " 30\n" +
+ " TOK_NULL\n" +
+ "b\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_NULL\n" +
+ "c\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " array\n" +
+ " TOK_NULL\n" +
+ "d\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " array\n" +
+ " TOK_NULL\n" +
+ "e\n" +
+ " TOK_LIMIT\n" +
+ " 0\n" +
+ " 0\n";
+}
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811288
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 15:32
Start Date: 22/Sep/22 15:32
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977804764
##
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TestASTConverter.java:
##
@@ -0,0 +1,102 @@
+package org.apache.hadoop.hive.ql.optimizer.calcite.translator;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.sql.type.ArraySqlType;
+import org.apache.calcite.sql.type.BasicSqlType;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveTypeSystemImpl;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.emptyPlan;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class TestASTConverter {
+ @Test
+ public void testEmptyPlanWhenInputSchemaIsEmpty() {
+IllegalArgumentException thrown =
Assertions.assertThrows(IllegalArgumentException.class, () -> {
+ emptyPlan(new RelRecordType(Collections.emptyList()));
+});
+
+Assertions.assertTrue(thrown.getMessage().contains("Schema is empty"));
+ }
+
+ @Test
+ public void testEmptyPlan() {
+List fields = asList(
+new RelDataTypeFieldImpl("a", 0, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.INTEGER)),
+new RelDataTypeFieldImpl("b", 1, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.CHAR, 30)),
+new RelDataTypeFieldImpl("c", 2, new BasicSqlType(new
HiveTypeSystemImpl(), SqlTypeName.NULL)),
+new RelDataTypeFieldImpl("d", 3, new ArraySqlType(new
BasicSqlType(new HiveTypeSystemImpl(), SqlTypeName.INTEGER), false)),
+new RelDataTypeFieldImpl("e", 4, new ArraySqlType(new
BasicSqlType(new HiveTypeSystemImpl(), SqlTypeName.INTEGER), true)));
+RelDataType dataType = new RelRecordType(fields);
+
+ASTNode tree = emptyPlan(dataType);
+
+// TOK_QUERY -> TOK_INSERT -> TOK_SELECT
+assertThat(tree.getChild(0).getChild(1).getChildCount(),
is(fields.size()));
+
+assertThat(tree.dump(), is(EXPECTED_TREE));
+ }
+
+ private static final String EXPECTED_TREE = "\n" +
+ "TOK_QUERY\n" +
+ " TOK_INSERT\n" +
+ " TOK_DESTINATION\n" +
+ " TOK_DIR\n" +
+ "TOK_TMP_FILE\n" +
+ " TOK_SELECT\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " TOK_INT\n" +
+ " TOK_NULL\n" +
+ "a\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " TOK_CHAR\n" +
+ " 30\n" +
+ " TOK_NULL\n" +
+ "b\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_NULL\n" +
+ "c\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " array\n" +
+ " TOK_NULL\n" +
+ "d\n" +
+ " TOK_SELEXPR\n" +
+ "TOK_FUNCTION\n" +
+ " array\n" +
+ " TOK_NULL\n" +
+ "e\n" +
+ " TOK_LIMIT\n" +
+ " 0\n" +
+ " 0\n";
+}
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811279
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 15:25
Start Date: 22/Sep/22 15:25
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977796570
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,83 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ private static ASTNode emptyPlan(RelDataType dataType) {
Review Comment:
Added UT.
Issue Time Tracking
---
Worklog Id: (was: 811279)
Time Spent: 3h 10m (was: 3h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811221
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 13:06
Start Date: 22/Sep/22 13:06
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977631693
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -308,17 +390,26 @@ private void convertOrderLimitToASTNode(HiveSortLimit
hiveSortLimit) {
private void convertSortToASTNode(HiveSortExchange hiveSortExchange) {
List fieldCollations =
hiveSortExchange.getCollation().getFieldCollations();
-convertFieldCollationsToASTNode(hiveSortExchange, new
Schema(hiveSortExchange), fieldCollations,
+convertFieldCollationsAndSetOrderAST(hiveSortExchange, new
Schema(hiveSortExchange), fieldCollations,
null, HiveParser.TOK_SORTBY, "TOK_SORTBY");
}
- private void convertFieldCollationsToASTNode(
+ private void convertFieldCollationsAndSetOrderAST(
RelNode node, Schema schema, List
fieldCollations, Map obRefToCallMap,
int astToken, String astText) {
+
if (fieldCollations.isEmpty()) {
return;
}
+hiveAST.order = convertFieldCollationsToASTNode(
+node.getCluster().getRexBuilder(), schema, fieldCollations,
obRefToCallMap, astToken, astText);
+ }
+
+ public static ASTNode convertFieldCollationsToASTNode(
Review Comment:
No, changed to `private`.
Issue Time Tracking
---
Worklog Id: (was: 811221)
Time Spent: 3h (was: 2h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811194
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 12:25
Start Date: 22/Sep/22 12:25
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977590776
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java:
##
@@ -717,9 +719,12 @@ public TrimResult trimFields(Aggregate aggregate,
ImmutableBitSet fieldsUsed, Se
final RexNode filterArg = aggCall.filterArg < 0 ? null
: relBuilder.field(Mappings.apply(inputMapping,
aggCall.filterArg));
RelBuilder.AggCall newAggCall =
-relBuilder.aggregateCall(aggCall.getAggregation(),
-aggCall.isDistinct(), aggCall.isApproximate(),
-filterArg, aggCall.name, args);
+relBuilder.aggregateCall(aggCall.getAggregation(), args)
+.distinct(aggCall.isDistinct())
+.filter(filterArg)
+.approximate(aggCall.isApproximate())
+.sort(relBuilder.fields(aggCall.collation))
+.as(aggCall.name);
Review Comment:
Thanks @kasakrisz, I had that impression but it sounded weird that we did
not handle collation before, but now it's clear why.
Issue Time Tracking
---
Worklog Id: (was: 811194)
Time Spent: 2h 50m (was: 2h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811188
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 12:16
Start Date: 22/Sep/22 12:16
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977581838
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveValues;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule FILTER_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyFilter")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Filter.class, singleRel -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule JOIN_LEFT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withOperandSupplier(b0 ->
+ b0.operand(Join.class).inputs(
+ b1 -> b1.operand(Values.class)
+
.predicate(Values::isEmpty).noInputs(),
+ b2 -> b2.operand(RelNode.class).anyInputs()))
+ .withDescription("HivePruneEmptyJoin(left)")
+ .as(JoinLeftEmptyRuleConfig.class)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ /**
+ * Improved version of Calcite's {@link
PruneEmptyRules.JoinLeftEmptyRuleConfig}.
+ * In case of right outer join if the left branch is empty the join operator
can be removed
+ * and take the right branch only.
+ *
+ * select * from (select * from emp where 1=0) right join dept
+ * to
+ * select null as emp.col0 ... null as emp.coln, dept.* from dept
+ */
+ public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
+@Override default PruneEmptyRule toRule() {
+ return new PruneEmptyRule(this) {
+
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811187
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 12:15
Start Date: 22/Sep/22 12:15
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977580067
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java:
##
@@ -717,9 +719,12 @@ public TrimResult trimFields(Aggregate aggregate,
ImmutableBitSet fieldsUsed, Se
final RexNode filterArg = aggCall.filterArg < 0 ? null
: relBuilder.field(Mappings.apply(inputMapping,
aggCall.filterArg));
RelBuilder.AggCall newAggCall =
-relBuilder.aggregateCall(aggCall.getAggregation(),
-aggCall.isDistinct(), aggCall.isApproximate(),
-filterArg, aggCall.name, args);
+relBuilder.aggregateCall(aggCall.getAggregation(), args)
+.distinct(aggCall.isDistinct())
+.filter(filterArg)
+.approximate(aggCall.isApproximate())
+.sort(relBuilder.fields(aggCall.collation))
+.as(aggCall.name);
Review Comment:
Collation was not handled at all in this customized trimmer and it was not
an issue in the past since it was not added to the CBO plan.
With this patch we start adding collation to aggregate calls so it must be
transformed by the trimmer like in the Calcite version.
Issue Time Tracking
---
Worklog Id: (was: 811187)
Time Spent: 2.5h (was: 2h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811185
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 12:09
Start Date: 22/Sep/22 12:09
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977574485
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java:
##
@@ -336,16 +336,13 @@ protected VBuilderPAP(Aggregate aggregate, Project
project, String sketchClass)
@Override
boolean isApplicable(AggregateCall aggCall) {
-if ((aggInput instanceof Project)
-&& !aggCall.isDistinct() && aggCall.getArgList().size() == 4
+if ((aggInput != null)
+&& !aggCall.isDistinct() && aggCall.getArgList().size() == 1
Review Comment:
This rule already has a good comment
https://github.com/apache/hive/blob/a017e54c98c76ccf0c185b47533b336b0a398dc7/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java#L296-L303
Before this patch function calls like
```
percentile_disc(0.2) within group (order by id)
```
was transformed at AST level to a 4 parameter function:
```
percentile_disc(0.2, id, asc, nulls_last)
```
and the collation was not represented in the CBO plan at all.
With this patch collation is properly set in these aggregate calls in CBO
plan so this rule must expect that version.
An extra predicate was added to this condition which checks the number of
order by keys:
```
&& aggCall.collation.getFieldCollations().size() == 1) {
```
It must be 1 in case of `percentile_disc`
Issue Time Tracking
---
Worklog Id: (was: 811185)
Time Spent: 2h 20m (was: 2h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811183
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 11:54
Start Date: 22/Sep/22 11:54
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977561476
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveValues;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule FILTER_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyFilter")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Filter.class, singleRel -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule JOIN_LEFT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withOperandSupplier(b0 ->
+ b0.operand(Join.class).inputs(
+ b1 -> b1.operand(Values.class)
+
.predicate(Values::isEmpty).noInputs(),
+ b2 -> b2.operand(RelNode.class).anyInputs()))
+ .withDescription("HivePruneEmptyJoin(left)")
+ .as(JoinLeftEmptyRuleConfig.class)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ /**
+ * Improved version of Calcite's {@link
PruneEmptyRules.JoinLeftEmptyRuleConfig}.
+ * In case of right outer join if the left branch is empty the join operator
can be removed
+ * and take the right branch only.
+ *
+ * select * from (select * from emp where 1=0) right join dept
+ * to
+ * select null as emp.col0 ... null as emp.coln, dept.* from dept
+ */
+ public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
+@Override default PruneEmptyRule toRule() {
+ return new PruneEmptyRule(this) {
+
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811180=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811180
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 11:50
Start Date: 22/Sep/22 11:50
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977557947
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveValues;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule FILTER_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyFilter")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Filter.class, singleRel -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule JOIN_LEFT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withOperandSupplier(b0 ->
+ b0.operand(Join.class).inputs(
+ b1 -> b1.operand(Values.class)
+
.predicate(Values::isEmpty).noInputs(),
+ b2 -> b2.operand(RelNode.class).anyInputs()))
+ .withDescription("HivePruneEmptyJoin(left)")
+ .as(JoinLeftEmptyRuleConfig.class)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ /**
+ * Improved version of Calcite's {@link
PruneEmptyRules.JoinLeftEmptyRuleConfig}.
+ * In case of right outer join if the left branch is empty the join operator
can be removed
+ * and take the right branch only.
+ *
+ * select * from (select * from emp where 1=0) right join dept
+ * to
+ * select null as emp.col0 ... null as emp.coln, dept.* from dept
+ */
+ public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
+@Override default PruneEmptyRule toRule() {
+ return new PruneEmptyRule(this) {
+
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811179
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 11:50
Start Date: 22/Sep/22 11:50
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977557467
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveValues;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule FILTER_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyFilter")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Filter.class, singleRel -> true)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ public static final RelOptRule JOIN_LEFT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withOperandSupplier(b0 ->
+ b0.operand(Join.class).inputs(
+ b1 -> b1.operand(Values.class)
+
.predicate(Values::isEmpty).noInputs(),
+ b2 -> b2.operand(RelNode.class).anyInputs()))
+ .withDescription("HivePruneEmptyJoin(left)")
+ .as(JoinLeftEmptyRuleConfig.class)
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
+
+ /**
+ * Improved version of Calcite's {@link
PruneEmptyRules.JoinLeftEmptyRuleConfig}.
+ * In case of right outer join if the left branch is empty the join operator
can be removed
+ * and take the right branch only.
+ *
+ * select * from (select * from emp where 1=0) right join dept
+ * to
+ * select null as emp.col0 ... null as emp.coln, dept.* from dept
+ */
+ public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
+@Override default PruneEmptyRule toRule() {
+ return new PruneEmptyRule(this) {
+
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=811178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811178
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 22/Sep/22 11:49
Start Date: 22/Sep/22 11:49
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r977556946
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveValues.java:
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.reloperators;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+
+import java.util.List;
+
+/**
+ * Subclass of {@link org.apache.calcite.rel.core.Values}.
+ * Specialized to Hive engine.
+ */
+public class HiveValues extends Values {
+
+ public HiveValues(
+ RelOptCluster cluster,
+ RelDataType rowType,
+ ImmutableList> tuples,
+ RelTraitSet traits) {
Review Comment:
A single line constructor head definition would be more than 120 character.
I know than there is no rule to enforce this limit but this was used before
we switched to PR and I still try to follow this rule.
(I experienced that Calcite has a 100 char limitation.)
So I defined this in two lines like we did it in case of HiveProject.
Issue Time Tracking
---
Worklog Id: (was: 811178)
Time Spent: 1h 40m (was: 1.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=810902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-810902
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 21/Sep/22 17:12
Start Date: 21/Sep/22 17:12
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r976632375
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -308,17 +390,26 @@ private void convertOrderLimitToASTNode(HiveSortLimit
hiveSortLimit) {
private void convertSortToASTNode(HiveSortExchange hiveSortExchange) {
List fieldCollations =
hiveSortExchange.getCollation().getFieldCollations();
-convertFieldCollationsToASTNode(hiveSortExchange, new
Schema(hiveSortExchange), fieldCollations,
+convertFieldCollationsAndSetOrderAST(hiveSortExchange, new
Schema(hiveSortExchange), fieldCollations,
null, HiveParser.TOK_SORTBY, "TOK_SORTBY");
}
- private void convertFieldCollationsToASTNode(
+ private void convertFieldCollationsAndSetOrderAST(
RelNode node, Schema schema, List
fieldCollations, Map obRefToCallMap,
int astToken, String astText) {
+
if (fieldCollations.isEmpty()) {
return;
}
+hiveAST.order = convertFieldCollationsToASTNode(
+node.getCluster().getRexBuilder(), schema, fieldCollations,
obRefToCallMap, astToken, astText);
+ }
+
+ public static ASTNode convertFieldCollationsToASTNode(
Review Comment:
Does it have to be `public`?
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java:
##
@@ -336,16 +336,13 @@ protected VBuilderPAP(Aggregate aggregate, Project
project, String sketchClass)
@Override
boolean isApplicable(AggregateCall aggCall) {
-if ((aggInput instanceof Project)
-&& !aggCall.isDistinct() && aggCall.getArgList().size() == 4
+if ((aggInput != null)
Review Comment:
We don't need the parentheses anymore:
```suggestion
if (aggInput != null
```
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveValues;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=810752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-810752
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 21/Sep/22 13:18
Start Date: 21/Sep/22 13:18
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1253699359
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[2
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[18 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 810752)
Time Spent: 1h 20m (was: 1h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=810386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-810386
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 20/Sep/22 13:44
Start Date: 20/Sep/22 13:44
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1252377068
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[2
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[13 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 810386)
Time Spent: 1h 10m (was: 1h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=81=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 19/Sep/22 11:47
Start Date: 19/Sep/22 11:47
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1250912746
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[2
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[16 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 81)
Time Spent: 1h (was: 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=809543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809543
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 14:20
Start Date: 16/Sep/22 14:20
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1249425376
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[18 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 809543)
Time Spent: 50m (was: 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=807519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807519
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 09/Sep/22 17:33
Start Date: 09/Sep/22 17:33
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1242268147
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 807519)
Time Spent: 40m (was: 0.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=807406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807406
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 09/Sep/22 13:20
Start Date: 09/Sep/22 13:20
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1241968817
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 807406)
Time Spent: 0.5h (was: 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=807295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807295
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 09/Sep/22 07:53
Start Date: 09/Sep/22 07:53
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3588:
URL: https://github.com/apache/hive/pull/3588#issuecomment-1241631243
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3588)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3588=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[14 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3588=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3588=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 807295)
Time Spent: 20m (was: 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=807279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807279
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 09/Sep/22 07:06
Start Date: 09/Sep/22 07:06
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request, #3588:
URL: https://github.com/apache/hive/pull/3588
### What changes were proposed in this pull request?
* Currently Hive represents the empty result operator with
`HiveSortLimit(fetch=0)`. Change this to `HiveValues(tuples[])` like Calcite
does.
* Improve and extend the `PruneEmptyRules` provided by Calcite with Hive
specific functionality.
* Represent the empty `HiveValues` operator with an AST tree of the query
```
select null as colName0... null as colNamen limit 0
```
when converting back the CBO plan to AST.
* Get the schema information from the `HiveValues` row type at CBO -> AST
conversion.
### Why are the changes needed?
* Calcite has built in rules to remove sections of a query plan known never
produces any rows. It makes the CBO plan much simpler.
* In some cases (ex. `select * from table1 where 1=0` ) the whole plan can
be removed and Hive already has an optimization not to execute queries which
does not provide any result. This optimization is built on checking the limit
value at the top level query.
### Does this PR introduce _any_ user-facing change?
No, but `explain` results.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=empty_result_outerjoin.q,empty_result.q,empty_result_union.q -pl
itests/qtest -Pitests
```
Issue Time Tracking
---
Worklog Id: (was: 807279)
Remaining Estimate: 0h
Time Spent: 10m
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
