[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860993 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 08/May/23 11:21 Start Date: 08/May/23 11:21 Worklog Time Spent: 10m Work Description: kasakrisz merged PR #4278: URL: https://github.com/apache/hive/pull/4278 Issue Time Tracking --- Worklog Id: (was: 860993) Time Spent: 5h 40m (was: 5.5h) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860991
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 08/May/23 11:04
Start Date: 08/May/23 11:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4278:
URL: https://github.com/apache/hive/pull/4278#discussion_r1187315293
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3096,6 +3096,8 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
assert t != null;
if (AcidUtils.isTransactionalTable(t) && sharedWrite) {
compBuilder.setSharedWrite();
+} else if (MetaStoreUtils.isNonNativeTable(t.getTTable())) {
+ compBuilder.setLock(getLockTypeFromStorageHandler(output, t));
Review Comment:
Issue Time Tracking
---
Worklog Id: (was: 860991)
Time Spent: 5.5h (was: 5h 20m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860585
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/May/23 16:35
Start Date: 04/May/23 16:35
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4278:
URL: https://github.com/apache/hive/pull/4278#discussion_r1185253932
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3096,6 +3096,8 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
assert t != null;
if (AcidUtils.isTransactionalTable(t) && sharedWrite) {
compBuilder.setSharedWrite();
+} else if (MetaStoreUtils.isNonNativeTable(t.getTTable())) {
+ compBuilder.setLock(getLockTypeFromStorageHandler(output, t));
Review Comment:
Yes, I think the function `AcidUtils.makeLockComponents()` has to be
reconsidered from Iceberg standpoint.
But I think that is beyond the scope of this patch.
Issue Time Tracking
---
Worklog Id: (was: 860585)
Time Spent: 5h 20m (was: 5h 10m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860557
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/May/23 12:48
Start Date: 04/May/23 12:48
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4278:
URL: https://github.com/apache/hive/pull/4278#discussion_r1184967460
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3096,6 +3096,8 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
assert t != null;
if (AcidUtils.isTransactionalTable(t) && sharedWrite) {
compBuilder.setSharedWrite();
+} else if (MetaStoreUtils.isNonNativeTable(t.getTTable())) {
+ compBuilder.setLock(getLockTypeFromStorageHandler(output, t));
Review Comment:
do we need locks for iceberg delete/update. That should be handled in
iceberg itself, isn't it?
Issue Time Tracking
---
Worklog Id: (was: 860557)
Time Spent: 5h 10m (was: 5h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860300 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 03/May/23 11:48 Start Date: 03/May/23 11:48 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4278: URL: https://github.com/apache/hive/pull/4278#issuecomment-1532887376 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4278) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4278=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4278=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 860300) Time Spent: 5h (was: 4h 50m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860209 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 03/May/23 04:46 Start Date: 03/May/23 04:46 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4278: URL: https://github.com/apache/hive/pull/4278#issuecomment-1532449054 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4278) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4278=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4278=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 860209) Time Spent: 4h 50m (was: 4h 40m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=860060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-860060 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 02/May/23 12:11 Start Date: 02/May/23 12:11 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4278: URL: https://github.com/apache/hive/pull/4278#issuecomment-1531361218 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4278) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4278=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4278=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 860060) Time Spent: 4h 40m (was: 4.5h) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=859984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859984 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 02/May/23 07:50 Start Date: 02/May/23 07:50 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4278: URL: https://github.com/apache/hive/pull/4278#issuecomment-1531033837 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4278) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4278=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4278=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859984) Time Spent: 4.5h (was: 4h 20m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=859672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859672 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 28/Apr/23 14:10 Start Date: 28/Apr/23 14:10 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4278: URL: https://github.com/apache/hive/pull/4278#issuecomment-1527631388 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4278) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4278=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4278=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4278=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4278=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859672) Time Spent: 4h 20m (was: 4h 10m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=859662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859662 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 28/Apr/23 13:07 Start Date: 28/Apr/23 13:07 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request, #4278: URL: https://github.com/apache/hive/pull/4278 ### What changes were proposed in this pull request? * Get the sort acid columns from StorageHandler in case of non native acid MVs when transforming the AST of the incremental mv rebuild. * Update some comments related to the AST transformation * Set `hive.txn.manager` in q tests. ### Why are the changes needed? Incremental rebuild of MV stored by Iceberg is failed if the MV has aggregate. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` mvn test -Dtest.output.overwrite -Dtest=TestIcebergCliDriver -Dqfile=mv_iceberg_orc7.q,mv_iceberg_orc6.q -pl itests/qtest-iceberg -Pitests,iceberg -Drat.skip ``` Issue Time Tracking --- Worklog Id: (was: 859662) Time Spent: 4h 10m (was: 4h) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=856314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-856314 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 12/Apr/23 03:49 Start Date: 12/Apr/23 03:49 Worklog Time Spent: 10m Work Description: kasakrisz merged PR #4166: URL: https://github.com/apache/hive/pull/4166 Issue Time Tracking --- Worklog Id: (was: 856314) Time Spent: 4h (was: 3h 50m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=856225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-856225 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 11/Apr/23 19:57 Start Date: 11/Apr/23 19:57 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1504010533 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 856225) Time Spent: 3h 50m (was: 3h 40m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=855998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855998 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 11/Apr/23 05:33 Start Date: 11/Apr/23 05:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1502711234 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 855998) Time Spent: 3h 40m (was: 3.5h) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854955 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 05/Apr/23 08:01 Start Date: 05/Apr/23 08:01 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1497081163 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854955) Time Spent: 3.5h (was: 3h 20m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854865 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 04/Apr/23 17:08 Start Date: 04/Apr/23 17:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1496319047 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854865) Time Spent: 3h 20m (was: 3h 10m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854845
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:02
Start Date: 04/Apr/23 16:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157469710
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
Renamed
Issue Time Tracking
---
Worklog Id: (was: 854845)
Time Spent: 3h 10m (was: 3h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854844
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:02
Start Date: 04/Apr/23 16:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157194454
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
I think this name emphasizes that it is used in case of native a storage
format. The same convention is used in case of `ColumnAppender`.
Issue Time Tracking
---
Worklog Id: (was: 854844)
Time Spent: 3h (was: 2h 50m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854783
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:46
Start Date: 04/Apr/23 12:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157194454
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
I think this name emphasizes that it is used in case of native a storage
format. The same convention is used in case of `ColumnAppender`.
Issue Time Tracking
---
Worklog Id: (was: 854783)
Time Spent: 2h 50m (was: 2h 40m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854782
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:36
Start Date: 04/Apr/23 12:36
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157183277
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
what about FullAcidMaterializedViewASTBuilder? Should it be
`NativeAcidMaterializedViewASTBuilder`?
Issue Time Tracking
---
Worklog Id: (was: 854782)
Time Spent: 2h 40m (was: 2.5h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854574 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 03/Apr/23 15:53 Start Date: 03/Apr/23 15:53 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1494575502 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854574) Time Spent: 2.5h (was: 2h 20m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854530
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 03/Apr/23 11:52
Start Date: 03/Apr/23 11:52
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1155857426
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -564,30 +582,38 @@ private void fixUpASTAggregateIncrementalRebuild(
HiveParser.TOK_INSERT, HiveParser.TOK_SELECT);
ASTNode selectExprNodeInputROJ = (ASTNode) ParseDriver.adaptor.create(
HiveParser.TOK_SELEXPR, "TOK_SELEXPR");
-ASTNode tableName = createRowIdNode(TableName.getDbTable(
+astBuilder.createAcidSortNodes(TableName.getDbTable(
materializationNode.getChild(0).getText(),
-materializationNode.getChild(1).getText()));
-ParseDriver.adaptor.addChild(selectExprNodeInputROJ, tableName);
-ParseDriver.adaptor.addChild(selectNodeInputROJ, selectExprNodeInputROJ);
+materializationNode.getChild(1).getText()))
+.forEach(astNode ->
ParseDriver.adaptor.addChild(selectNodeInputROJ, astNode));
// 4) Transform first INSERT branch into an UPDATE
-// 4.1) Adding ROW__ID field
-ASTNode selectNodeInUpdate = (ASTNode) updateNode.getChild(1);
-if (selectNodeInUpdate.getType() != HiveParser.TOK_SELECT) {
- throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
-}
-ASTNode selectExprNodeInUpdate = (ASTNode)
ParseDriver.adaptor.dupNode(selectExprNodeInputROJ);
-ParseDriver.adaptor.addChild(selectExprNodeInUpdate,
createRowIdNode((ASTNode) subqueryNodeInputROJ.getChild(1)));
-selectNodeInUpdate.insertChild(0, selectExprNodeInUpdate);
-// 4.2) Modifying filter condition.
+// 4.1) Modifying filter condition.
ASTNode whereClauseInUpdate = findWhereClause(updateNode);
if (whereClauseInUpdate.getChild(0).getType() != HiveParser.KW_OR) {
throw new SemanticException("OR clause expected below TOK_WHERE in
incremental rewriting");
}
// We bypass the OR clause and select the first disjunct for the Update
branch
ParseDriver.adaptor.setChild(whereClauseInUpdate, 0,
disjuncts.get(Context.DestClausePrefix.UPDATE));
+ASTNode updateDeleteNode = (ASTNode)
ParseDriver.adaptor.dupTree(updateNode);
+// 4.2) Adding ROW__ID field
+ASTNode selectNodeInUpdateDelete = (ASTNode) updateDeleteNode.getChild(1);
+if (selectNodeInUpdateDelete.getType() != HiveParser.TOK_SELECT) {
+ throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
+}
+// Remove children
+while (selectNodeInUpdateDelete.getChildCount() > 0) {
+ selectNodeInUpdateDelete.deleteChild(0);
Review Comment:
`org.antlr.runtime.tree.BaseTree` doesn't support removing all children in
one call.
Issue Time Tracking
---
Worklog Id: (was: 854530)
Time Spent: 2h 20m (was: 2h 10m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854527
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 03/Apr/23 11:47
Start Date: 03/Apr/23 11:47
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1155853269
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -564,30 +582,38 @@ private void fixUpASTAggregateIncrementalRebuild(
HiveParser.TOK_INSERT, HiveParser.TOK_SELECT);
ASTNode selectExprNodeInputROJ = (ASTNode) ParseDriver.adaptor.create(
HiveParser.TOK_SELEXPR, "TOK_SELEXPR");
-ASTNode tableName = createRowIdNode(TableName.getDbTable(
+astBuilder.createAcidSortNodes(TableName.getDbTable(
materializationNode.getChild(0).getText(),
-materializationNode.getChild(1).getText()));
-ParseDriver.adaptor.addChild(selectExprNodeInputROJ, tableName);
-ParseDriver.adaptor.addChild(selectNodeInputROJ, selectExprNodeInputROJ);
+materializationNode.getChild(1).getText()))
+.forEach(astNode ->
ParseDriver.adaptor.addChild(selectNodeInputROJ, astNode));
// 4) Transform first INSERT branch into an UPDATE
-// 4.1) Adding ROW__ID field
-ASTNode selectNodeInUpdate = (ASTNode) updateNode.getChild(1);
-if (selectNodeInUpdate.getType() != HiveParser.TOK_SELECT) {
- throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
-}
-ASTNode selectExprNodeInUpdate = (ASTNode)
ParseDriver.adaptor.dupNode(selectExprNodeInputROJ);
-ParseDriver.adaptor.addChild(selectExprNodeInUpdate,
createRowIdNode((ASTNode) subqueryNodeInputROJ.getChild(1)));
-selectNodeInUpdate.insertChild(0, selectExprNodeInUpdate);
-// 4.2) Modifying filter condition.
+// 4.1) Modifying filter condition.
ASTNode whereClauseInUpdate = findWhereClause(updateNode);
if (whereClauseInUpdate.getChild(0).getType() != HiveParser.KW_OR) {
throw new SemanticException("OR clause expected below TOK_WHERE in
incremental rewriting");
}
// We bypass the OR clause and select the first disjunct for the Update
branch
ParseDriver.adaptor.setChild(whereClauseInUpdate, 0,
disjuncts.get(Context.DestClausePrefix.UPDATE));
+ASTNode updateDeleteNode = (ASTNode)
ParseDriver.adaptor.dupTree(updateNode);
+// 4.2) Adding ROW__ID field
+ASTNode selectNodeInUpdateDelete = (ASTNode) updateDeleteNode.getChild(1);
+if (selectNodeInUpdateDelete.getType() != HiveParser.TOK_SELECT) {
+ throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
Review Comment:
I added printing `selectNodeInUpdateDelete.getType()` which is the int
representation if the token.
Issue Time Tracking
---
Worklog Id: (was: 854527)
Time Spent: 2h 10m (was: 2h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854523
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 03/Apr/23 11:46
Start Date: 03/Apr/23 11:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1155851502
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -465,21 +467,37 @@ private RelNode applyIncrementalRebuild(RelNode basePlan,
RelMetadataProvider md
protected ASTNode fixUpAfterCbo(ASTNode originalAst, ASTNode newAst,
CalcitePlanner.PreCboCtx cboCtx)
throws SemanticException {
ASTNode fixedAST = super.fixUpAfterCbo(originalAst, newAst, cboCtx);
+if (mvRebuildMode == MaterializationRebuildMode.INSERT_OVERWRITE_REBUILD) {
+ return fixedAST;
+} else if (mvRebuildMode ==
MaterializationRebuildMode.JOIN_INSERT_REBUILD) {
+ fixUpASTJoinInsertIncrementalRebuild(fixedAST);
+ return fixedAST;
+}
+
+MaterializedViewASTBuilder astBuilder;
+if (AcidUtils.isFullAcidTable(mvTable.getTTable())) {
Review Comment:
done
Issue Time Tracking
---
Worklog Id: (was: 854523)
Time Spent: 2h (was: 1h 50m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854521
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 03/Apr/23 11:45
Start Date: 03/Apr/23 11:45
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1155850815
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
Renamed to `NonNativeAcidMaterializedViewASTBuilder`
Issue Time Tracking
---
Worklog Id: (was: 854521)
Time Spent: 1h 40m (was: 1.5h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854522
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 03/Apr/23 11:45
Start Date: 03/Apr/23 11:45
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1155851179
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -299,7 +300,8 @@ private RelNode applyRecordIncrementalRebuildPlan(
// First we need to check if it is valid to convert to MERGE/INSERT INTO.
// If we succeed, we modify the plan and afterwards the AST.
// MV should be an acid table.
- boolean fullAcidView = AcidUtils.isFullAcidTable(mvTable.getTTable());
+ boolean fullAcidView = AcidUtils.isFullAcidTable(mvTable.getTTable())
Review Comment:
renamed to `acidView`
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -465,21 +467,37 @@ private RelNode applyIncrementalRebuild(RelNode basePlan,
RelMetadataProvider md
protected ASTNode fixUpAfterCbo(ASTNode originalAst, ASTNode newAst,
CalcitePlanner.PreCboCtx cboCtx)
throws SemanticException {
ASTNode fixedAST = super.fixUpAfterCbo(originalAst, newAst, cboCtx);
+if (mvRebuildMode == MaterializationRebuildMode.INSERT_OVERWRITE_REBUILD) {
Review Comment:
refactored
Issue Time Tracking
---
Worklog Id: (was: 854522)
Time Spent: 1h 50m (was: 1h 40m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854221
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 31/Mar/23 16:00
Start Date: 31/Mar/23 16:00
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1154650166
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -564,30 +582,38 @@ private void fixUpASTAggregateIncrementalRebuild(
HiveParser.TOK_INSERT, HiveParser.TOK_SELECT);
ASTNode selectExprNodeInputROJ = (ASTNode) ParseDriver.adaptor.create(
HiveParser.TOK_SELEXPR, "TOK_SELEXPR");
-ASTNode tableName = createRowIdNode(TableName.getDbTable(
+astBuilder.createAcidSortNodes(TableName.getDbTable(
materializationNode.getChild(0).getText(),
-materializationNode.getChild(1).getText()));
-ParseDriver.adaptor.addChild(selectExprNodeInputROJ, tableName);
-ParseDriver.adaptor.addChild(selectNodeInputROJ, selectExprNodeInputROJ);
+materializationNode.getChild(1).getText()))
+.forEach(astNode ->
ParseDriver.adaptor.addChild(selectNodeInputROJ, astNode));
// 4) Transform first INSERT branch into an UPDATE
-// 4.1) Adding ROW__ID field
-ASTNode selectNodeInUpdate = (ASTNode) updateNode.getChild(1);
-if (selectNodeInUpdate.getType() != HiveParser.TOK_SELECT) {
- throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
-}
-ASTNode selectExprNodeInUpdate = (ASTNode)
ParseDriver.adaptor.dupNode(selectExprNodeInputROJ);
-ParseDriver.adaptor.addChild(selectExprNodeInUpdate,
createRowIdNode((ASTNode) subqueryNodeInputROJ.getChild(1)));
-selectNodeInUpdate.insertChild(0, selectExprNodeInUpdate);
-// 4.2) Modifying filter condition.
+// 4.1) Modifying filter condition.
ASTNode whereClauseInUpdate = findWhereClause(updateNode);
if (whereClauseInUpdate.getChild(0).getType() != HiveParser.KW_OR) {
throw new SemanticException("OR clause expected below TOK_WHERE in
incremental rewriting");
}
// We bypass the OR clause and select the first disjunct for the Update
branch
ParseDriver.adaptor.setChild(whereClauseInUpdate, 0,
disjuncts.get(Context.DestClausePrefix.UPDATE));
+ASTNode updateDeleteNode = (ASTNode)
ParseDriver.adaptor.dupTree(updateNode);
+// 4.2) Adding ROW__ID field
+ASTNode selectNodeInUpdateDelete = (ASTNode) updateDeleteNode.getChild(1);
+if (selectNodeInUpdateDelete.getType() != HiveParser.TOK_SELECT) {
+ throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
+}
+// Remove children
+while (selectNodeInUpdateDelete.getChildCount() > 0) {
+ selectNodeInUpdateDelete.deleteChild(0);
Review Comment:
can we remove all children in 1 go and don't call
freshenParentAndChildIndexes for every child?
Issue Time Tracking
---
Worklog Id: (was: 854221)
Time Spent: 1.5h (was: 1h 20m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854218
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 31/Mar/23 15:49
Start Date: 31/Mar/23 15:49
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1154638925
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -564,30 +582,38 @@ private void fixUpASTAggregateIncrementalRebuild(
HiveParser.TOK_INSERT, HiveParser.TOK_SELECT);
ASTNode selectExprNodeInputROJ = (ASTNode) ParseDriver.adaptor.create(
HiveParser.TOK_SELEXPR, "TOK_SELEXPR");
-ASTNode tableName = createRowIdNode(TableName.getDbTable(
+astBuilder.createAcidSortNodes(TableName.getDbTable(
materializationNode.getChild(0).getText(),
-materializationNode.getChild(1).getText()));
-ParseDriver.adaptor.addChild(selectExprNodeInputROJ, tableName);
-ParseDriver.adaptor.addChild(selectNodeInputROJ, selectExprNodeInputROJ);
+materializationNode.getChild(1).getText()))
+.forEach(astNode ->
ParseDriver.adaptor.addChild(selectNodeInputROJ, astNode));
// 4) Transform first INSERT branch into an UPDATE
-// 4.1) Adding ROW__ID field
-ASTNode selectNodeInUpdate = (ASTNode) updateNode.getChild(1);
-if (selectNodeInUpdate.getType() != HiveParser.TOK_SELECT) {
- throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
-}
-ASTNode selectExprNodeInUpdate = (ASTNode)
ParseDriver.adaptor.dupNode(selectExprNodeInputROJ);
-ParseDriver.adaptor.addChild(selectExprNodeInUpdate,
createRowIdNode((ASTNode) subqueryNodeInputROJ.getChild(1)));
-selectNodeInUpdate.insertChild(0, selectExprNodeInUpdate);
-// 4.2) Modifying filter condition.
+// 4.1) Modifying filter condition.
ASTNode whereClauseInUpdate = findWhereClause(updateNode);
if (whereClauseInUpdate.getChild(0).getType() != HiveParser.KW_OR) {
throw new SemanticException("OR clause expected below TOK_WHERE in
incremental rewriting");
}
// We bypass the OR clause and select the first disjunct for the Update
branch
ParseDriver.adaptor.setChild(whereClauseInUpdate, 0,
disjuncts.get(Context.DestClausePrefix.UPDATE));
+ASTNode updateDeleteNode = (ASTNode)
ParseDriver.adaptor.dupTree(updateNode);
+// 4.2) Adding ROW__ID field
+ASTNode selectNodeInUpdateDelete = (ASTNode) updateDeleteNode.getChild(1);
+if (selectNodeInUpdateDelete.getType() != HiveParser.TOK_SELECT) {
+ throw new SemanticException("TOK_SELECT expected in incremental
rewriting");
Review Comment:
should we print the actual type?
Issue Time Tracking
---
Worklog Id: (was: 854218)
Time Spent: 1h 20m (was: 1h 10m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854217
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 31/Mar/23 15:42
Start Date: 31/Mar/23 15:42
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1154631673
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -465,21 +467,37 @@ private RelNode applyIncrementalRebuild(RelNode basePlan,
RelMetadataProvider md
protected ASTNode fixUpAfterCbo(ASTNode originalAst, ASTNode newAst,
CalcitePlanner.PreCboCtx cboCtx)
throws SemanticException {
ASTNode fixedAST = super.fixUpAfterCbo(originalAst, newAst, cboCtx);
+if (mvRebuildMode == MaterializationRebuildMode.INSERT_OVERWRITE_REBUILD) {
+ return fixedAST;
+} else if (mvRebuildMode ==
MaterializationRebuildMode.JOIN_INSERT_REBUILD) {
+ fixUpASTJoinInsertIncrementalRebuild(fixedAST);
+ return fixedAST;
+}
+
+MaterializedViewASTBuilder astBuilder;
+if (AcidUtils.isFullAcidTable(mvTable.getTTable())) {
Review Comment:
please extract this into the helper method: getMaterializedViewASTBuilder()
Issue Time Tracking
---
Worklog Id: (was: 854217)
Time Spent: 1h 10m (was: 1h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854216
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 31/Mar/23 15:41
Start Date: 31/Mar/23 15:41
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1154629845
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java:
##
@@ -465,21 +467,37 @@ private RelNode applyIncrementalRebuild(RelNode basePlan,
RelMetadataProvider md
protected ASTNode fixUpAfterCbo(ASTNode originalAst, ASTNode newAst,
CalcitePlanner.PreCboCtx cboCtx)
throws SemanticException {
ASTNode fixedAST = super.fixUpAfterCbo(originalAst, newAst, cboCtx);
+if (mvRebuildMode == MaterializationRebuildMode.INSERT_OVERWRITE_REBUILD) {
Review Comment:
could we use `switch` statement
Issue Time Tracking
---
Worklog Id: (was: 854216)
Time Spent: 1h (was: 50m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854215 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 31/Mar/23 15:39 Start Date: 31/Mar/23 15:39 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #4166: URL: https://github.com/apache/hive/pull/4166#discussion_r1154628405 ## ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java: ## @@ -299,7 +300,8 @@ private RelNode applyRecordIncrementalRebuildPlan( // First we need to check if it is valid to convert to MERGE/INSERT INTO. // If we succeed, we modify the plan and afterwards the AST. // MV should be an acid table. - boolean fullAcidView = AcidUtils.isFullAcidTable(mvTable.getTTable()); + boolean fullAcidView = AcidUtils.isFullAcidTable(mvTable.getTTable()) Review Comment: var name is confusing, maybe acidView or txnView? Issue Time Tracking --- Worklog Id: (was: 854215) Time Spent: 50m (was: 40m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854214
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 31/Mar/23 15:37
Start Date: 31/Mar/23 15:37
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1154626481
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
could we use consistent naming: NativeAcidMaterializedViewASTBuilder &
NonNativeAcidMaterializedViewASTBuilder? see ColumnAppender
Issue Time Tracking
---
Worklog Id: (was: 854214)
Time Spent: 40m (was: 0.5h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=853666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853666 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 29/Mar/23 12:01 Start Date: 29/Mar/23 12:01 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1488471135 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 853666) Time Spent: 0.5h (was: 20m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=853595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853595 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 29/Mar/23 07:36 Start Date: 29/Mar/23 07:36 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1488089545 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4166) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4166=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4166=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4166=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4166=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 853595) Time Spent: 20m (was: 10m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=853452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-853452 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 28/Mar/23 15:21 Start Date: 28/Mar/23 15:21 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request, #4166: URL: https://github.com/apache/hive/pull/4166 ### What changes were proposed in this pull request? * Enable Incremental rebuild plan transformation of materialized view stored by iceberg and definition contains aggregate. * query the acid sort columns from storage handler in case of non native tables when transforming the AST plan of the MV rebuild ### Why are the changes needed? Improve MV rebuild performance ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_create_rewrite_5.q,materialized_view_create_rewrite_6.q -pl itests/qtest -Pitests mvn test -Dtest.output.overwrite -Dtest=TestIcebergCliDriver -Dqfile=mv_iceberg_orc7.q -pl itests/qtest-iceberg -Pitests,iceberg -Drat.skip ``` Issue Time Tracking --- Worklog Id: (was: 853452) Remaining Estimate: 0h Time Spent: 10m > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and > update existing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
