[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=532253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532253
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 07/Jan/21 03:57
Start Date: 07/Jan/21 03:57
Worklog Time Spent: 10m
Work Description: kasakrisz merged pull request #1772:
URL: https://github.com/apache/hive/pull/1772
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532253)
Time Spent: 2.5h (was: 2h 20m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531794
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 10:16
Start Date: 06/Jan/21 10:16
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r552486656
##
File path:
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
##
@@ -33,4 +33,4 @@ POSTHOOK: type: CREATE_MATERIALIZED_VIEW
POSTHOOK: Input: default@amvrng_table
POSTHOOK: Output: database:default
POSTHOOK: Output: default@amvrng_mat_view
-FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation QUERY [[INSERT,
DELETE] on Object [type=TABLE_OR_VIEW, name=default.amvrng_mat_view,
action=INSERT_OVERWRITE]]
+FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation
ALTER_MATERIALIZED_VIEW_REBUILD [[OBJECT OWNERSHIP] on Object
[type=TABLE_OR_VIEW, name=default.amvrng_mat_view, action=INSERT_OVERWRITE]]
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531794)
Time Spent: 2h 20m (was: 2h 10m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531762
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 08:52
Start Date: 06/Jan/21 08:52
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r552443494
##
File path:
ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out
##
@@ -198,9 +198,7 @@ PREHOOK: type: QUERY
POSTHOOK: query: EXPLAIN INSERT OVERWRITE TABLE destinTable PARTITION
(ds='2011-11-11', hr='11') if not exists
SELECT one,two FROM sourceTable WHERE ds='2011-11-11' AND hr='12' order by one
desc, two desc limit 5
POSTHOOK: type: QUERY
-STAGE DEPENDENCIES:
Review comment:
Reverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531762)
Time Spent: 2h 10m (was: 2h)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531761
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 08:51
Start Date: 06/Jan/21 08:51
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r552443328
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -28,11 +28,14 @@
import com.google.common.util.concurrent.MoreExecutors;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import static
org.apache.hadoop.hive.conf.Constants.MATERIALIZED_VIEW_REWRITING_TIME_WINDOW;
import static
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE;
import static
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.getDefaultCatalog;
import static org.apache.hadoop.hive.ql.io.AcidUtils.getFullTableName;
import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.CALCITE;
import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.ALL;
+import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.CALCITE;
Review comment:
removed duplicates
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531761)
Time Spent: 2h (was: 1h 50m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531335
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 05/Jan/21 16:48
Start Date: 05/Jan/21 16:48
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r552045065
##
File path:
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
##
@@ -33,4 +33,4 @@ POSTHOOK: type: CREATE_MATERIALIZED_VIEW
POSTHOOK: Input: default@amvrng_table
POSTHOOK: Output: database:default
POSTHOOK: Output: default@amvrng_mat_view
-FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation QUERY [[INSERT,
DELETE] on Object [type=TABLE_OR_VIEW, name=default.amvrng_mat_view,
action=INSERT_OVERWRITE]]
+FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation
ALTER_MATERIALIZED_VIEW_REBUILD [[OBJECT OWNERSHIP] on Object
[type=TABLE_OR_VIEW, name=default.amvrng_mat_view, action=INSERT_OVERWRITE]]
Review comment:
It seems we need object ownership for rebuild now (this is probably
because of the changes in Operation2Privilege.java). We should mimic
HiveOperationType.QUERY privileges instead of other ALTER statements (we
probably need SELECT, INSERT, and DELETE on the MV).
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -28,11 +28,14 @@
import com.google.common.util.concurrent.MoreExecutors;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import static
org.apache.hadoop.hive.conf.Constants.MATERIALIZED_VIEW_REWRITING_TIME_WINDOW;
import static
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE;
import static
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.getDefaultCatalog;
import static org.apache.hadoop.hive.ql.io.AcidUtils.getFullTableName;
import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.CALCITE;
import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.ALL;
+import static
org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization.RewriteAlgorithm.CALCITE;
Review comment:
nit. repeated import
##
File path:
ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out
##
@@ -198,9 +198,7 @@ PREHOOK: type: QUERY
POSTHOOK: query: EXPLAIN INSERT OVERWRITE TABLE destinTable PARTITION
(ds='2011-11-11', hr='11') if not exists
SELECT one,two FROM sourceTable WHERE ds='2011-11-11' AND hr='12' order by one
desc, two desc limit 5
POSTHOOK: type: QUERY
-STAGE DEPENDENCIES:
Review comment:
I'm going to back down from this change. It seems there were occasions
where this was empty. Let's just keep it as it was for backwards compatibility.
If we want to remove them, we can do it in a separate JIRA. Sorry about the
confusion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531335)
Time Spent: 1h 50m (was: 1h 40m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531042&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531042
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 05/Jan/21 07:12
Start Date: 05/Jan/21 07:12
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551752970
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
+PREHOOK: query: explain
+alter materialized view mat1 rebuild
+PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+POSTHOOK: query: explain
+alter materialized view mat1 rebuild
+POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+STAGE DEPENDENCIES:
Review comment:
Reverted the part related to formatted mode since that could be a
breaking change if some 3rd party app relies on the existing json schema.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531042)
Time Spent: 1h 40m (was: 1.5h)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=530588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530588
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 10:06
Start Date: 04/Jan/21 10:06
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551220628
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
##
@@ -120,6 +120,8 @@
new Privilege[]{Privilege.DROP}),
ALTER_MATERIALIZED_VIEW_REWRITE("ALTER_MATERIALIZED_VIEW_REWRITE",
HiveParser.TOK_ALTER_MATERIALIZED_VIEW_REWRITE,
new Privilege[]{Privilege.ALTER_METADATA}, null),
+ ALTER_MATERIALIZED_VIEW_REBUILD("ALTER_MATERIALIZED_VIEW_REBUILD",
HiveParser.TOK_ALTER_MATERIALIZED_VIEW_REBUILD,
Review comment:
Without this an exception is thrown at execution time.
```
java.lang.IllegalStateException: Unknown HiveOperation(null) for
queryId=krisz_20210104002338_38173ed5-af09-4e65-af17-8be1dc49e62e
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.verifyState(DbTxnManager.java:315)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:389)
...
at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:329)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:193)
...
```
Fixed the inconsistency by setting back the operation type to
`ALTER_MATERIALIZED_VIEW_REBUILD` after analyzing the query like in the case of
`CREATE_MATERIALIZED_VIEW`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530588)
Time Spent: 1.5h (was: 1h 20m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=530559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530559
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 08:20
Start Date: 04/Jan/21 08:20
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551170582
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
+PREHOOK: query: explain
+alter materialized view mat1 rebuild
+PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+POSTHOOK: query: explain
+alter materialized view mat1 rebuild
+POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+STAGE DEPENDENCIES:
Review comment:
1. Moved `STAGE DEPENDENCIES` and `STAGE PLANS` constants to
`DagJsonParser` so these can be accessed from both `DagJsonParser` and
`ExplainTask`
2. In formatted mode If dependencies and plans are empty json objects do not
add them to the final json.
3. In normal mode added check if task is empty do not print these to output
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530559)
Time Spent: 1h 20m (was: 1h 10m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=530546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530546
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 07:41
Start Date: 04/Jan/21 07:41
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551156318
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
+PREHOOK: query: explain
+alter materialized view mat1 rebuild
+PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+POSTHOOK: query: explain
+alter materialized view mat1 rebuild
+POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+STAGE DEPENDENCIES:
+
+STAGE PLANS:
+Materialized view default.mat1 is up to date. Cancelling rebuild.
+PREHOOK: query: alter materialized view mat1 rebuild
+PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+POSTHOOK: query: alter materialized view mat1 rebuild
+POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: explain
+alter materialized view mat1 rebuild
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Output: default@mat1
+POSTHOOK: query: explain
+alter materialized view mat1 rebuild
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: default@mat1
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-2 depends on stages: Stage-1
+ Stage-0 depends on stages: Stage-2
+ Stage-3 depends on stages: Stage-0
+ Stage-4 depends on stages: Stage-3
+
+STAGE PLANS:
+ Stage: Stage-1
+Tez
+ A masked pattern was here
+ Edges:
+Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
+ A masked pattern was here
+ Vertices:
+Map 1
+Map Operator Tree:
+TableScan
+ alias: t1
+ filterExpr: ((ROW__ID.writeid > 1L) and (col0 = 1)) (type:
boolean)
+ Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
+ Filter Operator
+predicate: ((ROW__ID.writeid > 1L) and (col0 = 1)) (type:
boolean)
+Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: COMPLETE
+Select Operator
+ expressions: 1 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 4 Basic stats:
COMPLETE Column stats: COMPLETE
+ File Output Operator
+compressed: false
+Statistics: Num rows: 1 Data size: 4 Basic stats:
COMPLETE Column stats: COMPLETE
+table:
+input format:
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+output format:
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+name: default.mat1
+ Select Operator
+expressions: _col0 (type: int)
+outputColumnNames: col0
+Statistics: Num rows: 1 Data size: 4 Basic stats:
COMPLETE Colu
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=530527&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530527
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 07:00
Start Date: 04/Jan/21 07:00
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551144553
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
##
@@ -63,6 +66,20 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
unparseTranslator.addTableNameTranslation(tableTree,
SessionState.get().getCurrentDatabase());
return;
}
+
+try {
+ Boolean outdated = db.isOutdatedMaterializedView(getTxnMgr(), tableName);
+ if (outdated != null && !outdated) {
+String msg = String.format("Materialized view %s.%s is up to date.
Cancelling rebuild.",
+tableName.getDb(), tableName.getTable());
+LOG.info(msg);
+console.printInfo(msg, false);
Review comment:
Changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530527)
Time Spent: 1h (was: 50m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=530525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530525
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 06:56
Start Date: 04/Jan/21 06:56
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r551143401
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
Review comment:
This message belongs to the next command output:
```
explain
alter materialized view mat1 rebuild;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530525)
Time Spent: 50m (was: 40m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=527908&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-527908
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 24/Dec/20 00:45
Start Date: 24/Dec/20 00:45
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r548328477
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
+PREHOOK: query: explain
+alter materialized view mat1 rebuild
+PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+POSTHOOK: query: explain
+alter materialized view mat1 rebuild
+POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
+STAGE DEPENDENCIES:
Review comment:
If it is easy to do, can STAGE DEPENDENCIES and STAGE PLANS not be
printed, e.g., it could be that if they are made null instead of empty, they
are skipped in the EXPLAIN?
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: insert into t1(col0) values(1)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@t1
+POSTHOOK: query: insert into t1(col0) values(1)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@t1
+POSTHOOK: Lineage: t1.col0 SCRIPT []
+PREHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+PREHOOK: type: CREATE_MATERIALIZED_VIEW
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@mat1
+POSTHOOK: query: create materialized view mat1 as
+select col0 from t1 where col0 = 1
+POSTHOOK: type: CREATE_MATERIALIZED_VIEW
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@mat1
+Materialized view default.mat1 is up to date. Cancelling rebuild.
Review comment:
This should not be printed since this is a CREATE MV statement. Please
review the code, we may miss a check to avoid printing a message.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
##
@@ -120,6 +120,8 @@
new Privilege[]{Privilege.DROP}),
ALTER_MATERIALIZED_VIEW_REWRITE("ALTER_MATERIALIZED_VIEW_REWRITE",
HiveParser.TOK_ALTER_MATERIALIZED_VIEW_REWRITE,
new Privilege[]{Privilege.ALTER_METADATA}, null),
+ ALTER_MATERIALIZED_VIEW_REBUILD("ALTER_MATERIALIZED_VIEW_REBUILD",
HiveParser.TOK_ALTER_MATERIALIZED_VIEW_REBUILD,
Review comment:
Why do we need to add this operation? I saw a comment about it in
another conversation but it was not clear over there.
In the q file below, I see that that this change is leading to inconsistent
operation type when rebuild is executed (QUERY) vs skipped
(ALTER_MATERIALIZED_VIEW_REBUILD). This should not happen: The operation type
should be the same in both cases.
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rebuild_2.q.out
##
@@ -0,0 +1,171 @@
+PREHOOK: query: create table t1(col0 int) stored as orc TBLPROPERTIES
('transactional'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: databa
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=525009&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-525009
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 16/Dec/20 13:06
Start Date: 16/Dec/20 13:06
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r544282613
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
##
@@ -69,6 +72,23 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
LOG.debug("Rebuilding materialized view " +
tableName.getNotEmptyDbTable());
super.analyzeInternal(rewrittenAST);
+
+try {
+ Table table = db.getTable(tableName.getDb(), tableName.getTable());
Review comment:
Thanks for highlighting that we have all necessary info before calling
`super.analyzeInternal()`.
Creating `HiveOperation.ALTER_MATERIALIZED_VIEW_REBUILD` was necessary to
get this work but no need to modify the `rootTasks`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 525009)
Time Spent: 0.5h (was: 20m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=523925&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-523925
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 14/Dec/20 15:17
Start Date: 14/Dec/20 15:17
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r542464850
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
##
@@ -69,6 +72,23 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
LOG.debug("Rebuilding materialized view " +
tableName.getNotEmptyDbTable());
super.analyzeInternal(rewrittenAST);
+
+try {
+ Table table = db.getTable(tableName.getDb(), tableName.getTable());
Review comment:
Is there any reason not to do this check right after tableName is parsed
from the tree? Probably I'm missing something, but I believe that right after
line 62 this could be checked - no need to create the rootTasks then clear
them. Also it would be nice to move this logic into a separate function,
returning. boolean if it is outdated or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 523925)
Time Spent: 20m (was: 10m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=523799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-523799
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 14/Dec/20 10:44
Start Date: 14/Dec/20 10:44
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #1772:
URL: https://github.com/apache/hive/pull/1772
### What changes were proposed in this pull request?
1. If the materialized view after compilation in
`AlterMaterializedViewRebuildAnalyzer` is up to date clear all root task in
`SemanticAnalyzer` to cancel query execution and print a message to the console.
2. `validTxnsList`, `currentTxnWriteIds` and `timeWindow` was calculated
before every call of `isOutdatedMaterializedView`. These are moved to
`isOutdatedMaterializedView` method body. Since this requires `Hive` instance
`isOutdatedMaterializedView` is moved from `HiveMaterializedViewUtils` to
`Hive.java` as an instance method.
3. Call `Hive.isOutdatedMaterializedView` from
`AlterMaterializedViewRebuildAnalyzer` and `DescTableOperation`
### Why are the changes needed?
Optimize MV rewrite.
### Does this PR introduce _any_ user-facing change?
Yes. When executing `alter materialized view
rebuild` commands and MV is up to date a message like this is written into
console:
```
Materialized view default.mat1 is up to date. Cancelling rebuild.
```
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=mv_rebuild.q -pl itests/qtest -Pitests
mvnql
materialized_view_rewrite_1.q,materialized_view_rewrite_2.q,materialized_view_rewrite_3.q,materialized_view_rewrite_4.q,materialized_view_rewrite_5.q,materialized_view_rewrite_6.q,materialized_view_rewrite_7.q,materialized_view_rewrite_8.q,materialized_view_rewrite_9.q,materialized_view_rewrite_10
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 523799)
Remaining Estimate: 0h
Time Spent: 10m
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
