[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806891
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 08/Sep/22 05:20
Start Date: 08/Sep/22 05:20
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1240232954
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3552)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[17 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 806891)
Time Spent: 3.5h (was: 3h 20m)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[ https://issues.apache.org/jira/browse/HIVE-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25790 started by Teddy Choi. - > Make managed table copies handle updates (FileUtils) > > > Key: HIVE-25790 > URL: https://issues.apache.org/jira/browse/HIVE-25790 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Teddy Choi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26046) MySQL's bit datatype is default to void datatype in hive
[
https://issues.apache.org/jira/browse/HIVE-26046?focusedWorklogId=806861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806861
]
ASF GitHub Bot logged work on HIVE-26046:
-
Author: ASF GitHub Bot
Created on: 08/Sep/22 00:24
Start Date: 08/Sep/22 00:24
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #3276:
HIVE-26046: MySQL's bit datatype is default to void datatype in hive
URL: https://github.com/apache/hive/pull/3276
Issue Time Tracking
---
Worklog Id: (was: 806861)
Time Spent: 5h (was: 4h 50m)
> MySQL's bit datatype is default to void datatype in hive
>
>
> Key: HIVE-26046
> URL: https://issues.apache.org/jira/browse/HIVE-26046
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> describe on a table that contains a "bit" datatype gets mapped to void. We
> need a explicit conversion logic in the MySQL ConnectorProvider to map it to
> a suitable datatype in hive.
> {noformat}
> +---+---++
> | col_name| data_type
> | comment |
> +---+---++
> | tbl_id| bigint
> | from deserializer |
> | create_time | int
> | from deserializer |
> | db_id | bigint
> | from deserializer |
> | last_access_time | int
> | from deserializer |
> | owner | varchar(767)
> | from deserializer |
> | owner_type| varchar(10)
> | from deserializer |
> | retention | int
> | from deserializer |
> | sd_id | bigint
> | from deserializer |
> | tbl_name | varchar(256)
> | from deserializer |
> | tbl_type | varchar(128)
> | from deserializer |
> | view_expanded_text| string
> | from deserializer |
> | view_original_text| string
> | from deserializer |
> | is_rewrite_enabled| void
> | from deserializer |
> | write_id | bigint
> | from deserializer
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?focusedWorklogId=806860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806860 ] ASF GitHub Bot logged work on HIVE-26512: - Author: ASF GitHub Bot Created on: 07/Sep/22 23:58 Start Date: 07/Sep/22 23:58 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3580: URL: https://github.com/apache/hive/pull/3580#issuecomment-1240032686 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3580) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3580=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3580=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 806860) Time Spent: 40m (was: 0.5h) > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Sub-task >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?focusedWorklogId=806784=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806784 ] ASF GitHub Bot logged work on HIVE-26512: - Author: ASF GitHub Bot Created on: 07/Sep/22 19:02 Start Date: 07/Sep/22 19:02 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3580: URL: https://github.com/apache/hive/pull/3580#issuecomment-1239767116 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3580) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3580=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3580=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3580=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3580=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 806784) Time Spent: 0.5h (was: 20m) > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Sub-task >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806771
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 18:10
Start Date: 07/Sep/22 18:10
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1239717450
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3552)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[17 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 806771)
Time Spent: 3h 20m (was: 3h 10m)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26504) User is not able to drop table
[
https://issues.apache.org/jira/browse/HIVE-26504?focusedWorklogId=806753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806753
]
ASF GitHub Bot logged work on HIVE-26504:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 16:07
Start Date: 07/Sep/22 16:07
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3557:
URL: https://github.com/apache/hive/pull/3557#issuecomment-1239595651
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3557)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3557=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3557=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3557=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=CODE_SMELL)
[6 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3557=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3557=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3557=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 806753)
Time Spent: 40m (was: 0.5h)
> User is not able to drop table
> --
>
> Key: HIVE-26504
> URL: https://issues.apache.org/jira/browse/HIVE-26504
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Hive won't store anything in *TAB_COL_STATS* for partitioned table, whereas
> impala stores complete column stats in TAB_COL_STATS for partitioned table.
> Deleting entries in TAB_COL_STATS is based on (DB_NAME, TABLE_NAME), not by
> TBL_ID. Renamed tables were having old names in TAB_COL_STATS.
> To Repro:
> {code:java}
> beeline:
> set hive.create.as.insert.only=false;
> set hive.create.as.acid=false;
> create table testes.table_name_with_partition (id tinyint, name string)
> partitioned by (col_to_partition bigint) stored as parquet;
> insert into testes.table_name_with_partition (id, name, col_to_partition)
> values (1, "a", 2020), (2, "b", 2021), (3, "c", 2022);
> impala:
> compute stats testes.table_name_with_partition; -- backend shows new entries
> in TAB_COL_STATS
> beeline:
> alter table testes.table_name_with_partition rename to
> testes2.table_that_cant_be_droped;
> drop table testes2.table_that_cant_be_droped; -- This fails with
> TAB_COL_STATS_fkey constraint violation.
> {code}
> Exception
[jira] [Work logged] (HIVE-26521) Iceberg: Raise exception when running delete/update statements on V1 tables
[ https://issues.apache.org/jira/browse/HIVE-26521?focusedWorklogId=806751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806751 ] ASF GitHub Bot logged work on HIVE-26521: - Author: ASF GitHub Bot Created on: 07/Sep/22 15:51 Start Date: 07/Sep/22 15:51 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3579: URL: https://github.com/apache/hive/pull/3579#issuecomment-1239576793 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3579) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3579=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3579=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3579=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3579=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3579=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3579=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 806751) Time Spent: 0.5h (was: 20m) > Iceberg: Raise exception when running delete/update statements on V1 tables > --- > > Key: HIVE-26521 > URL: https://issues.apache.org/jira/browse/HIVE-26521 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Right now an exception is raised on the executor side when trying to commit > the delete file. We should throw an exception earlier, during the compilation > phase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?focusedWorklogId=806749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806749 ] ASF GitHub Bot logged work on HIVE-26512: - Author: ASF GitHub Bot Created on: 07/Sep/22 15:50 Start Date: 07/Sep/22 15:50 Worklog Time Spent: 10m Work Description: csringhofer commented on PR #3580: URL: https://github.com/apache/hive/pull/3580#issuecomment-1239575413 +1 Issue Time Tracking --- Worklog Id: (was: 806749) Time Spent: 20m (was: 10m) > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Sub-task >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26521) Iceberg: Raise exception when running delete/update statements on V1 tables
[ https://issues.apache.org/jira/browse/HIVE-26521?focusedWorklogId=806748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806748 ] ASF GitHub Bot logged work on HIVE-26521: - Author: ASF GitHub Bot Created on: 07/Sep/22 15:49 Start Date: 07/Sep/22 15:49 Worklog Time Spent: 10m Work Description: csringhofer commented on PR #3579: URL: https://github.com/apache/hive/pull/3579#issuecomment-1239574908 +1 Issue Time Tracking --- Worklog Id: (was: 806748) Time Spent: 20m (was: 10m) > Iceberg: Raise exception when running delete/update statements on V1 tables > --- > > Key: HIVE-26521 > URL: https://issues.apache.org/jira/browse/HIVE-26521 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Right now an exception is raised on the executor side when trying to commit > the delete file. We should throw an exception earlier, during the compilation > phase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26522) Metastore DelegationToken renewal is ineffective
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Lanka reassigned HIVE-26522:
--
Assignee: Pavan Lanka
> Metastore DelegationToken renewal is ineffective
>
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
>
> The HMS currently exposes method to renew an obtained delegation token
> {code:java}
> @Override
> public long renewDelegationToken(String tokenStrForm) throws MetaException,
> TException {
> if (localMetaStore) {
> return 0;
> }
> return client.renew_delegation_token(tokenStrForm);
> }{code}
> However on the server side, the renewal of the delegation token does not
> result in the update of the token information with the updated expiry
> {code:java}
> @Override
> public long renewToken(Token token, String
> renewer) throws IOException {
> // since renewal is KERBEROS authenticated token may not be cached
> final DelegationTokenIdentifier id = getTokenIdentifier(token);
> DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
> if (tokenInfo == null) {
> throw new InvalidToken("token does not exist: " + id); // no token found
> }
> // ensure associated master key is available
> if (!super.allKeys.containsKey(id.getMasterKeyId())) {
> LOGGER.info("Unknown master key (id={}), (re)loading keys from token
> store.",
> id.getMasterKeyId());
> reloadKeys();
> }
> // reuse super renewal logic
> synchronized (this) {
> --> super.currentTokens.put(id, tokenInfo);
> try {
> --> return super.renewToken(token, renewer);
> } finally {
> --> super.currentTokens.remove(id);
> }
> }
> } {code}
> Here you can see that we populate the `super.currentTokens` perform the
> renewal and then remove the token without updating the `tokenStore`
>
> As a result of this even though the call for renewal is successful the
> renewal time is not updated for the token and the token is invalidated based
> on the initial expiry time i.e based on when the token was created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26443) Add priority queueing to compaction
[ https://issues.apache.org/jira/browse/HIVE-26443?focusedWorklogId=806724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806724 ] ASF GitHub Bot logged work on HIVE-26443: - Author: ASF GitHub Bot Created on: 07/Sep/22 14:17 Start Date: 07/Sep/22 14:17 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3513: URL: https://github.com/apache/hive/pull/3513#issuecomment-1239453134 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3513) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [33 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3513=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3513=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 806724) Time Spent: 14h 50m (was: 14h 40m) > Add priority queueing to compaction > --- > > Key: HIVE-26443 > URL: https://issues.apache.org/jira/browse/HIVE-26443 > Project: Hive > Issue Type: New Feature >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Attachments: Pool based compaction queues.docx > > Time Spent: 14h 50m > Remaining Estimate: 0h > > The details can be found in the attached design doc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26222) Native GeoSpatial Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-26222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HIVE-26222: --- Assignee: Ayush Saxena (was: mahesh kumar behera) > Native GeoSpatial Support in Hive > - > > Key: HIVE-26222 > URL: https://issues.apache.org/jira/browse/HIVE-26222 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2 >Reporter: mahesh kumar behera >Assignee: Ayush Saxena >Priority: Major > > This is an epic Jira to support GeoSpatial datatypes natively in Hive. This > will cater to the applications which queries on large volumes of spatial > data. This support will be added in a phased manner. To start with, we are > planning to make use of framework developed by ESRI > [https://github.com/Esri/spatial-framework-for-hadoop > |https://github.com/Esri/spatial-framework-for-hadoop).] on the Apache 2.0 > license. This project is not very active and there is no release done to > maven central. So its not easy to get the jars downloaded directly using pom > dependency. Also the UDFs are based on older version of Hive. So we have > decided to make a copy of this repo and maintained it inside Hive. This will > make it easier to do any improvement and manage dependencies. As of now, the > data loading is done only on a binary data type. We need to enhance this to > make it more user friendly. In the next phase, a native Geometry/Geography > datatype will be supported. User can directly create a geometry type and > operate on it. Apart from these we can start adding support for different > indices like quad tree, R-tree, ORC/Parquet/Iceberg support etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26512: Parent: HIVE-26222 Issue Type: Sub-task (was: Bug) > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Sub-task >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26512: -- Labels: pull-request-available (was: ) > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26512) ST_GeometryProcessing is incorrectly registered as function
[ https://issues.apache.org/jira/browse/HIVE-26512?focusedWorklogId=806713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806713 ] ASF GitHub Bot logged work on HIVE-26512: - Author: ASF GitHub Bot Created on: 07/Sep/22 14:00 Start Date: 07/Sep/22 14:00 Worklog Time Spent: 10m Work Description: ayushtkn opened a new pull request, #3580: URL: https://github.com/apache/hive/pull/3580 Remove the unused udf function from being registered, it is just a wrapper class, no need to register and have runtime issues while using it. Issue Time Tracking --- Worklog Id: (was: 806713) Remaining Estimate: 0h Time Spent: 10m > ST_GeometryProcessing is incorrectly registered as function > --- > > Key: HIVE-26512 > URL: https://issues.apache.org/jira/browse/HIVE-26512 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Ayush Saxena >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > ST_GeometryProcessing is registered as function, while it is an empty base > class without evaluate(). > https://github.com/apache/hive/blob/2d8b0a31f461a33ef2e19c0269ec44778d5ca136/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L706 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26521) Iceberg: Raise exception when running delete/update statements on V1 tables
[ https://issues.apache.org/jira/browse/HIVE-26521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26521: -- Labels: pull-request-available (was: ) > Iceberg: Raise exception when running delete/update statements on V1 tables > --- > > Key: HIVE-26521 > URL: https://issues.apache.org/jira/browse/HIVE-26521 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Right now an exception is raised on the executor side when trying to commit > the delete file. We should throw an exception earlier, during the compilation > phase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26521) Iceberg: Raise exception when running delete/update statements on V1 tables
[ https://issues.apache.org/jira/browse/HIVE-26521?focusedWorklogId=806689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806689 ] ASF GitHub Bot logged work on HIVE-26521: - Author: ASF GitHub Bot Created on: 07/Sep/22 12:26 Start Date: 07/Sep/22 12:26 Worklog Time Spent: 10m Work Description: lcspinter opened a new pull request, #3579: URL: https://github.com/apache/hive/pull/3579 ### What changes were proposed in this pull request? We should raise an exception during the compilation phase when trying to run ACID operations. ### Why are the changes needed? An exception is raised during the execution phase, but that is a bit late. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Issue Time Tracking --- Worklog Id: (was: 806689) Remaining Estimate: 0h Time Spent: 10m > Iceberg: Raise exception when running delete/update statements on V1 tables > --- > > Key: HIVE-26521 > URL: https://issues.apache.org/jira/browse/HIVE-26521 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Right now an exception is raised on the executor side when trying to commit > the delete file. We should throw an exception earlier, during the compilation > phase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26521) Iceberg: Raise exception when running delete/update statements on V1 tables
[ https://issues.apache.org/jira/browse/HIVE-26521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér reassigned HIVE-26521: > Iceberg: Raise exception when running delete/update statements on V1 tables > --- > > Key: HIVE-26521 > URL: https://issues.apache.org/jira/browse/HIVE-26521 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > > Right now an exception is raised on the executor side when trying to commit > the delete file. We should throw an exception earlier, during the compilation > phase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601203#comment-17601203
]
Sergey edited comment on HIVE-21778 at 9/7/22 11:20 AM:
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have "set hive.cbo.enable=true".
was (Author: JIRAUSER295463):
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have "set hive.cbo.enable=true".
and i think the fix is not working.
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5, 4.0.0
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, HIVE-21778.4.patch, HIVE-21778.5.patch,
> HIVE-21778.6.patch, HIVE-21778.7.patch, HIVE-21778.8.patch, test_null.q,
> test_null.q.out
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806657
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 11:06
Start Date: 07/Sep/22 11:06
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1239244975
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3552)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[15 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 806657)
Time Spent: 3h 10m (was: 3h)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26437) dump unpartitioned Tables in parallel
[ https://issues.apache.org/jira/browse/HIVE-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26437: -- Labels: pull-request-available (was: ) > dump unpartitioned Tables in parallel > - > > Key: HIVE-26437 > URL: https://issues.apache.org/jira/browse/HIVE-26437 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Amit Saonerkar >Assignee: Amit Saonerkar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26437) dump unpartitioned Tables in parallel
[ https://issues.apache.org/jira/browse/HIVE-26437?focusedWorklogId=806654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806654 ] ASF GitHub Bot logged work on HIVE-26437: - Author: ASF GitHub Bot Created on: 07/Sep/22 11:01 Start Date: 07/Sep/22 11:01 Worklog Time Spent: 10m Work Description: atsaonerk commented on PR #3444: URL: https://github.com/apache/hive/pull/3444#issuecomment-1239240003 Results of jmh performance benchmark test indicates an improvement in TableExport operation. If table is dumped in parallel manner instead of serial , the operation completes much faster giving. Below is the result seen when 500 tablexport operations are done both in serial and parallel manner. Result "org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.parallel": N = 5 mean =640.862 ?(99.9%) 113.354 ms/op Result "org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.serial": N = 5 mean = 51697.322 ?(99.9%) 322.747 ms/op Benchmark Mode Cnt Score Error Units TableAndPartitionExportBench.BaseBench.parallelss 5 640.862? 113.354 ms/op TableAndPartitionExportBench.BaseBench.serial ss 5 51697.322 ? 322.747 ms/op Issue Time Tracking --- Worklog Id: (was: 806654) Remaining Estimate: 0h Time Spent: 10m > dump unpartitioned Tables in parallel > - > > Key: HIVE-26437 > URL: https://issues.apache.org/jira/browse/HIVE-26437 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Amit Saonerkar >Assignee: Amit Saonerkar >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26507) Do not allow hive to iceberg migration if source table contains CHAR or VARCHAR columns
[
https://issues.apache.org/jira/browse/HIVE-26507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Pintér resolved HIVE-26507.
--
Resolution: Fixed
> Do not allow hive to iceberg migration if source table contains CHAR or
> VARCHAR columns
> ---
>
> Key: HIVE-26507
> URL: https://issues.apache.org/jira/browse/HIVE-26507
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: László Pintér
>Priority: Major
> Labels: iceberg, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "alter table" statements can be used for generating iceberg metadata
> information (i.e for converting external tables -> iceberg tables).
> As part of this process, it also converts certain datatypes to iceberg
> compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion"
> enables this conversion.
> This could cause certain issues at runtime. Here is an example
> {noformat}
> Before conversion:
> ==
> -- external table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 27440
> after conversion:
> =
> -- iceberg table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 0
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';
> 27440
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26507) Do not allow hive to iceberg migration if source table contains CHAR or VARCHAR columns
[
https://issues.apache.org/jira/browse/HIVE-26507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601246#comment-17601246
]
László Pintér commented on HIVE-26507:
--
Merged into master. Thanks, [~szita] for the review!
> Do not allow hive to iceberg migration if source table contains CHAR or
> VARCHAR columns
> ---
>
> Key: HIVE-26507
> URL: https://issues.apache.org/jira/browse/HIVE-26507
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: László Pintér
>Priority: Major
> Labels: iceberg, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "alter table" statements can be used for generating iceberg metadata
> information (i.e for converting external tables -> iceberg tables).
> As part of this process, it also converts certain datatypes to iceberg
> compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion"
> enables this conversion.
> This could cause certain issues at runtime. Here is an example
> {noformat}
> Before conversion:
> ==
> -- external table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 27440
> after conversion:
> =
> -- iceberg table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 0
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';
> 27440
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26507) Do not allow hive to iceberg migration if source table contains CHAR or VARCHAR columns
[
https://issues.apache.org/jira/browse/HIVE-26507?focusedWorklogId=806653=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806653
]
ASF GitHub Bot logged work on HIVE-26507:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 10:58
Start Date: 07/Sep/22 10:58
Worklog Time Spent: 10m
Work Description: lcspinter merged PR #3570:
URL: https://github.com/apache/hive/pull/3570
Issue Time Tracking
---
Worklog Id: (was: 806653)
Time Spent: 40m (was: 0.5h)
> Do not allow hive to iceberg migration if source table contains CHAR or
> VARCHAR columns
> ---
>
> Key: HIVE-26507
> URL: https://issues.apache.org/jira/browse/HIVE-26507
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: László Pintér
>Priority: Major
> Labels: iceberg, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "alter table" statements can be used for generating iceberg metadata
> information (i.e for converting external tables -> iceberg tables).
> As part of this process, it also converts certain datatypes to iceberg
> compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion"
> enables this conversion.
> This could cause certain issues at runtime. Here is an example
> {noformat}
> Before conversion:
> ==
> -- external table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 27440
> after conversion:
> =
> -- iceberg table
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and cd_education_status = '2 yr Degree';
> 0
> select count(*) from customer_demographics where cd_gender = 'F' and
> cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';
> 27440
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet edited comment on HIVE-20006 at 9/7/22 10:55 AM: --- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask was (Author: JIRAUSER295079): apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet commented on HIVE-20006: -- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found ``` metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask ``` > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet edited comment on HIVE-20006 at 9/7/22 10:55 AM: --- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask was (Author: JIRAUSER295079): apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found ``` metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask ``` > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet edited comment on HIVE-20006 at 9/7/22 10:55 AM: --- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask was (Author: JIRAUSER295079): apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet edited comment on HIVE-20006 at 9/7/22 10:55 AM: --- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask was (Author: JIRAUSER295079): apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26253) upgrade postgresql to 42.4.1 due to security issues
[ https://issues.apache.org/jira/browse/HIVE-26253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng resolved HIVE-26253. Fix Version/s: 4.0.0 Resolution: Fixed Merged into master. Thank you [~danielzhu] for the contribution and [~ayushtkn] for the review. > upgrade postgresql to 42.4.1 due to security issues > --- > > Key: HIVE-26253 > URL: https://issues.apache.org/jira/browse/HIVE-26253 > Project: Hive > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > issues visible in > [https://mvnrepository.com/artifact/org.postgresql/postgresql] > 42.4.1 might be a good option -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26253) upgrade postgresql to 42.4.1 due to security issues
[ https://issues.apache.org/jira/browse/HIVE-26253?focusedWorklogId=806649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806649 ] ASF GitHub Bot logged work on HIVE-26253: - Author: ASF GitHub Bot Created on: 07/Sep/22 10:34 Start Date: 07/Sep/22 10:34 Worklog Time Spent: 10m Work Description: dengzhhu653 merged PR #3554: URL: https://github.com/apache/hive/pull/3554 Issue Time Tracking --- Worklog Id: (was: 806649) Time Spent: 50m (was: 40m) > upgrade postgresql to 42.4.1 due to security issues > --- > > Key: HIVE-26253 > URL: https://issues.apache.org/jira/browse/HIVE-26253 > Project: Hive > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > issues visible in > [https://mvnrepository.com/artifact/org.postgresql/postgresql] > 42.4.1 might be a good option -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26443) Add priority queueing to compaction
[ https://issues.apache.org/jira/browse/HIVE-26443?focusedWorklogId=806636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806636 ] ASF GitHub Bot logged work on HIVE-26443: - Author: ASF GitHub Bot Created on: 07/Sep/22 09:42 Start Date: 07/Sep/22 09:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3513: URL: https://github.com/apache/hive/pull/3513#issuecomment-1239156972 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3513) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3513=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [31 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3513=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3513=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3513=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 806636) Time Spent: 14h 40m (was: 14.5h) > Add priority queueing to compaction > --- > > Key: HIVE-26443 > URL: https://issues.apache.org/jira/browse/HIVE-26443 > Project: Hive > Issue Type: New Feature >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Attachments: Pool based compaction queues.docx > > Time Spent: 14h 40m > Remaining Estimate: 0h > > The details can be found in the attached design doc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806634
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 09:22
Start Date: 07/Sep/22 09:22
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3552:
URL: https://github.com/apache/hive/pull/3552#discussion_r964602225
##
iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_orc.q:
##
@@ -0,0 +1,35 @@
+
Issue Time Tracking
---
Worklog Id: (was: 806634)
Time Spent: 3h (was: 2h 50m)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601203#comment-17601203
]
Sergey edited comment on HIVE-21778 at 9/7/22 8:53 AM:
---
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have "set hive.cbo.enable=true".
and i think the fix is not working.
was (Author: JIRAUSER295463):
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have "set hive.cbo.enable=true".
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5, 4.0.0
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, HIVE-21778.4.patch, HIVE-21778.5.patch,
> HIVE-21778.6.patch, HIVE-21778.7.patch, HIVE-21778.8.patch, test_null.q,
> test_null.q.out
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601203#comment-17601203
]
Sergey edited comment on HIVE-21778 at 9/7/22 8:46 AM:
---
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have "set hive.cbo.enable=true".
was (Author: JIRAUSER295463):
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have set hive.cbo.enable=true.
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5, 4.0.0
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, HIVE-21778.4.patch, HIVE-21778.5.patch,
> HIVE-21778.6.patch, HIVE-21778.7.patch, HIVE-21778.8.patch, test_null.q,
> test_null.q.out
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26500) Improve TestHiveMetastore
[
https://issues.apache.org/jira/browse/HIVE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-26500.
-
Resolution: Fixed
> Improve TestHiveMetastore
> -
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we
> expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26500) Improve TestHiveMetastore
[
https://issues.apache.org/jira/browse/HIVE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-26500:
Fix Version/s: 4.0.0-alpha-2
> Improve TestHiveMetastore
> -
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we
> expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601203#comment-17601203
]
Sergey commented on HIVE-21778:
---
[ql/src/test/queries/clientpositive/structin.q|https://github.com/apache/hive/pull/928/files#diff-5e07f847d74d58dcf1f07b57748da8b2bab7aa65635fd50594ca72bf0f066371]
- not full test, test no have set hive.cbo.enable=true.
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5, 4.0.0
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, HIVE-21778.4.patch, HIVE-21778.5.patch,
> HIVE-21778.6.patch, HIVE-21778.7.patch, HIVE-21778.8.patch, test_null.q,
> test_null.q.out
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26500) Improve TestHiveMetastore
[
https://issues.apache.org/jira/browse/HIVE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601202#comment-17601202
]
László Bodor commented on HIVE-26500:
-
merged to master, thanks [~ayushtkn]
note: finally I improved single Thread.sleep with a loop, which saves ~10
seconds most of the time too
> Improve TestHiveMetastore
> -
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we
> expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26500) Improve TestHiveMetastore
[
https://issues.apache.org/jira/browse/HIVE-26500?focusedWorklogId=806627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806627
]
ASF GitHub Bot logged work on HIVE-26500:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 08:43
Start Date: 07/Sep/22 08:43
Worklog Time Spent: 10m
Work Description: abstractdog merged PR #3556:
URL: https://github.com/apache/hive/pull/3556
Issue Time Tracking
---
Worklog Id: (was: 806627)
Time Spent: 2h 10m (was: 2h)
> Improve TestHiveMetastore
> -
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we
> expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26500) Improve TestHiveMetastore
[
https://issues.apache.org/jira/browse/HIVE-26500?focusedWorklogId=806626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806626
]
ASF GitHub Bot logged work on HIVE-26500:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 08:42
Start Date: 07/Sep/22 08:42
Worklog Time Spent: 10m
Work Description: abstractdog commented on PR #3556:
URL: https://github.com/apache/hive/pull/3556#issuecomment-1239090642
> LGTM. Sonar has some complains.
https://sonarcloud.io/project/issues?resolved=false=CODE_SMELL=3556=apache_hive=AYMS1ejLkC-HJGhR5irJ
>
> I am not sure how we are handling them..
sonar is absolutely right when saying "Using Thread.sleep in a test is just
generally a bad idea.", that's exactly what I fixed with this, but it hasn't
realized :)
Issue Time Tracking
---
Worklog Id: (was: 806626)
Time Spent: 2h (was: 1h 50m)
> Improve TestHiveMetastore
> -
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we
> expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806598
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 06:37
Start Date: 07/Sep/22 06:37
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1238968835
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3552)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[14 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 806598)
Time Spent: 2h 50m (was: 2h 40m)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild
[
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=806595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-806595
]
ASF GitHub Bot logged work on HIVE-26498:
-
Author: ASF GitHub Bot
Created on: 07/Sep/22 06:11
Start Date: 07/Sep/22 06:11
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3552:
URL: https://github.com/apache/hive/pull/3552#discussion_r964419562
##
storage-api/src/java/org/apache/hadoop/hive/common/MaterializationSnapshot.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.io.UncheckedIOException;
+import java.io.Writer;
+import java.util.Map;
+
+/**
+ * Class to store snapshot data of Materialized view source tables.
+ * The data represents the state of the source tables when the view was
created/last rebuilt.
+ */
+public class MaterializationSnapshot {
+
+ public static MaterializationSnapshot fromJson(String jsonString) {
+try {
+ return new ObjectMapper().readValue(jsonString,
MaterializationSnapshot.class);
+} catch (JsonProcessingException e) {
+ // this is not a jsonString, fall back to treating it as
ValidTxnWriteIdList
+ return new MaterializationSnapshot(jsonString, null);
+}
+ }
+
+ // snapshot of native ACID tables
+ private String validTxnList;
+ // snapshot of non-native ACID tables. Key is the fully qualified name of
the table.
+ // Value is the unique id of the snapshot provided by the table's storage
HiveStorageHandler
+ private Map tableSnapshots;
+
+ private MaterializationSnapshot() {
+ }
+
+ public MaterializationSnapshot(String validTxnList, Map
tableSnapshots) {
+this.validTxnList = validTxnList;
+this.tableSnapshots = tableSnapshots;
+ }
+
+ public String asJsonString() {
Review Comment:
added
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/MaterializedViewMetadata.java:
##
@@ -81,15 +88,20 @@ public Collection getSourceTables() {
return unmodifiableList(creationMetadata.getSourceTables());
}
- public String getValidTxnList() {
-return creationMetadata.getValidTxnList();
+ public MaterializationSnapshot getSnapshot() {
Review Comment:
added
Issue Time Tracking
---
Worklog Id: (was: 806595)
Time Spent: 2h 40m (was: 2.5h)
> Implement MV maintenance with Iceberg sources using full rebuild
>
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
> HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
