[jira] [Work logged] (HIVE-26811) SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans
[ https://issues.apache.org/jira/browse/HIVE-26811?focusedWorklogId=831951&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831951 ] ASF GitHub Bot logged work on HIVE-26811: - Author: ASF GitHub Bot Created on: 08/Dec/22 07:35 Start Date: 08/Dec/22 07:35 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3837: URL: https://github.com/apache/hive/pull/3837#discussion_r1043015655 ## iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_merge_schema.q.out: ## @@ -0,0 +1,163 @@ +PREHOOK: query: CREATE TABLE calls ( Review Comment: fixed Issue Time Tracking --- Worklog Id: (was: 831951) Time Spent: 1h (was: 50m) > SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans > -- > > Key: HIVE-26811 > URL: https://issues.apache.org/jira/browse/HIVE-26811 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?focusedWorklogId=831940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831940
]
ASF GitHub Bot logged work on HIVE-26578:
-
Author: ASF GitHub Bot
Created on: 08/Dec/22 04:41
Start Date: 08/Dec/22 04:41
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3823:
URL: https://github.com/apache/hive/pull/3823#issuecomment-1342007433
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3823)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3823&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3823&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831940)
Time Spent: 1h 40m (was: 1.5h)
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26799) Make authorizations on custom UDFs involved in tables/view configurable.
[
https://issues.apache.org/jira/browse/HIVE-26799?focusedWorklogId=831936&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831936
]
ASF GitHub Bot logged work on HIVE-26799:
-
Author: ASF GitHub Bot
Created on: 08/Dec/22 03:51
Start Date: 08/Dec/22 03:51
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3821:
URL: https://github.com/apache/hive/pull/3821#issuecomment-1341953553
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3821)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3821&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3821&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831936)
Time Spent: 2h 20m (was: 2h 10m)
> Make authorizations on custom UDFs involved in tables/view configurable.
>
>
> Key: HIVE-26799
> URL: https://issues.apache.org/jira/browse/HIVE-26799
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2, Security
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When Hive is using Ranger/Sentry as an authorization service, consider the
> following scenario.
> {code:java}
> > create table test_udf(st string); // privileged user operation
> > create function Udf_UPPER as 'openkb.hive.udf.MyUpper' using jar
> > 'hdfs:///tmp/MyUpperUDF-1.0.0.jar'; // privileged user operation
> > create view v1_udf as select udf_upper(st) from test_udf; // privileged
> > user operation
> //unprivileged user test_user is given select per
[jira] [Work logged] (HIVE-23559) Optimise Hive::moveAcidFiles for cloud storage
[ https://issues.apache.org/jira/browse/HIVE-23559?focusedWorklogId=831934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831934 ] ASF GitHub Bot logged work on HIVE-23559: - Author: ASF GitHub Bot Created on: 08/Dec/22 02:44 Start Date: 08/Dec/22 02:44 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3795: URL: https://github.com/apache/hive/pull/3795#issuecomment-1341903633 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3795) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3795&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3795&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3795&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3795&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3795&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3795&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831934) Time Spent: 1h 40m (was: 1.5h) > Optimise Hive::moveAcidFiles for cloud storage > -- > > Key: HIVE-23559 > URL: https://issues.apache.org/jira/browse/HIVE-23559 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4752] > It ends up transferring DELTA, DELETE_DELTA, BASE prefixes sequentially from > staging to final location. > This causes delays even with simple updates statements, which updates smaller > number of records in cloud storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644572#comment-17644572
]
tanghui commented on HIVE-26158:
[~LiPenglin] Yes, if your table is an external table not a
TRANSLATED_TO_EXTERNAL table , the previous rules will be maintained when
rename table, that is, the storage path of the data will not be modified。
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[ https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831928 ] ASF GitHub Bot logged work on HIVE-26569: - Author: ASF GitHub Bot Created on: 08/Dec/22 01:45 Start Date: 08/Dec/22 01:45 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3626: URL: https://github.com/apache/hive/pull/3626#issuecomment-1341855377 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3626) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [19 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3626&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3626&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831928) Time Spent: 5h 10m (was: 5h) > Support renewal and recreation of LLAP_TOKENs > - > > Key: HIVE-26569 > URL: https://issues.apache.org/jira/browse/HIVE-26569 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26790) Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1
[ https://issues.apache.org/jira/browse/HIVE-26790?focusedWorklogId=831919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831919 ] ASF GitHub Bot logged work on HIVE-26790: - Author: ASF GitHub Bot Created on: 08/Dec/22 00:59 Start Date: 08/Dec/22 00:59 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3814: URL: https://github.com/apache/hive/pull/3814#issuecomment-1341821847 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3814) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3814&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3814&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831919) Time Spent: 1h (was: 50m) > Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1 > --- > > Key: HIVE-26790 > URL: https://issues.apache.org/jira/browse/HIVE-26790 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Whilst trying to build for Apple M1, bumped into grpc (1.24.0) missing > binaries for Apple M1. > Updating to grpc 1.51.0 and protobuf 3.21.7 solves the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?focusedWorklogId=831918&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831918 ] ASF GitHub Bot logged work on HIVE-26813: - Author: ASF GitHub Bot Created on: 08/Dec/22 00:29 Start Date: 08/Dec/22 00:29 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3839: URL: https://github.com/apache/hive/pull/3839#issuecomment-1341798987 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3839) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3839&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3839&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3839&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3839&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3839&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3839&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831918) Time Spent: 1h 10m (was: 1h) > Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HIVE-26815. - Fix Version/s: 3.2.0 Hadoop Flags: Reviewed Resolution: Fixed > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.3 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26809) Upgrade ORC to 1.8.0
[ https://issues.apache.org/jira/browse/HIVE-26809?focusedWorklogId=831910&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831910 ] ASF GitHub Bot logged work on HIVE-26809: - Author: ASF GitHub Bot Created on: 07/Dec/22 22:53 Start Date: 07/Dec/22 22:53 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3833: URL: https://github.com/apache/hive/pull/3833#issuecomment-1341703476 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3833) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3833&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3833&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3833&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3833&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3833&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3833&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831910) Time Spent: 1h (was: 50m) > Upgrade ORC to 1.8.0 > > > Key: HIVE-26809 > URL: https://issues.apache.org/jira/browse/HIVE-26809 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Dmitriy Fingerman >Assignee: Dmitriy Fingerman >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-19457) Support Iceberg as a table format
[
https://issues.apache.org/jira/browse/HIVE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-19457.
-
Resolution: Duplicate
> Support Iceberg as a table format
> -
>
> Key: HIVE-19457
> URL: https://issues.apache.org/jira/browse/HIVE-19457
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, StorageHandler
>Reporter: Elliot West
>Priority: Major
>
> From [Netflix/iceberg|https://github.com/Netflix/iceberg]:
> {quote}
> Iceberg is a new table format for storing large, slow-moving tabular data. It
> is designed to improve on the de-facto standard table layout built into Hive,
> Presto, and Spark.
> {quote}
> Iceberg elegantly solves many data and metadata challenges when handling
> large datasets, especially when using eventually consistent file stores. By
> integrating iceberg with Hive we could hope to gain the following:
> * Improved performance when reading large numbers of partitions (HMS is no
> longer a bottleneck).
> * Users receiving a consistent view when reading changing datasets (snapshot
> isolation).
> * Strong consistency on eventually consistent file stores without the need
> for additional infrastructure (data is immutable, eventually consistent ops
> are avoided; data is always {{PUT}} and {{LIST}} is avoided).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HIVE-19457) Support Iceberg as a table format
[
https://issues.apache.org/jira/browse/HIVE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reopened HIVE-19457:
-
> Support Iceberg as a table format
> -
>
> Key: HIVE-19457
> URL: https://issues.apache.org/jira/browse/HIVE-19457
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, StorageHandler
>Reporter: Elliot West
>Priority: Major
>
> From [Netflix/iceberg|https://github.com/Netflix/iceberg]:
> {quote}
> Iceberg is a new table format for storing large, slow-moving tabular data. It
> is designed to improve on the de-facto standard table layout built into Hive,
> Presto, and Spark.
> {quote}
> Iceberg elegantly solves many data and metadata challenges when handling
> large datasets, especially when using eventually consistent file stores. By
> integrating iceberg with Hive we could hope to gain the following:
> * Improved performance when reading large numbers of partitions (HMS is no
> longer a bottleneck).
> * Users receiving a consistent view when reading changing datasets (snapshot
> isolation).
> * Strong consistency on eventually consistent file stores without the need
> for additional infrastructure (data is immutable, eventually consistent ops
> are avoided; data is always {{PUT}} and {{LIST}} is avoided).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-19457) Support Iceberg as a table format
[
https://issues.apache.org/jira/browse/HIVE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-19457.
-
Resolution: Fixed
> Support Iceberg as a table format
> -
>
> Key: HIVE-19457
> URL: https://issues.apache.org/jira/browse/HIVE-19457
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, StorageHandler
>Reporter: Elliot West
>Priority: Major
>
> From [Netflix/iceberg|https://github.com/Netflix/iceberg]:
> {quote}
> Iceberg is a new table format for storing large, slow-moving tabular data. It
> is designed to improve on the de-facto standard table layout built into Hive,
> Presto, and Spark.
> {quote}
> Iceberg elegantly solves many data and metadata challenges when handling
> large datasets, especially when using eventually consistent file stores. By
> integrating iceberg with Hive we could hope to gain the following:
> * Improved performance when reading large numbers of partitions (HMS is no
> longer a bottleneck).
> * Users receiving a consistent view when reading changing datasets (snapshot
> isolation).
> * Strong consistency on eventually consistent file stores without the need
> for additional infrastructure (data is immutable, eventually consistent ops
> are avoided; data is always {{PUT}} and {{LIST}} is avoided).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-19457) Support Iceberg as a table format
[
https://issues.apache.org/jira/browse/HIVE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644528#comment-17644528
]
László Bodor commented on HIVE-19457:
-
hive today supports iceberg, I cannot really link to one specific ticket, but I
have to, I might be HIVE-24867
let me close this one
> Support Iceberg as a table format
> -
>
> Key: HIVE-19457
> URL: https://issues.apache.org/jira/browse/HIVE-19457
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, StorageHandler
>Reporter: Elliot West
>Priority: Major
>
> From [Netflix/iceberg|https://github.com/Netflix/iceberg]:
> {quote}
> Iceberg is a new table format for storing large, slow-moving tabular data. It
> is designed to improve on the de-facto standard table layout built into Hive,
> Presto, and Spark.
> {quote}
> Iceberg elegantly solves many data and metadata challenges when handling
> large datasets, especially when using eventually consistent file stores. By
> integrating iceberg with Hive we could hope to gain the following:
> * Improved performance when reading large numbers of partitions (HMS is no
> longer a bottleneck).
> * Users receiving a consistent view when reading changing datasets (snapshot
> isolation).
> * Strong consistency on eventually consistent file stores without the need
> for additional infrastructure (data is immutable, eventually consistent ops
> are avoided; data is always {{PUT}} and {{LIST}} is avoided).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-19457) Support Iceberg as a table format
[
https://issues.apache.org/jira/browse/HIVE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644528#comment-17644528
]
László Bodor edited comment on HIVE-19457 at 12/7/22 10:00 PM:
---
thanks for reporting this [~teabot], hive today supports iceberg, I cannot
really link to one specific ticket, but I have to, it might be HIVE-24867
let me close this one
was (Author: abstractdog):
hive today supports iceberg, I cannot really link to one specific ticket, but I
have to, I might be HIVE-24867
let me close this one
> Support Iceberg as a table format
> -
>
> Key: HIVE-19457
> URL: https://issues.apache.org/jira/browse/HIVE-19457
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, StorageHandler
>Reporter: Elliot West
>Priority: Major
>
> From [Netflix/iceberg|https://github.com/Netflix/iceberg]:
> {quote}
> Iceberg is a new table format for storing large, slow-moving tabular data. It
> is designed to improve on the de-facto standard table layout built into Hive,
> Presto, and Spark.
> {quote}
> Iceberg elegantly solves many data and metadata challenges when handling
> large datasets, especially when using eventually consistent file stores. By
> integrating iceberg with Hive we could hope to gain the following:
> * Improved performance when reading large numbers of partitions (HMS is no
> longer a bottleneck).
> * Users receiving a consistent view when reading changing datasets (snapshot
> isolation).
> * Strong consistency on eventually consistent file stores without the need
> for additional infrastructure (data is immutable, eventually consistent ops
> are avoided; data is always {{PUT}} and {{LIST}} is avoided).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[ https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831898&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831898 ] ASF GitHub Bot logged work on HIVE-26569: - Author: ASF GitHub Bot Created on: 07/Dec/22 21:34 Start Date: 07/Dec/22 21:34 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3626: URL: https://github.com/apache/hive/pull/3626#issuecomment-1341625960 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3626) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3626&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [29 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3626&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3626&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3626&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831898) Time Spent: 5h (was: 4h 50m) > Support renewal and recreation of LLAP_TOKENs > - > > Key: HIVE-26569 > URL: https://issues.apache.org/jira/browse/HIVE-26569 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26799) Make authorizations on custom UDFs involved in tables/view configurable.
[
https://issues.apache.org/jira/browse/HIVE-26799?focusedWorklogId=831894&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831894
]
ASF GitHub Bot logged work on HIVE-26799:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 21:25
Start Date: 07/Dec/22 21:25
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3821:
URL: https://github.com/apache/hive/pull/3821#issuecomment-1341617295
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3821)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3821&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[9 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3821&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3821&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3821&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831894)
Time Spent: 2h 10m (was: 2h)
> Make authorizations on custom UDFs involved in tables/view configurable.
>
>
> Key: HIVE-26799
> URL: https://issues.apache.org/jira/browse/HIVE-26799
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2, Security
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> When Hive is using Ranger/Sentry as an authorization service, consider the
> following scenario.
> {code:java}
> > create table test_udf(st string); // privileged user operation
> > create function Udf_UPPER as 'openkb.hive.udf.MyUpper' using jar
> > 'hdfs:///tmp/MyUpperUDF-1.0.0.jar'; // privileged user operation
> > create view v1_udf as select udf_upper(st) from test_udf; // privileged
> > user operation
> //unprivileged user test_user is given select permis
[jira] [Work logged] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?focusedWorklogId=831893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831893 ] ASF GitHub Bot logged work on HIVE-26815: - Author: ASF GitHub Bot Created on: 07/Dec/22 21:23 Start Date: 07/Dec/22 21:23 Worklog Time Spent: 10m Work Description: sunchao commented on PR #3840: URL: https://github.com/apache/hive/pull/3840#issuecomment-1341615164 Merged, thanks @yigress ! Issue Time Tracking --- Worklog Id: (was: 831893) Time Spent: 0.5h (was: 20m) > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.3 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?focusedWorklogId=831892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831892 ] ASF GitHub Bot logged work on HIVE-26815: - Author: ASF GitHub Bot Created on: 07/Dec/22 21:23 Start Date: 07/Dec/22 21:23 Worklog Time Spent: 10m Work Description: sunchao merged PR #3840: URL: https://github.com/apache/hive/pull/3840 Issue Time Tracking --- Worklog Id: (was: 831892) Time Spent: 20m (was: 10m) > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.3 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhang reassigned HIVE-26815: --- Assignee: Yi Zhang > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.3 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26819) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26819?focusedWorklogId=831878&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831878
]
ASF GitHub Bot logged work on HIVE-26819:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 20:41
Start Date: 07/Dec/22 20:41
Worklog Time Spent: 10m
Work Description: yigress opened a new pull request, #3844:
URL: https://github.com/apache/hive/pull/3844
### What changes were proposed in this pull request?
fix data correctness bug when query on map in orc table in vectorization
mode.
### Why are the changes needed?
to fix data correctness bug
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested with qtest and locally
Issue Time Tracking
---
Worklog Id: (was: 831878)
Remaining Estimate: 0h
Time Spent: 10m
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26819
> URL: https://issues.apache.org/jira/browse/HIVE-26819
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.3
>Reporter: Yi Zhang
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
> Example reproducible case:
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26819) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang reassigned HIVE-26819:
---
Assignee: Yi Zhang
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26819
> URL: https://issues.apache.org/jira/browse/HIVE-26819
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.3
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
> Example reproducible case:
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26819) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26819:
--
Labels: pull-request-available (was: )
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26819
> URL: https://issues.apache.org/jira/browse/HIVE-26819
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.3
>Reporter: Yi Zhang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
> Example reproducible case:
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26447) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang updated HIVE-26447:
Affects Version/s: (was: 3.1.3)
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26447
> URL: https://issues.apache.org/jira/browse/HIVE-26447
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Example reproducible case:
>
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[
https://issues.apache.org/jira/browse/HIVE-26813?focusedWorklogId=831873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831873
]
ASF GitHub Bot logged work on HIVE-26813:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 20:18
Start Date: 07/Dec/22 20:18
Worklog Time Spent: 10m
Work Description: cnauroth commented on code in PR #3839:
URL: https://github.com/apache/hive/pull/3839#discussion_r1042642131
##
ql/pom.xml:
##
@@ -1113,6 +1113,12 @@
META-INF/licenses/slf4j*/**
+
+ com.zaxxer:HikariCP
+
+META-INF/versions/11/module-info.class
+
+
Review Comment:
The new version of HikariCP starts declaring itself as a Jigsaw module. When
shading this dependency, the default behavior is to bring in all of the files,
including the compiled module declaration. With that, hive-exec.jar would be
declaring itself as the HikariCP module, which wouldn't be correct:
```
> jar xf ~/git/apache/hive/ql/target/hive-exec-4.0.0-SNAPSHOT.jar
> javap -l -p -c -s META-INF/versions/11/module-info.class
Compiled from "module-info.java"
module com.zaxxer.hikari@4.0.3 {
requires java.base;
requires java.sql;
requires java.management;
requires java.naming;
requires org.slf4j;
requires static org.hibernate.orm.core;
requires static simpleclient;
requires static metrics.core;
requires static metrics.healthchecks;
requires static micrometer.core;
exports com.zaxxer.hikari;
exports com.zaxxer.hikari.hibernate;
exports com.zaxxer.hikari.metrics;
exports com.zaxxer.hikari.metrics.dropwizard;
exports com.zaxxer.hikari.metrics.micrometer;
exports com.zaxxer.hikari.metrics.prometheus;
exports com.zaxxer.hikari.pool;
exports com.zaxxer.hikari.util;
}
```
Hadoop has needed to apply similar filter rules on some of the dependencies
it shades:
https://github.com/apache/hadoop/blob/rel/release-3.3.4/hadoop-client-modules/hadoop-client-runtime/pom.xml#L245-L250
Hmmm... on a side note, Hive shades Jackson too, so I wonder if we have an
existing problem on that one. I'll follow up on that later.
In terms of practical problems, I think a conflict could only arise if
running hive-exec.jar in a JVM with module support (Java 9+) and actually
attempting to use module resolution. That's an edge case for sure, but I think
it's best to omit the incorrect module-info.class.
While reviewing this, I also realized that the same problem applies to a few
of our other shaded modules: beeline and jdbc. I've updated the patch to add
the filter rules there too.
Issue Time Tracking
---
Worklog Id: (was: 831873)
Time Spent: 1h (was: 50m)
> Upgrade HikariCP from 2.6.1 to 4.0.3.
> -
>
> Key: HIVE-26813
> URL: https://issues.apache.org/jira/browse/HIVE-26813
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Hive Metastore currently integrates with HikariCP 2.6.1 for database
> connection pooling. This version was released in 2017. The most recent Java
> 8-compatible release is 4.0.3, released earlier this year. This bug proposes
> to upgrade so that we can include the past few years of development and bug
> fixes in the 4.0.0 GA release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?focusedWorklogId=831872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831872 ] ASF GitHub Bot logged work on HIVE-26813: - Author: ASF GitHub Bot Created on: 07/Dec/22 20:17 Start Date: 07/Dec/22 20:17 Worklog Time Spent: 10m Work Description: cnauroth commented on code in PR #3839: URL: https://github.com/apache/hive/pull/3839#discussion_r1042641356 ## pom.xml: ## @@ -142,7 +142,7 @@ 0.7.2 Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26819) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang updated HIVE-26819:
Description:
Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
Example reproducible case:
set hive.vectorized.execution.enabled=true;
set hive.fetch.task.conversion=none;
create temporary table foo (id int, x map) stored as orc;
insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
8)), (4, map('ABC', 9));
select id from foo where x['ABC']=9;
this only gives 1, when correct result should be 1,4
For every VectorizedRowBatch, only the first row is checked.
This seems to be a corner case of ORC table have repeating string type key for
map field in the MapColumnVector.
was:Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26819
> URL: https://issues.apache.org/jira/browse/HIVE-26819
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.3
>Reporter: Yi Zhang
>Priority: Minor
>
> Same issue is fixed by HIVE-26447, this is to fix in 3.2.0.
> Example reproducible case:
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26817) Set column names in result schema when plan has Values root
[
https://issues.apache.org/jira/browse/HIVE-26817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26817:
--
Labels: pull-request-available (was: )
> Set column names in result schema when plan has Values root
> ---
>
> Key: HIVE-26817
> URL: https://issues.apache.org/jira/browse/HIVE-26817
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The query
> {code}
> select b1, count(a1) count1 from (select a1, b1 from t1) s where 1=0 group by
> b1;
> {code}
> should have a result with column names
> {code}
> b1count1
> {code}
> but it is
> {code}
> $f0 $f1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26817) Set column names in result schema when plan has Values root
[
https://issues.apache.org/jira/browse/HIVE-26817?focusedWorklogId=831871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831871
]
ASF GitHub Bot logged work on HIVE-26817:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 19:52
Start Date: 07/Dec/22 19:52
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3843:
URL: https://github.com/apache/hive/pull/3843#issuecomment-1341509496
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3843)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3843&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3843&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3843&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3843&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3843&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3843&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831871)
Remaining Estimate: 0h
Time Spent: 10m
> Set column names in result schema when plan has Values root
> ---
>
> Key: HIVE-26817
> URL: https://issues.apache.org/jira/browse/HIVE-26817
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The query
> {code}
> select b1, count(a1) count1 from (select a1, b1 from t1) s where 1=0 group by
> b1;
> {code}
> should have a result with column names
> {code}
> b1count1
> {code}
> but it is
> {code}
> $f0 $f1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhang updated HIVE-26815: Affects Version/s: 3.1.3 (was: 3.2.0) > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.3 >Reporter: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26447) Vectorization: wrong results when filter on repeating map key orc table
[
https://issues.apache.org/jira/browse/HIVE-26447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang updated HIVE-26447:
Affects Version/s: 4.0.0-alpha-1
(was: 4.0.0)
> Vectorization: wrong results when filter on repeating map key orc table
> ---
>
> Key: HIVE-26447
> URL: https://issues.apache.org/jira/browse/HIVE-26447
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.3, 4.0.0-alpha-1
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Example reproducible case:
>
> set hive.vectorized.execution.enabled=true;
> set hive.fetch.task.conversion=none;
> create temporary table foo (id int, x map) stored as orc;
> insert into foo values(1, map('ABC', 9)), (2, map('ABC', 7)), (3, map('ABC',
> 8)), (4, map('ABC', 9));
> select id from foo where x['ABC']=9;
> this only gives 1, when correct result should be 1,4
> For every VectorizedRowBatch, only the first row is checked.
> This seems to be a corner case of ORC table have repeating string type key
> for map field in the MapColumnVector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?focusedWorklogId=831850&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831850 ] ASF GitHub Bot logged work on HIVE-26813: - Author: ASF GitHub Bot Created on: 07/Dec/22 18:45 Start Date: 07/Dec/22 18:45 Worklog Time Spent: 10m Work Description: zabetak commented on code in PR #3839: URL: https://github.com/apache/hive/pull/3839#discussion_r1042560998 ## pom.xml: ## @@ -142,7 +142,7 @@ 0.7.2 Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26810) Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-26810. Fix Version/s: 4.0.0 Resolution: Fixed Fixed in https://github.com/apache/hive/commit/ebf05894edf80fae6280b656d3718685c930401d. Thanks for the PR [~asolimando]! > Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in > implementation > -- > > Key: HIVE-26810 > URL: https://issues.apache.org/jira/browse/HIVE-26810 > Project: Hive > Issue Type: Task > Components: CBO >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > After HIVE-26762, the _onMatch_ method is now the same as in the Calcite > implementation, we can drop the Hive's override in order to avoid the risk of > them drifting away again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26810) Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26810?focusedWorklogId=831846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831846 ] ASF GitHub Bot logged work on HIVE-26810: - Author: ASF GitHub Bot Created on: 07/Dec/22 18:40 Start Date: 07/Dec/22 18:40 Worklog Time Spent: 10m Work Description: zabetak closed pull request #3842: HIVE-26810: Replace HiveFilterSetOpTransposeRule onMatch method with … URL: https://github.com/apache/hive/pull/3842 Issue Time Tracking --- Worklog Id: (was: 831846) Time Spent: 0.5h (was: 20m) > Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in > implementation > -- > > Key: HIVE-26810 > URL: https://issues.apache.org/jira/browse/HIVE-26810 > Project: Hive > Issue Type: Task > Components: CBO >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > After HIVE-26762, the _onMatch_ method is now the same as in the Calcite > implementation, we can drop the Hive's override in order to avoid the risk of > them drifting away again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?focusedWorklogId=831835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831835
]
ASF GitHub Bot logged work on HIVE-26578:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 18:20
Start Date: 07/Dec/22 18:20
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3823:
URL: https://github.com/apache/hive/pull/3823#issuecomment-1341385005
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3823)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3823&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3823&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3823&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3823&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831835)
Time Spent: 1.5h (was: 1h 20m)
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26221) Add histogram-based column statistics
[ https://issues.apache.org/jira/browse/HIVE-26221?focusedWorklogId=831833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831833 ] ASF GitHub Bot logged work on HIVE-26221: - Author: ASF GitHub Bot Created on: 07/Dec/22 18:13 Start Date: 07/Dec/22 18:13 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3137: URL: https://github.com/apache/hive/pull/3137#issuecomment-1341375676 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3137) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [38 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3137&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3137&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831833) Time Spent: 5.5h (was: 5h 20m) > Add histogram-based column statistics > - > > Key: HIVE-26221 > URL: https://issues.apache.org/jira/browse/HIVE-26221 > Project: Hive > Issue Type: Improvement > Components: CBO, Metastore, Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 5.5h > Remaining Estimate: 0h > > Hive does not support histogram statistics, which are particularly useful for > skewed data (which is very common in practice) and range predicates. > Hive's current selectivity estimation for range predicates is based on a > hard-coded value of 1/3 (see > [FilterSelectivityEstimator.java#L138-L144|https://github.com/apache/hive/blob/56c336268ea8c281d23c22d89271af37cb7e2572/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java#L138-L144]).]) > The current proposal aims at integrating histo
[jira] [Commented] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644450#comment-17644450 ] Dagang Wei commented on HIVE-26813: --- +1 > Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26785) Remove explicit protobuf-java dependency from blobstore and minikdc modules
[ https://issues.apache.org/jira/browse/HIVE-26785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-26785. Fix Version/s: 4.0.0 Resolution: Fixed Fixed in https://github.com/apache/hive/commit/b47798e56f14321db251652fa1c43887282bbf9b. Thanks for the reviews [~ayushsaxena] and [~cnauroth] (didn't see your +1 in time to give credits in the commit message). > Remove explicit protobuf-java dependency from blobstore and minikdc modules > --- > > Key: HIVE-26785 > URL: https://issues.apache.org/jira/browse/HIVE-26785 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The modules do not directly need protobuf dependency so it is misleading to > declare it explicitly. > Moreover, these modules use a different protobuf version (3.3.0) than the > rest of the project (3.21.4) which can lead to compatibility problems, > inconsistent behavior in tests, and undesired transitive propagation to other > modules. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26812) Compile error in hive-it-util module when itests are built separately
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1767#comment-1767
]
Stamatis Zampetakis commented on HIVE-26812:
[~asolimando] It works when compiling from the root directory cause somehow the
original hive-beeline pom file is used (and not the dependency-reduced.xml).
When compiling separately the only place where we can find the pom is in ~/.m2
and there we have the reduced one.
Try:
{noformat}
hive$ mvn dependency:tree -Pitests
hive/itests$ mvn dependency:tree
{noformat}
and check the dependencies of hive-it-util module. In the first case the
transitive deps appear below hive-beeline while in the second case they don't.
> Compile error in hive-it-util module when itests are built separately
> -
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26812) Compile error in hive-it-util module when itests are built separately
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-26812.
Fix Version/s: 4.0.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/6bba5de9fcff5c81e203f6202c72a9f6b8c388ae.
Thanks for the PR [~asolimando] and [~cnauroth], [~ayushsaxena] for the reviews.
> Compile error in hive-it-util module when itests are built separately
> -
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26812) Compile error in hive-it-util module when itests are built separately
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-26812:
---
Summary: Compile error in hive-it-util module when itests are built
separately (was: hive-it-util module misses a dependency on hive-jdbc)
> Compile error in hive-it-util module when itests are built separately
> -
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26785) Remove explicit protobuf-java dependency from blobstore and minikdc modules
[ https://issues.apache.org/jira/browse/HIVE-26785?focusedWorklogId=831819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831819 ] ASF GitHub Bot logged work on HIVE-26785: - Author: ASF GitHub Bot Created on: 07/Dec/22 17:34 Start Date: 07/Dec/22 17:34 Worklog Time Spent: 10m Work Description: zabetak closed pull request #3809: HIVE-26785: Remove explicit protobuf-java dependency from blobstore and minikdc modules URL: https://github.com/apache/hive/pull/3809 Issue Time Tracking --- Worklog Id: (was: 831819) Time Spent: 20m (was: 10m) > Remove explicit protobuf-java dependency from blobstore and minikdc modules > --- > > Key: HIVE-26785 > URL: https://issues.apache.org/jira/browse/HIVE-26785 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The modules do not directly need protobuf dependency so it is misleading to > declare it explicitly. > Moreover, these modules use a different protobuf version (3.3.0) than the > rest of the project (3.21.4) which can lead to compatibility problems, > inconsistent behavior in tests, and undesired transitive propagation to other > modules. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26812) hive-it-util module misses a dependency on hive-jdbc
[jira] [Work logged] (HIVE-26811) SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans
[ https://issues.apache.org/jira/browse/HIVE-26811?focusedWorklogId=831806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831806 ] ASF GitHub Bot logged work on HIVE-26811: - Author: ASF GitHub Bot Created on: 07/Dec/22 16:11 Start Date: 07/Dec/22 16:11 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3837: URL: https://github.com/apache/hive/pull/3837#issuecomment-1341209227 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3837) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3837&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3837&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831806) Time Spent: 50m (was: 40m) > SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans > -- > > Key: HIVE-26811 > URL: https://issues.apache.org/jira/browse/HIVE-26811 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?focusedWorklogId=831801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831801
]
ASF GitHub Bot logged work on HIVE-26578:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 15:40
Start Date: 07/Dec/22 15:40
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3823:
URL: https://github.com/apache/hive/pull/3823#discussion_r1042367508
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/create/CreateMaterializedViewDesc.java:
##
@@ -237,11 +237,17 @@ public void setTblProps(Map tblProps) {
this.tblProps = tblProps;
}
- @Explain(displayName = "table properties")
public Map getTblProps() {
return tblProps;
}
+ @Explain(displayName = "table properties")
+ public Map getTblPropsExplain() { // only for displaying plan
+return PlanUtils.getPropertiesForExplain(tblProps,
+hive_metastoreConstants.TABLE_IS_CTAS,
Review Comment:
Renamed the constant but kept the value since it would break compatibility.
Issue Time Tracking
---
Worklog Id: (was: 831801)
Time Spent: 1h 20m (was: 1h 10m)
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26817) Set column names in result schema when plan has Values root
[
https://issues.apache.org/jira/browse/HIVE-26817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-26817:
-
> Set column names in result schema when plan has Values root
> ---
>
> Key: HIVE-26817
> URL: https://issues.apache.org/jira/browse/HIVE-26817
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>
> The query
> {code}
> select b1, count(a1) count1 from (select a1, b1 from t1) s where 1=0 group by
> b1;
> {code}
> should have a result with column names
> {code}
> b1count1
> {code}
> but it is
> {code}
> $f0 $f1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26790) Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1
[ https://issues.apache.org/jira/browse/HIVE-26790?focusedWorklogId=831790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831790 ] ASF GitHub Bot logged work on HIVE-26790: - Author: ASF GitHub Bot Created on: 07/Dec/22 14:54 Start Date: 07/Dec/22 14:54 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3814: URL: https://github.com/apache/hive/pull/3814#issuecomment-1341084109 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3814) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3814&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3814&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3814&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3814&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831790) Time Spent: 50m (was: 40m) > Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1 > --- > > Key: HIVE-26790 > URL: https://issues.apache.org/jira/browse/HIVE-26790 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Whilst trying to build for Apple M1, bumped into grpc (1.24.0) missing > binaries for Apple M1. > Updating to grpc 1.51.0 and protobuf 3.21.7 solves the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26771) Use DDLTask to created Iceberg table when running ctas statement
[ https://issues.apache.org/jira/browse/HIVE-26771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa resolved HIVE-26771. --- Resolution: Fixed Merged to master. Thanks [~dkuzmenko] for review. > Use DDLTask to created Iceberg table when running ctas statement > > > Key: HIVE-26771 > URL: https://issues.apache.org/jira/browse/HIVE-26771 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > When Iceberg table is created via ctas statement the table is created in > HiveIcebergSerDe and no DDL task is executed. > Negative effects of this workflow: > * Missing table properties: format-version, write.[delete|update|merge].mode > * Default privileges of the new table are not granted. > * The new Iceberg table can be seen by other transactions at compile time of > ctas. > * Table creation and table properties are not shown in explain ctas output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26790) Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1
[ https://issues.apache.org/jira/browse/HIVE-26790?focusedWorklogId=831783&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831783 ] ASF GitHub Bot logged work on HIVE-26790: - Author: ASF GitHub Bot Created on: 07/Dec/22 14:37 Start Date: 07/Dec/22 14:37 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on PR #3814: URL: https://github.com/apache/hive/pull/3814#issuecomment-1341060336 Hello @henrib, cloud you please rebase you changes on master? that would get a much quicker run. Issue Time Tracking --- Worklog Id: (was: 831783) Time Spent: 40m (was: 0.5h) > Upgrade to protobuf 3.21.7 / grpc 1.51.0; build on Apple M1 > --- > > Key: HIVE-26790 > URL: https://issues.apache.org/jira/browse/HIVE-26790 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Whilst trying to build for Apple M1, bumped into grpc (1.24.0) missing > binaries for Apple M1. > Updating to grpc 1.51.0 and protobuf 3.21.7 solves the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26810) Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26810?focusedWorklogId=831781&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831781 ] ASF GitHub Bot logged work on HIVE-26810: - Author: ASF GitHub Bot Created on: 07/Dec/22 14:29 Start Date: 07/Dec/22 14:29 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3842: URL: https://github.com/apache/hive/pull/3842#issuecomment-1341049577 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3842) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3842&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3842&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3842&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=CODE_SMELL) [17 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3842&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3842&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3842&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831781) Time Spent: 20m (was: 10m) > Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in > implementation > -- > > Key: HIVE-26810 > URL: https://issues.apache.org/jira/browse/HIVE-26810 > Project: Hive > Issue Type: Task > Components: CBO >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > After HIVE-26762, the _onMatch_ method is now the same as in the Calcite > implementation, we can drop the Hive's override in order to avoid the risk of > them drifting away again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-23559) Optimise Hive::moveAcidFiles for cloud storage
[
https://issues.apache.org/jira/browse/HIVE-23559?focusedWorklogId=831771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831771
]
ASF GitHub Bot logged work on HIVE-23559:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:54
Start Date: 07/Dec/22 13:54
Worklog Time Spent: 10m
Work Description: difin commented on code in PR #3795:
URL: https://github.com/apache/hive/pull/3795#discussion_r1042233921
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -5208,55 +5208,94 @@ private static void moveAcidFiles(String deltaFileType,
PathFilter pathFilter, F
}
LOG.debug("Acid move found " + deltaStats.length + " " + deltaFileType + "
files");
+List> futures = new LinkedList<>();
+final ExecutorService pool =
conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25) > 0 ?
+
Executors.newFixedThreadPool(conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname,
25),
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Move-Acid-Files-Thread-%d").build())
: null;
+
for (FileStatus deltaStat : deltaStats) {
- Path deltaPath = deltaStat.getPath();
- // Create the delta directory. Don't worry if it already exists,
- // as that likely means another task got to it first. Then move each of
the buckets.
- // it would be more efficient to try to move the delta with it's buckets
but that is
- // harder to make race condition proof.
- Path deltaDest = new Path(dst, deltaPath.getName());
- try {
-if (!createdDeltaDirs.contains(deltaDest)) {
- try {
-if(fs.mkdirs(deltaDest)) {
- try {
-
fs.rename(AcidUtils.OrcAcidVersion.getVersionFilePath(deltaStat.getPath()),
-AcidUtils.OrcAcidVersion.getVersionFilePath(deltaDest));
- } catch (FileNotFoundException fnf) {
-// There might be no side file. Skip in this case.
- }
+
+ if (null == pool) {
+moveAcidFilesForDelta(deltaFileType, fs, dst, createdDeltaDirs,
newFiles, deltaStat);
+ } else {
+futures.add(pool.submit(new Callable() {
+ @Override
+ public Void call() throws HiveException {
+try {
+ moveAcidFilesForDelta(deltaFileType, fs, dst, createdDeltaDirs,
newFiles, deltaStat);
+} catch (Exception e) {
+ final String poolMsg =
+ "Unable to move source " + deltaStat.getPath().getName()
+ " to destination " + dst.getName();
+ throw getHiveException(e, poolMsg);
}
-createdDeltaDirs.add(deltaDest);
- } catch (IOException swallowIt) {
-// Don't worry about this, as it likely just means it's already
been created.
-LOG.info("Unable to create " + deltaFileType + " directory " +
deltaDest +
-", assuming it already exists: " + swallowIt.getMessage());
+return null;
}
+}));
+ }
+}
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 831771)
Time Spent: 1h 20m (was: 1h 10m)
> Optimise Hive::moveAcidFiles for cloud storage
> --
>
> Key: HIVE-23559
> URL: https://issues.apache.org/jira/browse/HIVE-23559
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Dmitriy Fingerman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4752]
> It ends up transferring DELTA, DELETE_DELTA, BASE prefixes sequentially from
> staging to final location.
> This causes delays even with simple updates statements, which updates smaller
> number of records in cloud storage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-23559) Optimise Hive::moveAcidFiles for cloud storage
[
https://issues.apache.org/jira/browse/HIVE-23559?focusedWorklogId=831772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831772
]
ASF GitHub Bot logged work on HIVE-23559:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:54
Start Date: 07/Dec/22 13:54
Worklog Time Spent: 10m
Work Description: difin commented on code in PR #3795:
URL: https://github.com/apache/hive/pull/3795#discussion_r104223
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -5208,55 +5208,94 @@ private static void moveAcidFiles(String deltaFileType,
PathFilter pathFilter, F
}
LOG.debug("Acid move found " + deltaStats.length + " " + deltaFileType + "
files");
+List> futures = new LinkedList<>();
+final ExecutorService pool =
conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25) > 0 ?
+
Executors.newFixedThreadPool(conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname,
25),
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Move-Acid-Files-Thread-%d").build())
: null;
+
for (FileStatus deltaStat : deltaStats) {
- Path deltaPath = deltaStat.getPath();
- // Create the delta directory. Don't worry if it already exists,
- // as that likely means another task got to it first. Then move each of
the buckets.
- // it would be more efficient to try to move the delta with it's buckets
but that is
- // harder to make race condition proof.
- Path deltaDest = new Path(dst, deltaPath.getName());
- try {
-if (!createdDeltaDirs.contains(deltaDest)) {
- try {
-if(fs.mkdirs(deltaDest)) {
- try {
-
fs.rename(AcidUtils.OrcAcidVersion.getVersionFilePath(deltaStat.getPath()),
-AcidUtils.OrcAcidVersion.getVersionFilePath(deltaDest));
- } catch (FileNotFoundException fnf) {
-// There might be no side file. Skip in this case.
- }
+
+ if (null == pool) {
+moveAcidFilesForDelta(deltaFileType, fs, dst, createdDeltaDirs,
newFiles, deltaStat);
+ } else {
+futures.add(pool.submit(new Callable() {
+ @Override
+ public Void call() throws HiveException {
+try {
+ moveAcidFilesForDelta(deltaFileType, fs, dst, createdDeltaDirs,
newFiles, deltaStat);
+} catch (Exception e) {
+ final String poolMsg =
+ "Unable to move source " + deltaStat.getPath().getName()
+ " to destination " + dst.getName();
+ throw getHiveException(e, poolMsg);
}
-createdDeltaDirs.add(deltaDest);
- } catch (IOException swallowIt) {
-// Don't worry about this, as it likely just means it's already
been created.
-LOG.info("Unable to create " + deltaFileType + " directory " +
deltaDest +
-", assuming it already exists: " + swallowIt.getMessage());
+return null;
}
+}));
+ }
+}
+
+if (null != pool) {
+ pool.shutdown();
+ for (Future future : futures) {
+try {
+ future.get();
+} catch (Exception e) {
+ throw handlePoolException(pool, e);
}
-FileStatus[] bucketStats = fs.listStatus(deltaPath,
AcidUtils.bucketFileFilter);
-LOG.debug("Acid move found " + bucketStats.length + " bucket files");
-for (FileStatus bucketStat : bucketStats) {
- Path bucketSrc = bucketStat.getPath();
- Path bucketDest = new Path(deltaDest, bucketSrc.getName());
- final String msg = "Unable to move source " + bucketSrc + " to
destination " +
- bucketDest;
- LOG.info("Moving bucket " + bucketSrc.toUri().toString() + " to " +
- bucketDest.toUri().toString());
- try {
-fs.rename(bucketSrc, bucketDest);
-if (newFiles != null) {
- newFiles.add(bucketDest);
+ }
+}
+ }
+
+ private static void moveAcidFilesForDelta(String deltaFileType, FileSystem
fs,
+Path dst, Set
createdDeltaDirs,
+List newFiles, FileStatus
deltaStat) throws HiveException {
+
+Path deltaPath = deltaStat.getPath();
+// Create the delta directory. Don't worry if it already exists,
+// as that likely means another task got to it first. Then move each of
the buckets.
+// it would be more efficient to try to move the delta with it's buckets
but that is
+// harder to make race condition proof.
+Path deltaDest = new Path(dst, deltaPath.getName());
+try {
+ if (!createdDeltaDirs.contains(deltaDest)) {
+try {
+ if(fs.mkdirs(deltaDest)) {
+try {
+
fs.rename(AcidUtils.OrcA
[jira] [Work logged] (HIVE-26221) Add histogram-based column statistics
[ https://issues.apache.org/jira/browse/HIVE-26221?focusedWorklogId=831765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831765 ] ASF GitHub Bot logged work on HIVE-26221: - Author: ASF GitHub Bot Created on: 07/Dec/22 13:45 Start Date: 07/Dec/22 13:45 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3137: URL: https://github.com/apache/hive/pull/3137#issuecomment-1340990094 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3137) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3137&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [47 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3137&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3137&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3137&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831765) Time Spent: 5h 20m (was: 5h 10m) > Add histogram-based column statistics > - > > Key: HIVE-26221 > URL: https://issues.apache.org/jira/browse/HIVE-26221 > Project: Hive > Issue Type: Improvement > Components: CBO, Metastore, Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > Hive does not support histogram statistics, which are particularly useful for > skewed data (which is very common in practice) and range predicates. > Hive's current selectivity estimation for range predicates is based on a > hard-coded value of 1/3 (see > [FilterSelectivityEstimator.java#L138-L144|https://github.com/apache/hive/blob/56c336268ea8c281d23c22d89271af37cb7e2572/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java#L138-L144]).]) > The current proposal aims at integrating hi
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[
https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831764
]
ASF GitHub Bot logged work on HIVE-26569:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:42
Start Date: 07/Dec/22 13:42
Worklog Time Spent: 10m
Work Description: abstractdog commented on code in PR #3626:
URL: https://github.com/apache/hive/pull/3626#discussion_r1042221065
##
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java:
##
@@ -403,9 +401,8 @@ public void indicateError(Throwable t) {
+ " appId=" +
currentQueryIdentifierProto.getApplicationIdString()
+ " dagId=" + currentQueryIdentifierProto.getDagIndex()
+ " to node " + node.getHost());
- if (!processSendError(t)) {
-callback.setError(null, t);
- }
+ processSendError(t);
Review Comment:
discussed on slack: I haven't touched getDelegationToken API with this
patch, appId is of no relevance (it's not even used later)
Issue Time Tracking
---
Worklog Id: (was: 831764)
Time Spent: 4h 50m (was: 4h 40m)
> Support renewal and recreation of LLAP_TOKENs
> -
>
> Key: HIVE-26569
> URL: https://issues.apache.org/jira/browse/HIVE-26569
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[
https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831756&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831756
]
ASF GitHub Bot logged work on HIVE-26569:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:31
Start Date: 07/Dec/22 13:31
Worklog Time Spent: 10m
Work Description: abstractdog commented on code in PR #3626:
URL: https://github.com/apache/hive/pull/3626#discussion_r1042207296
##
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java:
##
@@ -31,6 +31,9 @@
public class LlapServerSecurityInfo extends SecurityInfo {
private static final Logger LOG =
LoggerFactory.getLogger(LlapServerSecurityInfo.class);
+ public LlapServerSecurityInfo() {
+ }
+
Review Comment:
checked, it's not needed, removing
##
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java:
##
@@ -30,6 +30,9 @@
public class LlapTokenSelector implements TokenSelector {
private static final Logger LOG =
LoggerFactory.getLogger(LlapTokenSelector.class);
+ public LlapTokenSelector() {
+ }
+
Review Comment:
checked, it's not needed, removing
Issue Time Tracking
---
Worklog Id: (was: 831756)
Time Spent: 4h 40m (was: 4.5h)
> Support renewal and recreation of LLAP_TOKENs
> -
>
> Key: HIVE-26569
> URL: https://issues.apache.org/jira/browse/HIVE-26569
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[
https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831754&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831754
]
ASF GitHub Bot logged work on HIVE-26569:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:30
Start Date: 07/Dec/22 13:30
Worklog Time Spent: 10m
Work Description: abstractdog commented on code in PR #3626:
URL: https://github.com/apache/hive/pull/3626#discussion_r1042205765
##
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java:
##
@@ -30,6 +30,9 @@
public class LlapTokenSelector implements TokenSelector {
private static final Logger LOG =
LoggerFactory.getLogger(LlapTokenSelector.class);
+ public LlapTokenSelector() {
+ }
+
Review Comment:
checked, it's not needed, removing
Issue Time Tracking
---
Worklog Id: (was: 831754)
Time Spent: 4.5h (was: 4h 20m)
> Support renewal and recreation of LLAP_TOKENs
> -
>
> Key: HIVE-26569
> URL: https://issues.apache.org/jira/browse/HIVE-26569
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-26812) hive-it-util module misses a dependency on hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644328#comment-17644328
]
Alessandro Solimando edited comment on HIVE-26812 at 12/7/22 1:10 PM:
--
[~zabetak], thanks for your input, I agree on the analysis, we can fix it here
as per the PR, but we should open another ticket to track the issue in the
beeline module.
EDIT: as per why it works when compiling from the main directory with -Pitests,
given your findings, I am wondering if the dependency-reduced.xml from
hive-beeline module with the needed dependencies is used?
was (Author: asolimando):
[~zabetak], thanks for your input, I agree on the analysis, we can fix it here
as per the PR, but we should open another ticket to track the issue in the
beeline module.
> hive-it-util module misses a dependency on hive-jdbc
>
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26812) hive-it-util module misses a dependency on hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644328#comment-17644328
]
Alessandro Solimando commented on HIVE-26812:
-
[~zabetak], thanks for your input, I agree on the analysis, we can fix it here
as per the PR, but we should open another ticket to track the issue in the
beeline module.
> hive-it-util module misses a dependency on hive-jdbc
>
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[
https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831747&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831747
]
ASF GitHub Bot logged work on HIVE-26569:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 13:03
Start Date: 07/Dec/22 13:03
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #3626:
URL: https://github.com/apache/hive/pull/3626#discussion_r1042178119
##
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java:
##
@@ -403,9 +401,8 @@ public void indicateError(Throwable t) {
+ " appId=" +
currentQueryIdentifierProto.getApplicationIdString()
+ " dagId=" + currentQueryIdentifierProto.getDagIndex()
+ " to node " + node.getHost());
- if (!processSendError(t)) {
-callback.setError(null, t);
- }
+ processSendError(t);
Review Comment:
I meant like that method in the end reaches fetchtoken, and there we are
putting appid as null
Can we not propogate the appId from there and use it here
```
private synchronized void fetchToken() {
LOG.debug("Trying to fetch a new token...");
try {
Token newToken =
tokenClient.withCurrentToken(new
Token(token)).getDelegationToken(null); <
Issue Time Tracking
---
Worklog Id: (was: 831747)
Time Spent: 4h 20m (was: 4h 10m)
> Support renewal and recreation of LLAP_TOKENs
> -
>
> Key: HIVE-26569
> URL: https://issues.apache.org/jira/browse/HIVE-26569
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?focusedWorklogId=831743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831743
]
ASF GitHub Bot logged work on HIVE-26578:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 12:47
Start Date: 07/Dec/22 12:47
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3823:
URL: https://github.com/apache/hive/pull/3823#discussion_r1042155672
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/create/CreateMaterializedViewDesc.java:
##
@@ -237,11 +237,17 @@ public void setTblProps(Map tblProps) {
this.tblProps = tblProps;
}
- @Explain(displayName = "table properties")
public Map getTblProps() {
return tblProps;
}
+ @Explain(displayName = "table properties")
+ public Map getTblPropsExplain() { // only for displaying plan
+return PlanUtils.getPropertiesForExplain(tblProps,
+hive_metastoreConstants.TABLE_IS_CTAS,
Review Comment:
should we rename the config to something like TABLE_IS_CTAS_OR_CMV or add a
new one?
Issue Time Tracking
---
Worklog Id: (was: 831743)
Time Spent: 1h 10m (was: 1h)
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?focusedWorklogId=831739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831739
]
ASF GitHub Bot logged work on HIVE-26578:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 12:40
Start Date: 07/Dec/22 12:40
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3823:
URL: https://github.com/apache/hive/pull/3823#discussion_r1042155672
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/create/CreateMaterializedViewDesc.java:
##
@@ -237,11 +237,17 @@ public void setTblProps(Map tblProps) {
this.tblProps = tblProps;
}
- @Explain(displayName = "table properties")
public Map getTblProps() {
return tblProps;
}
+ @Explain(displayName = "table properties")
+ public Map getTblPropsExplain() { // only for displaying plan
+return PlanUtils.getPropertiesForExplain(tblProps,
+hive_metastoreConstants.TABLE_IS_CTAS,
Review Comment:
should we rename the config to something like isCTASorCM or add a new one?
Issue Time Tracking
---
Worklog Id: (was: 831739)
Time Spent: 1h (was: 50m)
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26811) SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans
[ https://issues.apache.org/jira/browse/HIVE-26811?focusedWorklogId=831738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831738 ] ASF GitHub Bot logged work on HIVE-26811: - Author: ASF GitHub Bot Created on: 07/Dec/22 12:33 Start Date: 07/Dec/22 12:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3837: URL: https://github.com/apache/hive/pull/3837#issuecomment-1340902586 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3837) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3837&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3837&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3837&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3837&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831738) Time Spent: 40m (was: 0.5h) > SharedWorkOptimizer: take the union of virtual columns in mergeable TableScans > -- > > Key: HIVE-26811 > URL: https://issues.apache.org/jira/browse/HIVE-26811 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26812) hive-it-util module misses a dependency on hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644309#comment-17644309
]
Stamatis Zampetakis commented on HIVE-26812:
I have the impression that this build problem is caused by HIVE-25750.
The util module has a dependency to hive-beeline and the latter depends on
hive-jdbc.
Normally the hive-jdbc should be available in util module (either transitively
either via the shading beeline jar).
Due to shading a dependecy-reduced.xml file is installed in the local maven
repo (~/.m2/repository/org/apache/hive/hive-beeline/4.0.0-SNAPSHOT/) for
beeline which indicates that the jar doesn't have any transitive dependencies.
If we were publishing the shaded jar that would be true but we publish the
regular jar.
At this point every project/module which includes hive-beeline as a maven
dependency is broken.
For the time being we can include explicitly hive-jdbc in utils module to
overcome this failure but the real problem is in beeline.
> hive-it-util module misses a dependency on hive-jdbc
>
>
> Key: HIVE-26812
> URL: https://issues.apache.org/jira/browse/HIVE-26812
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Building from $hive/itests fails as follows:
> {noformat}
> [INFO] Hive Integration - Testing Utilities ... FAILURE [ 6.492
> s]
> ...
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 56.499 s
> [INFO] Finished at: 2022-12-06T19:24:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hive-it-util: Compilation failure
> [ERROR]
> /Users/asolimando/git/hive/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:[51,28]
> cannot find symbol
> [ERROR] symbol: class Utils
> [ERROR] location: package org.apache.hive.jdbc
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-it-util{noformat}
> Surprisingly, building from the top directory with -Pitests does not fail.
> There is a missing dependency on the hive-jdbc module, when adding that, the
> error gets fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26569) Support renewal and recreation of LLAP_TOKENs
[
https://issues.apache.org/jira/browse/HIVE-26569?focusedWorklogId=831735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831735
]
ASF GitHub Bot logged work on HIVE-26569:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 12:17
Start Date: 07/Dec/22 12:17
Worklog Time Spent: 10m
Work Description: abstractdog commented on code in PR #3626:
URL: https://github.com/apache/hive/pull/3626#discussion_r1042134071
##
llap-client/src/java/org/apache/hadoop/hive/llap/tez/LlapProtocolClientProxy.java:
##
@@ -94,6 +105,35 @@ public void sendUpdateFragment(final
UpdateFragmentRequestProto request, final S
queueRequest(new SendUpdateFragmentCallable(nodeId, request, callback));
}
+ protected void initPeriodicTokenRefresh(Configuration conf) {
+if (!UserGroupInformation.isSecurityEnabled()) {
+ return;
+}
+long tokenRenewInterval =
+HiveConf.getTimeVar(conf,
ConfVars.LLAP_DELEGATION_TOKEN_RENEW_INTERVAL, TimeUnit.SECONDS);
+// if the tokenRenewInterval is low (e.g. testing), let's use the half of
it as interval instead of the constant
+long interval = Math.min(tokenRenewInterval / 2,
LLAP_TOKEN_REFRESH_INTERVAL_IN_AM_SECONDS);
Review Comment:
you know what, I'm removing this logic altogether and using
LLAP_TOKEN_REFRESH_INTERVAL_IN_AM_SECONDS
let's keep this simple
Issue Time Tracking
---
Worklog Id: (was: 831735)
Time Spent: 4h 10m (was: 4h)
> Support renewal and recreation of LLAP_TOKENs
> -
>
> Key: HIVE-26569
> URL: https://issues.apache.org/jira/browse/HIVE-26569
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26785) Remove explicit protobuf-java dependency from blobstore and minikdc modules
[ https://issues.apache.org/jira/browse/HIVE-26785?focusedWorklogId=831729&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831729 ] ASF GitHub Bot logged work on HIVE-26785: - Author: ASF GitHub Bot Created on: 07/Dec/22 11:35 Start Date: 07/Dec/22 11:35 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3809: URL: https://github.com/apache/hive/pull/3809#issuecomment-1340839499 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3809) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3809&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3809&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3809&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3809&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3809&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3809&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831729) Remaining Estimate: 0h Time Spent: 10m > Remove explicit protobuf-java dependency from blobstore and minikdc modules > --- > > Key: HIVE-26785 > URL: https://issues.apache.org/jira/browse/HIVE-26785 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The modules do not directly need protobuf dependency so it is misleading to > declare it explicitly. > Moreover, these modules use a different protobuf version (3.3.0) than the > rest of the project (3.21.4) which can lead to compatibility problems, > inconsistent behavior in tests, and undesired transitive propagation to other > modules. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26785) Remove explicit protobuf-java dependency from blobstore and minikdc modules
[ https://issues.apache.org/jira/browse/HIVE-26785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26785: -- Labels: pull-request-available (was: ) > Remove explicit protobuf-java dependency from blobstore and minikdc modules > --- > > Key: HIVE-26785 > URL: https://issues.apache.org/jira/browse/HIVE-26785 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The modules do not directly need protobuf dependency so it is misleading to > declare it explicitly. > Moreover, these modules use a different protobuf version (3.3.0) than the > rest of the project (3.21.4) which can lead to compatibility problems, > inconsistent behavior in tests, and undesired transitive propagation to other > modules. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26810) Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26810: -- Labels: pull-request-available (was: ) > Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in > implementation > -- > > Key: HIVE-26810 > URL: https://issues.apache.org/jira/browse/HIVE-26810 > Project: Hive > Issue Type: Task > Components: CBO >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > After HIVE-26762, the _onMatch_ method is now the same as in the Calcite > implementation, we can drop the Hive's override in order to avoid the risk of > them drifting away again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26810) Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26810?focusedWorklogId=831727&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831727 ] ASF GitHub Bot logged work on HIVE-26810: - Author: ASF GitHub Bot Created on: 07/Dec/22 11:04 Start Date: 07/Dec/22 11:04 Worklog Time Spent: 10m Work Description: asolimando opened a new pull request, #3842: URL: https://github.com/apache/hive/pull/3842 …Calcite's built-in implementation ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 831727) Remaining Estimate: 0h Time Spent: 10m > Replace HiveFilterSetOpTransposeRule onMatch method with Calcite's built-in > implementation > -- > > Key: HIVE-26810 > URL: https://issues.apache.org/jira/browse/HIVE-26810 > Project: Hive > Issue Type: Task > Components: CBO >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > After HIVE-26762, the _onMatch_ method is now the same as in the Calcite > implementation, we can drop the Hive's override in order to avoid the risk of > them drifting away again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26770) Make "end of loop" compaction logs appear more selectively
[
https://issues.apache.org/jira/browse/HIVE-26770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Mathur updated HIVE-26770:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Make "end of loop" compaction logs appear more selectively
> --
>
> Key: HIVE-26770
> URL: https://issues.apache.org/jira/browse/HIVE-26770
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-1
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Currently Initiator, Worker, and Cleaner threads log something like "finished
> one loop" on INFO level.
> This is useful to figure out if one of these threads is taking too long to
> finish a loop, but expensive in general.
>
> Suggested Time: 20mins
> Logging this should be changed in the following way
> # If loop finished within a predefined amount of time, level should be DEBUG
> and message should look like: *Initiator loop took \{ellapsedTime} seconds to
> finish.*
> # If loop ran longer than this predefined amount, level should be WARN and
> message should look like: *Possible Initiator slowdown, loop took
> \{ellapsedTime} seconds to finish.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26770) Make "end of loop" compaction logs appear more selectively
[
https://issues.apache.org/jira/browse/HIVE-26770?focusedWorklogId=831719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831719
]
ASF GitHub Bot logged work on HIVE-26770:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 10:14
Start Date: 07/Dec/22 10:14
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #3832:
URL: https://github.com/apache/hive/pull/3832
Issue Time Tracking
---
Worklog Id: (was: 831719)
Time Spent: 7h 10m (was: 7h)
> Make "end of loop" compaction logs appear more selectively
> --
>
> Key: HIVE-26770
> URL: https://issues.apache.org/jira/browse/HIVE-26770
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-1
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Currently Initiator, Worker, and Cleaner threads log something like "finished
> one loop" on INFO level.
> This is useful to figure out if one of these threads is taking too long to
> finish a loop, but expensive in general.
>
> Suggested Time: 20mins
> Logging this should be changed in the following way
> # If loop finished within a predefined amount of time, level should be DEBUG
> and message should look like: *Initiator loop took \{ellapsedTime} seconds to
> finish.*
> # If loop ran longer than this predefined amount, level should be WARN and
> message should look like: *Possible Initiator slowdown, loop took
> \{ellapsedTime} seconds to finish.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26392) Move StringableMap tests into a dedicated test class
[ https://issues.apache.org/jira/browse/HIVE-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Rátkai updated HIVE-26392: - Fix Version/s: 4.0.0 (was: 4.0.0-alpha-2) > Move StringableMap tests into a dedicated test class > > > Key: HIVE-26392 > URL: https://issues.apache.org/jira/browse/HIVE-26392 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `StringableMap` has unit tests in `TestWorker.java`. They could be in their > own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644252#comment-17644252
]
Stamatis Zampetakis commented on HIVE-26806:
I plan to keep the the timeout to the same value (12h) for a while longer since
I've seen a recent build
(http://ci.hive.apache.org/job/hive-precommit/job/master/1519/) taking 5h30.
Although the 5h are not spend executing tests, but also allocating executors, I
don't want to risk destabilising CI again now.
Let's monitor
http://ci.hive.apache.org/job/hive-precommit/job/master/buildTimeTrend for the
following days and take a decision next week.
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26794) Explore retiring TxnHandler#connPoolMutex idle connections
[ https://issues.apache.org/jira/browse/HIVE-26794?focusedWorklogId=831701&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831701 ] ASF GitHub Bot logged work on HIVE-26794: - Author: ASF GitHub Bot Created on: 07/Dec/22 09:28 Start Date: 07/Dec/22 09:28 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3817: URL: https://github.com/apache/hive/pull/3817#issuecomment-1340645791 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3817) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3817&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3817&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3817&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3817&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3817&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3817&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 831701) Time Spent: 2h 10m (was: 2h) > Explore retiring TxnHandler#connPoolMutex idle connections > -- > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to a more dynamic size pool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26392) Move StringableMap tests into a dedicated test class
[ https://issues.apache.org/jira/browse/HIVE-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26392: Fix Version/s: 4.0.0-alpha-2 > Move StringableMap tests into a dedicated test class > > > Key: HIVE-26392 > URL: https://issues.apache.org/jira/browse/HIVE-26392 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `StringableMap` has unit tests in `TestWorker.java`. They could be in their > own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26392) Move StringableMap tests into a dedicated test class
[ https://issues.apache.org/jira/browse/HIVE-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-26392. - Resolution: Fixed > Move StringableMap tests into a dedicated test class > > > Key: HIVE-26392 > URL: https://issues.apache.org/jira/browse/HIVE-26392 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `StringableMap` has unit tests in `TestWorker.java`. They could be in their > own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26392) Move StringableMap tests into a dedicated test class
[ https://issues.apache.org/jira/browse/HIVE-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644226#comment-17644226 ] László Bodor commented on HIVE-26392: - merged to master, thanks [~zratkai] for the patch and [~InvisibleProgrammer] for the review! > Move StringableMap tests into a dedicated test class > > > Key: HIVE-26392 > URL: https://issues.apache.org/jira/browse/HIVE-26392 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `StringableMap` has unit tests in `TestWorker.java`. They could be in their > own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26392) Move StringableMap tests into a dedicated test class
[ https://issues.apache.org/jira/browse/HIVE-26392?focusedWorklogId=831688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831688 ] ASF GitHub Bot logged work on HIVE-26392: - Author: ASF GitHub Bot Created on: 07/Dec/22 08:50 Start Date: 07/Dec/22 08:50 Worklog Time Spent: 10m Work Description: abstractdog merged PR #3741: URL: https://github.com/apache/hive/pull/3741 Issue Time Tracking --- Worklog Id: (was: 831688) Time Spent: 1h 20m (was: 1h 10m) > Move StringableMap tests into a dedicated test class > > > Key: HIVE-26392 > URL: https://issues.apache.org/jira/browse/HIVE-26392 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > `StringableMap` has unit tests in `TestWorker.java`. They could be in their > own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26794) Explore retiring TxnHandler#connPoolMutex idle connections
[ https://issues.apache.org/jira/browse/HIVE-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-26794: --- Description: Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the pool can be assigned to a more dynamic size pool due to: * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex support to maintenance tasks running inside the Metastore, these tasks are not user faced; * A fixed size connection pool as same as the pool used in ObjectStore is a waste for other non leaders in the warehouse; was: Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the pool can be assigned to NoPoolConnectionPool due to: * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex support to maintenance tasks running inside the Metastore, these tasks are not user faced; * A fixed size connection pool as same as the pool used in ObjectStore is a waste for other non leaders in the warehouse; The NoPoolConnectionPool provides connection on demand, and TxnHandler#MutexAPI only uses getConnection method to fetch a connection from the pool, so it's doable to change the pool to NoPoolConnectionPool, this would make the HMS more scaleable. > Explore retiring TxnHandler#connPoolMutex idle connections > -- > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to a more dynamic size pool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26794) Explore retiring TxnHandler#connPoolMutex idle connections
[ https://issues.apache.org/jira/browse/HIVE-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-26794: -- Assignee: Zhihua Deng > Explore retiring TxnHandler#connPoolMutex idle connections > -- > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to NoPoolConnectionPool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > The NoPoolConnectionPool provides connection on demand, and > TxnHandler#MutexAPI only uses getConnection method to fetch a connection from > the pool, so it's doable to change the pool to NoPoolConnectionPool, this > would make the HMS more scaleable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26794) Explore retiring TxnHandler#connPoolMutex idle connections
[ https://issues.apache.org/jira/browse/HIVE-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-26794: --- Summary: Explore retiring TxnHandler#connPoolMutex idle connections (was: Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool) > Explore retiring TxnHandler#connPoolMutex idle connections > -- > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to NoPoolConnectionPool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > The NoPoolConnectionPool provides connection on demand, and > TxnHandler#MutexAPI only uses getConnection method to fetch a connection from > the pool, so it's doable to change the pool to NoPoolConnectionPool, this > would make the HMS more scaleable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26816) Add Jenkins file for branch-3
[ https://issues.apache.org/jira/browse/HIVE-26816?focusedWorklogId=831682&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831682 ] ASF GitHub Bot logged work on HIVE-26816: - Author: ASF GitHub Bot Created on: 07/Dec/22 08:27 Start Date: 07/Dec/22 08:27 Worklog Time Spent: 10m Work Description: amanraj2520 opened a new pull request, #3841: URL: https://github.com/apache/hive/pull/3841 HIVE-26816 : Add Jenkins file for branch-3 ### What changes were proposed in this pull request? Added Jenkins file for pull requests made to branch-3. ### Why are the changes needed? To validate that the changes people are making to branch-3 do not break. ### Does this PR introduce _any_ user-facing change? No Issue Time Tracking --- Worklog Id: (was: 831682) Remaining Estimate: 0h Time Spent: 10m > Add Jenkins file for branch-3 > - > > Key: HIVE-26816 > URL: https://issues.apache.org/jira/browse/HIVE-26816 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Blocker > Labels: hive-3.2.0-must > Time Spent: 10m > Remaining Estimate: 0h > > We should add Jenkinsfile for branch-3 so that people can file PR against it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26816) Add Jenkins file for branch-3
[ https://issues.apache.org/jira/browse/HIVE-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26816: -- Labels: hive-3.2.0-must pull-request-available (was: hive-3.2.0-must) > Add Jenkins file for branch-3 > - > > Key: HIVE-26816 > URL: https://issues.apache.org/jira/browse/HIVE-26816 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Blocker > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We should add Jenkinsfile for branch-3 so that people can file PR against it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-26816) Add Jenkins file for branch-3
[ https://issues.apache.org/jira/browse/HIVE-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26816 started by Aman Raj. --- > Add Jenkins file for branch-3 > - > > Key: HIVE-26816 > URL: https://issues.apache.org/jira/browse/HIVE-26816 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Blocker > Labels: hive-3.2.0-must > > We should add Jenkinsfile for branch-3 so that people can file PR against it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26816) Add Jenkins file for branch-3
[ https://issues.apache.org/jira/browse/HIVE-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj updated HIVE-26816: Labels: hive-3.2.0-must (was: ) > Add Jenkins file for branch-3 > - > > Key: HIVE-26816 > URL: https://issues.apache.org/jira/browse/HIVE-26816 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Blocker > Labels: hive-3.2.0-must > > We should add Jenkinsfile for branch-3 so that people can file PR against it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26816) Add Jenkins file for branch-3
[ https://issues.apache.org/jira/browse/HIVE-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj reassigned HIVE-26816: --- > Add Jenkins file for branch-3 > - > > Key: HIVE-26816 > URL: https://issues.apache.org/jira/browse/HIVE-26816 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Blocker > > We should add Jenkinsfile for branch-3 so that people can file PR against it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26794) Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool
[ https://issues.apache.org/jira/browse/HIVE-26794?focusedWorklogId=831680&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831680 ] ASF GitHub Bot logged work on HIVE-26794: - Author: ASF GitHub Bot Created on: 07/Dec/22 08:18 Start Date: 07/Dec/22 08:18 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on PR #3817: URL: https://github.com/apache/hive/pull/3817#issuecomment-1340567809 Change the `TxnHandler#connPoolMutex` from a fixed size connection pool to a dynamic size(2 ~ maxPoolSize) pool, so that HMS won't suffer from starvation of connection or establish numerous connections to underlying database, Tested HikariCP, the pool can grow or shrink on demand: > 2022-12-07T06:45:55,589 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=3, active=1, idle=2, waiting=0) 2022-12-07T06:46:25,587 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=10, active=10, idle=0, waiting=294) 2022-12-07T06:46:25,587 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=10, active=10, idle=0, waiting=294) 2022-12-07T06:46:55,588 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=7, active=1, idle=6, waiting=0) 2022-12-07T06:46:55,588 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=7, active=1, idle=6, waiting=0) 2022-12-07T06:47:25,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=3, active=2, idle=1, waiting=0) 2022-12-07T06:47:25,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=3, active=2, idle=1, waiting=0) 2022-12-07T06:47:55,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=4, active=1, idle=3, waiting=0) 2022-12-07T06:47:55,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=4, active=1, idle=3, waiting=0) 2022-12-07T06:48:25,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=4, active=1, idle=3, waiting=0) 2022-12-07T06:48:25,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex - After cleanup stats (total=4, active=1, idle=3, waiting=0) 2022-12-07T06:48:55,593 DEBUG [mutex housekeeper] pool.HikariPool: mutex - Before cleanup stats (total=3, active=1, idle=2, waiting=0) The DBCP has a little bit different from `HikariCP`, though I have set `maxIdle` and `timeBetweenEvictionRunsMillis` ```java objectPool.setMaxIdle(Math.min(maxPoolSize, 2)); objectPool.setTimeBetweenEvictionRunsMillis(30); ``` but don't see a drop down of connections when HMS is free. Issue Time Tracking --- Worklog Id: (was: 831680) Time Spent: 2h (was: 1h 50m) > Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool > - > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to NoPoolConnectionPool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > The NoPoolConnectionPool provides connection on demand, and > TxnHandler#MutexAPI only uses getConnection method to fetch a connection from > the pool, so it's doable to change the pool to NoPoolConnectionPool, this > would make the HMS more scaleable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[
https://issues.apache.org/jira/browse/HIVE-26815?focusedWorklogId=831677&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831677
]
ASF GitHub Bot logged work on HIVE-26815:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 08:13
Start Date: 07/Dec/22 08:13
Worklog Time Spent: 10m
Work Description: yigress opened a new pull request, #3840:
URL: https://github.com/apache/hive/pull/3840
### What changes were proposed in this pull request?
1. add a hive configuration hive.use.scratchdir.for.staging
2. for native table, no-mm, no-direct-insert, no-acid, change dynamic
partition staging directory layout from
///
to
///
3. when hive.use.scratchdir.for.staging=true, FileSinkOperator's dirName,
DynamicContext's sourcePath change from
/{hive.exec.stagingdir}
to
for example for query
insert into/overwrite table partition(year=2001, season) select...
before the change, the FileSinkOperator conf has
/year=2001/.staging_dir/season=xxx
after the change, it has
/.staging_dir/year=2001/season=xxx
This change allow to swap with another path such as
, and the moveTask will move into
### Why are the changes needed?
In the S3 blobstorage optimization, HIVE-15121 / HIVE-17620 changed interim
job path to use hive.exec.scracthdir, final job to use hive.exec.stagingdir.
https://issues.apache.org/jira/browse/HIVE-15215 is open whether to use scratch
for staging dir for S3.
However for blobstorage where 'rename' is slow and no encryption, it can
help performance to use scratchdir to staging query results and use the
MoveTask to copy to blobstorage. This is especially true when there is
FileMerge task.
This may also help cross-filesystem when user wants to use local cluster
filesystem to staging query results and move the results to destination
filesystem.
### Does this PR introduce _any_ user-facing change?
This adds a new hive configuration hive.use.scratchdir.for.staging, default
false
### How was this patch tested?
Tested with patch on hive 3.1.2
Issue Time Tracking
---
Worklog Id: (was: 831677)
Remaining Estimate: 0h
Time Spent: 10m
> Backport HIVE-26758 (Allow use scratchdir for staging final job)
>
>
> Key: HIVE-26815
> URL: https://issues.apache.org/jira/browse/HIVE-26815
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Yi Zhang
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-26758 add an option to allow choose set final job staging with
> hive.exec.scratchdir. This is to backport this into 3.2.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26815: -- Labels: pull-request-available (was: ) > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhang updated HIVE-26815: Summary: Backport HIVE-26758 (Allow use scratchdir for staging final job) (was: Backport HIVE-26758 (Allow use scratchdir for staging final job) to 3.2.0) > Backport HIVE-26758 (Allow use scratchdir for staging final job) > > > Key: HIVE-26815 > URL: https://issues.apache.org/jira/browse/HIVE-26815 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Yi Zhang >Priority: Minor > > HIVE-26758 add an option to allow choose set final job staging with > hive.exec.scratchdir. This is to backport this into 3.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=831667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831667
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 07/Dec/22 07:59
Start Date: 07/Dec/22 07:59
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3806:
URL: https://github.com/apache/hive/pull/3806#issuecomment-1340545069
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3806)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3806&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3806&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3806&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3806&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3806&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3806&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 831667)
Time Spent: 4h (was: 3h 50m)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
