[jira] [Work logged] (HIVE-27177) Add alter table...Convert to Iceberg command
[
https://issues.apache.org/jira/browse/HIVE-27177?focusedWorklogId=854947&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854947
]
ASF GitHub Bot logged work on HIVE-27177:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 06:58
Start Date: 05/Apr/23 06:58
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4155:
URL: https://github.com/apache/hive/pull/4155#issuecomment-1497010348
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4155)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4155&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4155&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 854947)
Time Spent: 2h 10m (was: 2h)
> Add alter table...Convert to Iceberg command
>
>
> Key: HIVE-27177
> URL: https://issues.apache.org/jira/browse/HIVE-27177
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Add an alter table convert to Iceberg [TBLPROPERTIES('','')] to
> convert exiting external tables to iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854946&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854946
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 06:53
Start Date: 05/Apr/23 06:53
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1158093290
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -875,10 +876,17 @@ public void alterTable(String catName, String dbName,
String tblName, Table newT
if (newTbl.getParameters() != null) {
newTbl.getParameters().remove(hive_metastoreConstants.DDL_TIME);
}
- newTbl.checkValidity(conf);
if (environmentContext == null) {
environmentContext = new EnvironmentContext();
}
+ if (environmentContext.isSetProperties() &&
AlterTableType.RENAME.toString()
Review Comment:
**environmentContext.isSetProperties()** is a **NULL** check for properties
map inside, if that isn't set the map inside is null, trying to fetch value
from that would fetch us an **NPE**.
done at all places:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java#L462
have refactored to use`AlterTableType.valueOf(..)`
Issue Time Tracking
---
Worklog Id: (was: 854946)
Time Spent: 2h 40m (was: 2.5h)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[
https://issues.apache.org/jira/browse/HIVE-27203?focusedWorklogId=854940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854940
]
ASF GitHub Bot logged work on HIVE-27203:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 05:54
Start Date: 05/Apr/23 05:54
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4181:
URL: https://github.com/apache/hive/pull/4181#discussion_r1158051755
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
+
+create table orc_bucketed (a int, b string) clustered by (a) into 3 buckets
stored as orc TBLPROPERTIES('transactional'='true',
'transactional_properties'='insert_only');
+
+insert into orc_bucketed values('1', 'text1');
+insert into orc_bucketed values('2', 'text2');
+insert into orc_bucketed values('3', 'text3');
+insert into orc_bucketed values('4', 'text4');
+insert into orc_bucketed values('5', 'text5');
+insert into orc_bucketed values('6', 'text6');
+insert into orc_bucketed values('7', 'text7');
+insert into orc_bucketed values('8', 'text8');
+insert into orc_bucketed values('9', 'text9');
+insert into orc_bucketed values('10', 'text10');
+
+describe extended orc_bucketed;
+alter table orc_bucketed compact 'MAJOR' and wait;
+analyze table orc_bucketed compute statistics;
Review Comment:
@akshat0395 `hive.stats.autogather` is already true by default. This config
is used by insert/delete/update statements to compute stats once records are
written.
Compaction checks this config - `hive.compactor.gather.stats` to see whether
it has to perform stats computation of the table which has been compacted.
Since its turned off by MiniLlapLocalCompactorCliDriver, stats computation is
disabled.
Issue Time Tracking
---
Worklog Id: (was: 854940)
Time Spent: 1h 40m (was: 1.5h)
> Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID,
> and combination Tables
> --
>
> Key: HIVE-27203
> URL: https://issues.apache.org/jira/browse/HIVE-27203
> Project: Hive
> Issue Type: Test
>Affects Versions: 4.0.0
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Improve Qtest Coverage for Compaction use cases for ACID Tables:
> # Partitioned Tables( Major & Minor )
> # Insert-Only Clustered( Major & Minor )
> # Insert-Only Partitioned( Major & Minor )
> # Insert-Only Clustered and Partitioned( Major & Minor )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24706) Spark SQL access hive on HBase table access exception
[
https://issues.apache.org/jira/browse/HIVE-24706?focusedWorklogId=854939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854939
]
ASF GitHub Bot logged work on HIVE-24706:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 05:53
Start Date: 05/Apr/23 05:53
Worklog Time Spent: 10m
Work Description: alexdongli0829 opened a new pull request, #4199:
URL: https://github.com/apache/hive/pull/4199
### What changes were proposed in this pull request?
For HIVE-24706, the main issue here is the HiveHbaseTableInput format
implements two version of InputFormat, which make the spark cannot get the
correct version correctly, and this is indeed not very clear implementation.
So in this request, instead of directly extending TableInputFormatBase, I
put it as a delegate which do the exactly the same as before, but avoid the
confusing because the HbaseStorageHandler just need the old version InputFormat.
In the long term, I think hive should update the storage handler instead of
keep mixing these two different API versions
### Why are the changes needed?
Its impacting the spark and hive compatible and reported by different uses
in hive and spark
### Does this PR introduce _any_ user-facing change?
There is configuration parameter added hive.hbase.inputformat.v2, so maybe
need update doc to keep the end user informed
### How was this patch tested?
create hbase table
```
echo "create 'students','account','address'" | sudo -u hbase hbase shell -n
echo "put 'students','student1','account:name','Alice'" |sudo -u hbase hbase
shell -n
```
create hive table
```
hive -e "create external table test1 (key string, value string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with serdeproperties ('hbase.columns.mapping' = ':key,account:name')
> tblproperties ('hbase.table.name' = 'students')"
SLF4J: Class path contains multiple SLF4J bindings.
Logging initialized using configuration in
file:/etc/hive/conf.dist/hive-log4j2.properties Async: true
Hive Session ID = 05b4ec22-d15f-4614-9bc3-6c183e868728
OK
Time taken: 2.913 seconds
```
Spark test:
spark-sql --jars
/usr/lib/hive/lib/hive-hbase-handler.jar,/usr/lib/hbase/hbase-common-2.4.4.jar,/usr/lib/hbase/hbase-client-2.4.4.jar,/usr/lib/hbase/lib/hbase-mapreduce-2.4.4.jar,/usr/lib/hbase/lib/shaded-clients/hbase-shaded-client-2.4.4.jar
--conf spark.hive.hbase.inputformat.v2=true
```
spark-sql> select * from test1;
student1Alice
```
Unit Test
```
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hive.hbase.TestHiveHBaseTableInputFormatV2
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.069
s - in org.apache.hadoop.hive.hbase.TestHiveHBaseTableInputFormatV2
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
```
Issue Time Tracking
---
Worklog Id: (was: 854939)
Time Spent: 1h (was: 50m)
> Spark SQL access hive on HBase table access exception
> -
>
> Key: HIVE-24706
> URL: https://issues.apache.org/jira/browse/HIVE-24706
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: zhangzhanchang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-01-30-15-51-58-665.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Hivehbasetableinputformat relies on two versions of inputformat，one is
> org.apache.hadoop.mapred.InputFormat, the other is
> org.apache.hadoop.mapreduce.InputFormat,Causes
> spark 3.0(https://github.com/apache/spark/pull/31302) both conditions to be
> true:
> # classOf[oldInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true
> # classOf[newInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true
> !image-2021-01-30-15-51-58-665.png|width=430,height=137!
> Hivehbasetableinputformat relies on inputformat to be changed to
> org.apache.hadoop.mapreduce or org.apache.hadoop.mapred?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24706) Spark SQL access hive on HBase table access exception
[ https://issues.apache.org/jira/browse/HIVE-24706?focusedWorklogId=854938&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854938 ] ASF GitHub Bot logged work on HIVE-24706: - Author: ASF GitHub Bot Created on: 05/Apr/23 05:49 Start Date: 05/Apr/23 05:49 Worklog Time Spent: 10m Work Description: alexdongli0829 closed pull request #4063: HIVE-24706: add the HiveHBaseTableInputFormatV2 to fix the compatible issue with spark URL: https://github.com/apache/hive/pull/4063 Issue Time Tracking --- Worklog Id: (was: 854938) Time Spent: 50m (was: 40m) > Spark SQL access hive on HBase table access exception > - > > Key: HIVE-24706 > URL: https://issues.apache.org/jira/browse/HIVE-24706 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: zhangzhanchang >Priority: Major > Labels: pull-request-available > Attachments: image-2021-01-30-15-51-58-665.png > > Time Spent: 50m > Remaining Estimate: 0h > > Hivehbasetableinputformat relies on two versions of inputformat，one is > org.apache.hadoop.mapred.InputFormat, the other is > org.apache.hadoop.mapreduce.InputFormat,Causes > spark 3.0(https://github.com/apache/spark/pull/31302) both conditions to be > true: > # classOf[oldInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true > # classOf[newInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true > !image-2021-01-30-15-51-58-665.png|width=430,height=137! > Hivehbasetableinputformat relies on inputformat to be changed to > org.apache.hadoop.mapreduce or org.apache.hadoop.mapred? > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[
https://issues.apache.org/jira/browse/HIVE-27203?focusedWorklogId=854937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854937
]
ASF GitHub Bot logged work on HIVE-27203:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 05:15
Start Date: 05/Apr/23 05:15
Worklog Time Spent: 10m
Work Description: akshat0395 commented on code in PR #4181:
URL: https://github.com/apache/hive/pull/4181#discussion_r1158031774
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
+
+create table orc_bucketed (a int, b string) clustered by (a) into 3 buckets
stored as orc TBLPROPERTIES('transactional'='true',
'transactional_properties'='insert_only');
+
+insert into orc_bucketed values('1', 'text1');
+insert into orc_bucketed values('2', 'text2');
+insert into orc_bucketed values('3', 'text3');
+insert into orc_bucketed values('4', 'text4');
+insert into orc_bucketed values('5', 'text5');
+insert into orc_bucketed values('6', 'text6');
+insert into orc_bucketed values('7', 'text7');
+insert into orc_bucketed values('8', 'text8');
+insert into orc_bucketed values('9', 'text9');
+insert into orc_bucketed values('10', 'text10');
+
+describe extended orc_bucketed;
+alter table orc_bucketed compact 'MAJOR' and wait;
+analyze table orc_bucketed compute statistics;
Review Comment:
@SourabhBadhya IMK hive.compactor.gather.stats is used for Compaction stats.
the analyze table command is for table stats and for auto generation of
those stats `hive.stats.autogather=true` is used.
Ref:
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/performance-tuning/content/hive_generate_hive_statistics.html.
Please correct me if Im wrong
Issue Time Tracking
---
Worklog Id: (was: 854937)
Time Spent: 1.5h (was: 1h 20m)
> Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID,
> and combination Tables
> --
>
> Key: HIVE-27203
> URL: https://issues.apache.org/jira/browse/HIVE-27203
> Project: Hive
> Issue Type: Test
>Affects Versions: 4.0.0
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Improve Qtest Coverage for Compaction use cases for ACID Tables:
> # Partitioned Tables( Major & Minor )
> # Insert-Only Clustered( Major & Minor )
> # Insert-Only Partitioned( Major & Minor )
> # Insert-Only Clustered and Partitioned( Major & Minor )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[
https://issues.apache.org/jira/browse/HIVE-27203?focusedWorklogId=854936&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854936
]
ASF GitHub Bot logged work on HIVE-27203:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 05:09
Start Date: 05/Apr/23 05:09
Worklog Time Spent: 10m
Work Description: akshat0395 commented on code in PR #4181:
URL: https://github.com/apache/hive/pull/4181#discussion_r1158028978
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
+
+create table orc_bucketed (a int, b string) clustered by (a) into 3 buckets
stored as orc TBLPROPERTIES('transactional'='true',
'transactional_properties'='insert_only');
+
+insert into orc_bucketed values('1', 'text1');
+insert into orc_bucketed values('2', 'text2');
+insert into orc_bucketed values('3', 'text3');
+insert into orc_bucketed values('4', 'text4');
+insert into orc_bucketed values('5', 'text5');
+insert into orc_bucketed values('6', 'text6');
+insert into orc_bucketed values('7', 'text7');
+insert into orc_bucketed values('8', 'text8');
+insert into orc_bucketed values('9', 'text9');
+insert into orc_bucketed values('10', 'text10');
+
+describe extended orc_bucketed;
+alter table orc_bucketed compact 'MAJOR' and wait;
+analyze table orc_bucketed compute statistics;
+
+describe extended orc_bucketed;
Review Comment:
Added
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
Review Comment:
Updated
Issue Time Tracking
---
Worklog Id: (was: 854936)
Time Spent: 1h 20m (was: 1h 10m)
> Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID,
> and combination Tables
> --
>
> Key: HIVE-27203
> URL: https://issues.apache.org/jira/browse/HIVE-27203
> Project: Hive
> Issue Type: Test
>Affects Versions: 4.0.0
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Improve Qtest Coverage for Compaction use cases for ACID Tables:
> # Partitioned Tables( Major & Minor )
> # Insert-Only Clustered( Major & Minor )
> # Insert-Only Partitioned( Major & Minor )
> # Insert-Only Clustered and Partitioned( Major & Minor )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[
https://issues.apache.org/jira/browse/HIVE-27203?focusedWorklogId=854933&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854933
]
ASF GitHub Bot logged work on HIVE-27203:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 04:40
Start Date: 05/Apr/23 04:40
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4181:
URL: https://github.com/apache/hive/pull/4181#discussion_r1158012483
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
+
+create table orc_bucketed (a int, b string) clustered by (a) into 3 buckets
stored as orc TBLPROPERTIES('transactional'='true',
'transactional_properties'='insert_only');
+
+insert into orc_bucketed values('1', 'text1');
+insert into orc_bucketed values('2', 'text2');
+insert into orc_bucketed values('3', 'text3');
+insert into orc_bucketed values('4', 'text4');
+insert into orc_bucketed values('5', 'text5');
+insert into orc_bucketed values('6', 'text6');
+insert into orc_bucketed values('7', 'text7');
+insert into orc_bucketed values('8', 'text8');
+insert into orc_bucketed values('9', 'text9');
+insert into orc_bucketed values('10', 'text10');
+
+describe extended orc_bucketed;
+alter table orc_bucketed compact 'MAJOR' and wait;
+analyze table orc_bucketed compute statistics;
Review Comment:
Stats update on query based compaction depends upon this config -
`hive.compactor.gather.stats`
Source -
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java#L103
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java#L444
After taking a deeper look, this config seems to be disabled in
MiniLlapLocalCompactorCliDriver here as part of HIVE-26802 -
https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java#L271
@zratkai Is there any reason why this was disabled as part of HIVE-26802? I
did try some tests locally and I was able to see correct stats after removing
this disabling config line from CliConfigs.
Issue Time Tracking
---
Worklog Id: (was: 854933)
Time Spent: 1h 10m (was: 1h)
> Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID,
> and combination Tables
> --
>
> Key: HIVE-27203
> URL: https://issues.apache.org/jira/browse/HIVE-27203
> Project: Hive
> Issue Type: Test
>Affects Versions: 4.0.0
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Improve Qtest Coverage for Compaction use cases for ACID Tables:
> # Partitioned Tables( Major & Minor )
> # Insert-Only Clustered( Major & Minor )
> # Insert-Only Partitioned( Major & Minor )
> # Insert-Only Clustered and Partitioned( Major & Minor )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26537) Deprecate older APIs in the HMS
[ https://issues.apache.org/jira/browse/HIVE-26537?focusedWorklogId=854932&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854932 ] ASF GitHub Bot logged work on HIVE-26537: - Author: ASF GitHub Bot Created on: 05/Apr/23 04:33 Start Date: 05/Apr/23 04:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3599: URL: https://github.com/apache/hive/pull/3599#issuecomment-1496903970 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3599) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3599&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3599&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3599&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=CODE_SMELL) [92 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3599&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3599&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3599&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854932) Time Spent: 6h 40m (was: 6.5h) > Deprecate older APIs in the HMS > --- > > Key: HIVE-26537 > URL: https://issues.apache.org/jira/browse/HIVE-26537 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Critical > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > This Jira is to track the clean-up(deprecate older APIs and point the HMS > client to the newer APIs) work in the hive metastore server. > More details will be added here soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27189) Remove duplicate debug log in Hive.isSubDIr
[
https://issues.apache.org/jira/browse/HIVE-27189?focusedWorklogId=854926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854926
]
ASF GitHub Bot logged work on HIVE-27189:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 03:23
Start Date: 05/Apr/23 03:23
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4167:
URL: https://github.com/apache/hive/pull/4167#issuecomment-1496869872
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4167)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4167&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4167&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4167&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4167&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4167&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4167&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 854926)
Time Spent: 1.5h (was: 1h 20m)
> Remove duplicate debug log in Hive.isSubDIr
> ---
>
> Key: HIVE-27189
> URL: https://issues.apache.org/jira/browse/HIVE-27189
> Project: Hive
> Issue Type: Improvement
>Reporter: shuyouZZ
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
> {{isSubDir}} will print twice
> {code:java}
> LOG.debug("The source path is " + fullF1 + " and the destination path is " +
> fullF2);{code}
> we should remove the duplicate debug log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[ https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=854925&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854925 ] ASF GitHub Bot logged work on HIVE-26659: - Author: ASF GitHub Bot Created on: 05/Apr/23 03:12 Start Date: 05/Apr/23 03:12 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4190: URL: https://github.com/apache/hive/pull/4190#issuecomment-1496862774 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4190) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4190&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4190&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4190&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4190&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4190&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4190&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854925) Time Spent: 1h 10m (was: 1h) > TPC-DS query 16, 69, 94 return wrong results. > - > > Key: HIVE-26659 > URL: https://issues.apache.org/jira/browse/HIVE-26659 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0-alpha-2 >Reporter: Sungwoo Park >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join > is set to true. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[
https://issues.apache.org/jira/browse/HIVE-27203?focusedWorklogId=854924&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854924
]
ASF GitHub Bot logged work on HIVE-27203:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 03:10
Start Date: 05/Apr/23 03:10
Worklog Time Spent: 10m
Work Description: akshat0395 commented on code in PR #4181:
URL: https://github.com/apache/hive/pull/4181#discussion_r1157976783
##
ql/src/test/queries/clientpositive/compaction_query_based_insert_only_clustered.q:
##
@@ -0,0 +1,33 @@
+--! qt:replace:/createTime:(\d+)/#Masked#/
+--! qt:replace:/location:(\S+)/#Masked#/
+--! qt:replace:/lastAccessTime:(\d+)/#Masked#/
+--! qt:replace:/ownerType:(\S*)/#Masked#/
+--! qt:replace:/owner:(\S*)/#Masked#/
+--! qt:replace:/skewedColValueLocationMaps:(\S*)/#Masked#/
+--! qt:replace:/transient_lastDdlTime=(\d+)/#Masked#/
+--! qt:replace:/totalSize=(\d+)/#Masked#/
+--! qt:replace:/rawDataSize=(\d+)/#Masked#/
+--! qt:replace:/writeId:(\d+)/#Masked#/
+--! qt:replace:/bucketing_version=(\d+)/#Masked#/
+--! qt:replace:/id:(\d+)/#Masked#/
+
+drop table orc_bucketed;
+
+create table orc_bucketed (a int, b string) clustered by (a) into 3 buckets
stored as orc TBLPROPERTIES('transactional'='true',
'transactional_properties'='insert_only');
+
+insert into orc_bucketed values('1', 'text1');
+insert into orc_bucketed values('2', 'text2');
+insert into orc_bucketed values('3', 'text3');
+insert into orc_bucketed values('4', 'text4');
+insert into orc_bucketed values('5', 'text5');
+insert into orc_bucketed values('6', 'text6');
+insert into orc_bucketed values('7', 'text7');
+insert into orc_bucketed values('8', 'text8');
+insert into orc_bucketed values('9', 'text9');
+insert into orc_bucketed values('10', 'text10');
+
+describe extended orc_bucketed;
+alter table orc_bucketed compact 'MAJOR' and wait;
+analyze table orc_bucketed compute statistics;
Review Comment:
Thanks for the review @SourabhBadhya, The reason why this is required here
is to make sure the test runs independent of Hive config. The automatic Stats
updates happens when hive.stats.autogather=true.
To ensure we have the data in q.out regardless of config analyze statement
is used. Let me know if you have any thought on this.
Issue Time Tracking
---
Worklog Id: (was: 854924)
Time Spent: 1h (was: 50m)
> Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID,
> and combination Tables
> --
>
> Key: HIVE-27203
> URL: https://issues.apache.org/jira/browse/HIVE-27203
> Project: Hive
> Issue Type: Test
>Affects Versions: 4.0.0
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Improve Qtest Coverage for Compaction use cases for ACID Tables:
> # Partitioned Tables( Major & Minor )
> # Insert-Only Clustered( Major & Minor )
> # Insert-Only Partitioned( Major & Minor )
> # Insert-Only Clustered and Partitioned( Major & Minor )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[ https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=854923&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854923 ] ASF GitHub Bot logged work on HIVE-26659: - Author: ASF GitHub Bot Created on: 05/Apr/23 02:43 Start Date: 05/Apr/23 02:43 Worklog Time Spent: 10m Work Description: ngsg commented on code in PR #4190: URL: https://github.com/apache/hive/pull/4190#discussion_r1157957495 ## ql/src/test/queries/clientpositive/antijoin2.q: ## @@ -0,0 +1,41 @@ +set hive.merge.nway.joins=false; Review Comment: I added explain and explain cbo for each of the queries. Issue Time Tracking --- Worklog Id: (was: 854923) Time Spent: 1h (was: 50m) > TPC-DS query 16, 69, 94 return wrong results. > - > > Key: HIVE-26659 > URL: https://issues.apache.org/jira/browse/HIVE-26659 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0-alpha-2 >Reporter: Sungwoo Park >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join > is set to true. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[
https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=854922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854922
]
ASF GitHub Bot logged work on HIVE-26659:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 02:36
Start Date: 05/Apr/23 02:36
Worklog Time Spent: 10m
Work Description: ngsg commented on code in PR #4190:
URL: https://github.com/apache/hive/pull/4190#discussion_r1157954219
##
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java:
##
@@ -556,6 +556,9 @@ public void process(Object row, int tag) throws
HiveException {
}
} else {
storage[pos] = emptyList;
+ if (pos != 0 && condn[pos - 1].getType() == JoinDesc.ANTI_JOIN) {
Review Comment:
Thank you for your review. I added a comment which explains the reason for
proceeding join.
Issue Time Tracking
---
Worklog Id: (was: 854922)
Time Spent: 50m (was: 40m)
> TPC-DS query 16, 69, 94 return wrong results.
> -
>
> Key: HIVE-26659
> URL: https://issues.apache.org/jira/browse/HIVE-26659
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sungwoo Park
>Assignee: Seonggon Namgung
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join
> is set to true.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[
https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=854911&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854911
]
ASF GitHub Bot logged work on HIVE-26400:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 01:17
Start Date: 05/Apr/23 01:17
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157915578
##
packaging/src/docker/README.md:
##
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide
the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+ Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading,
writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of
the dependent, as well as build from source.
+
+# Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+# Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop
+-tez
+-hive
+```
+If the version is not provided, it will read the version from current
`pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez
respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version`
and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local
`apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't
exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by
default, the image is tagged by the provided Hive version.
+
+ Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION`
first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate
-Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name
metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive
table's schema and data, start the container with an external Postgres and
Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
This jar is already here in the hive tar ball, so I skip to fetch it from
external
Issue Time Tracking
---
Worklog Id: (was: 854911)
Time Spent: 13h 10m (was: 13h)
> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
> Issue Type: Sub-task
> Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
> Labels: hive-4.0.0-must, pull-request-available
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed
> mode, with MySQL/Derby as its back database, provide the following:
> * Quick-start/Debugging/Prepare a test env for Hive;
> * Tools to build target image with specified version of Hive and its
> dependencies;
> * Images can be used as the basis for the Kubernetes operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27205) Update jackson-databind for CVE fix for CVE-2022-42003
[ https://issues.apache.org/jira/browse/HIVE-27205?focusedWorklogId=854910&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854910 ] ASF GitHub Bot logged work on HIVE-27205: - Author: ASF GitHub Bot Created on: 05/Apr/23 01:03 Start Date: 05/Apr/23 01:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4184: URL: https://github.com/apache/hive/pull/4184#issuecomment-1496785816 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4184) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4184&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4184&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4184&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4184&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4184&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4184&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854910) Time Spent: 1h 10m (was: 1h) > Update jackson-databind for CVE fix for CVE-2022-42003 > -- > > Key: HIVE-27205 > URL: https://issues.apache.org/jira/browse/HIVE-27205 > Project: Hive > Issue Type: Task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Update jackson-databind for CVE fix for CVE-2022-42003 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27218) Hive-3 switch hive.materializedview.rewriting default to false
[ https://issues.apache.org/jira/browse/HIVE-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhang updated HIVE-27218: Summary: Hive-3 switch hive.materializedview.rewriting default to false (was: Hive-3 set hive.materializedview.rewriting default to false) > Hive-3 switch hive.materializedview.rewriting default to false > -- > > Key: HIVE-27218 > URL: https://issues.apache.org/jira/browse/HIVE-27218 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0, 3.1.2, 3.1.3 >Reporter: Yi Zhang >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-19973 switched > hive.materializedview.rewriting default from false to true, however user > observed high latency at query compilation as they have large amount of > databases (5k), each call to a remote metastore DB adds up the latency to > minutes, each query compilation time is high. > as hive-4 have improvements in HIVE-21631 and HIVE-21344 backport is > unlikely, suggest to turn this false by default in hive-3. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27145) Use StrictMath for remaining Math functions as followup of HIVE-23133
[
https://issues.apache.org/jira/browse/HIVE-27145?focusedWorklogId=854908&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854908
]
ASF GitHub Bot logged work on HIVE-27145:
-
Author: ASF GitHub Bot
Created on: 05/Apr/23 00:09
Start Date: 05/Apr/23 00:09
Worklog Time Spent: 10m
Work Description: rbalamohan commented on PR #4122:
URL: https://github.com/apache/hive/pull/4122#issuecomment-1496750915
This is specific to JDK 9 issue.
In recent JDKs it is back to the original implementation. i.e strictfp is
removed and it is back to using Math.toDegrees in StrictMath.
https://github.com/openjdk/jdk/commit/0ae4ceb413a7ea92a41cd55cd338f866098eb9b4
_Until we move to the latest JDK, it is fine to make this change._
Note that it will not have any difference in JDK 8 as well, since
StrictMath::toDegrees uses Math::toDegrees/
Issue Time Tracking
---
Worklog Id: (was: 854908)
Time Spent: 1h (was: 50m)
> Use StrictMath for remaining Math functions as followup of HIVE-23133
> -
>
> Key: HIVE-27145
> URL: https://issues.apache.org/jira/browse/HIVE-27145
> Project: Hive
> Issue Type: Task
> Components: UDF
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> [HIVE-23133|https://issues.apache.org/jira/browse/HIVE-23133] started using
> {{StrictMath}} for {{cos, exp, log}} UDFs to fix QTests failing as results
> vary based on hardware when using Math library.
> Follow it up by using {{StrictMath}} for other Math functions that can have
> same impact of underlying hardware namely, {{sin, tan, asin, acos, atan,
> sqrt, pow, cbrt}}.
> [JDK-4477961|https://bugs.openjdk.org/browse/JDK-4477961] (in Java 9) changed
> radians and degrees calculation leading to Q Test failures when tests are run
> on Java 9+, fix such tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?focusedWorklogId=854904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854904 ] ASF GitHub Bot logged work on HIVE-27184: - Author: ASF GitHub Bot Created on: 04/Apr/23 23:37 Start Date: 04/Apr/23 23:37 Worklog Time Spent: 10m Work Description: rbalamohan commented on PR #4196: URL: https://github.com/apache/hive/pull/4196#issuecomment-1496726017 Thanks @difin. LGTM. minor comment: Need to check if parameters with "$" (e.g java classnames) should be decoded. It can be a separate ticket. Issue Time Tracking --- Worklog Id: (was: 854904) Time Spent: 40m (was: 0.5h) > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance, pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27177) Add alter table...Convert to Iceberg command
[
https://issues.apache.org/jira/browse/HIVE-27177?focusedWorklogId=854903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854903
]
ASF GitHub Bot logged work on HIVE-27177:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 22:29
Start Date: 04/Apr/23 22:29
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4155:
URL: https://github.com/apache/hive/pull/4155#issuecomment-1496680595
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4155)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4155&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4155&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4155&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4155&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 854903)
Time Spent: 2h (was: 1h 50m)
> Add alter table...Convert to Iceberg command
>
>
> Key: HIVE-27177
> URL: https://issues.apache.org/jira/browse/HIVE-27177
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Add an alter table convert to Iceberg [TBLPROPERTIES('','')] to
> convert exiting external tables to iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?focusedWorklogId=854901&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854901 ] ASF GitHub Bot logged work on HIVE-27217: - Author: ASF GitHub Bot Created on: 04/Apr/23 21:25 Start Date: 04/Apr/23 21:25 Worklog Time Spent: 10m Work Description: jfsii commented on PR #4197: URL: https://github.com/apache/hive/pull/4197#issuecomment-1496625307 Re: @TuroczyX When it is re-thrown it will bubble up to the user as an exception during execution. It would be treated as other thrift exceptions would be - such as if the HMS server went down or the call timed out. It is not a good choice to swallow here, because a missing write notification will affect various systems such as maybe event notification and replication. It is much preferable to fail and make it visible so a user can take action (either by checking HMS logs, making sure HMS connectivity is good, etc) rather than having to figure out why the notification log is missing entries long after the fact. I feel like this is an oversight when attempting the fallback mechanism rather than a purposeful choice to ignore these exceptions. The reason I found this is due to the Impala catalogd implementation not implementing add_write_notification_log_in_batch which causes it to throw a different TApplicationException and thus the add_write_notification_log_in_batch is failed silently. Since catalogd also relies on the notification log/events, depending on timing of events, the metadata cache would end up containing incorrect metadata because it didn't realize a partition was updated. This could be fixed on the catalogd side of things to throw a UNKNOWN_METHOD or WRONG_METHOD_NAME exception, but it is still entirely possible for a TApplicationException of a different type to still be thrown. I've seen "UNKNOWN" in the past, but there also exists INVALID_MESSAGE_TYPE/BAD_SEQUENCE_ID/MISSING_RESULT which could very well show up depending on network conditions and/or issues during the processing on the HMS side. Issue Time Tracking --- Worklog Id: (was: 854901) Time Spent: 40m (was: 0.5h) > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27205) Update jackson-databind for CVE fix for CVE-2022-42003
[ https://issues.apache.org/jira/browse/HIVE-27205?focusedWorklogId=854900&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854900 ] ASF GitHub Bot logged work on HIVE-27205: - Author: ASF GitHub Bot Created on: 04/Apr/23 21:21 Start Date: 04/Apr/23 21:21 Worklog Time Spent: 10m Work Description: Diksha628 commented on code in PR #4184: URL: https://github.com/apache/hive/pull/4184#discussion_r1157776610 ## pom.xml: ## @@ -148,7 +148,7 @@ 4.5.13 4.4.13 2.5.1 -2.12.7 +2.12.7.1 Review Comment: Thanks @ayushtkn for pointing out, will make the change . Issue Time Tracking --- Worklog Id: (was: 854900) Time Spent: 1h (was: 50m) > Update jackson-databind for CVE fix for CVE-2022-42003 > -- > > Key: HIVE-27205 > URL: https://issues.apache.org/jira/browse/HIVE-27205 > Project: Hive > Issue Type: Task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Update jackson-databind for CVE fix for CVE-2022-42003 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27205) Update jackson-databind for CVE fix for CVE-2022-42003
[ https://issues.apache.org/jira/browse/HIVE-27205?focusedWorklogId=854898&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854898 ] ASF GitHub Bot logged work on HIVE-27205: - Author: ASF GitHub Bot Created on: 04/Apr/23 21:15 Start Date: 04/Apr/23 21:15 Worklog Time Spent: 10m Work Description: Diksha628 commented on code in PR #4184: URL: https://github.com/apache/hive/pull/4184#discussion_r1157776610 ## pom.xml: ## @@ -148,7 +148,7 @@ 4.5.13 4.4.13 2.5.1 -2.12.7 +2.12.7.1 Review Comment: Thanks @ayushtkn for pointing out, will raise another PR with the right version. Issue Time Tracking --- Worklog Id: (was: 854898) Time Spent: 50m (was: 40m) > Update jackson-databind for CVE fix for CVE-2022-42003 > -- > > Key: HIVE-27205 > URL: https://issues.apache.org/jira/browse/HIVE-27205 > Project: Hive > Issue Type: Task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Update jackson-databind for CVE fix for CVE-2022-42003 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27127) Upgrade jackson version to 2.13.5
[ https://issues.apache.org/jira/browse/HIVE-27127?focusedWorklogId=854895&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854895 ] ASF GitHub Bot logged work on HIVE-27127: - Author: ASF GitHub Bot Created on: 04/Apr/23 21:03 Start Date: 04/Apr/23 21:03 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4102: URL: https://github.com/apache/hive/pull/4102#issuecomment-1496603104 @veghlaci05 Could you pls help to @shreeyasand tomorrow? Issue Time Tracking --- Worklog Id: (was: 854895) Time Spent: 1h 10m (was: 1h) > Upgrade jackson version to 2.13.5 > - > > Key: HIVE-27127 > URL: https://issues.apache.org/jira/browse/HIVE-27127 > Project: Hive > Issue Type: Improvement >Reporter: Sand Shreeya >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?focusedWorklogId=854894&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854894 ] ASF GitHub Bot logged work on HIVE-27217: - Author: ASF GitHub Bot Created on: 04/Apr/23 20:52 Start Date: 04/Apr/23 20:52 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4197: URL: https://github.com/apache/hive/pull/4197#issuecomment-1496591734 This is truly a best practice in any OOP language to rethrow the exception. To swallow it is rarely a good idea, but **could be** a good choice especially when the domain logic should manage those error scenario where exception is an expected behaviors' from the business logic perspective. Here as I see this notification log is mainly for `TApplicationException.UNKNOWN_METHOD || TApplicationException.WRONG_METHOD_NAME) ` Which is the main use-case when a method is not implemented and it can't be goes into notification log. Do we have here different use cases as well? If yes then your point is right to rethrow it. Also if you rethrow where it will be catched? Issue Time Tracking --- Worklog Id: (was: 854894) Time Spent: 0.5h (was: 20m) > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27165) PART_COL_STATS metastore query not hitting the index
[ https://issues.apache.org/jira/browse/HIVE-27165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongdan Zhu updated HIVE-27165: --- Fix Version/s: 4.0.0 > PART_COL_STATS metastore query not hitting the index > > > Key: HIVE-27165 > URL: https://issues.apache.org/jira/browse/HIVE-27165 > Project: Hive > Issue Type: Improvement >Reporter: Hongdan Zhu >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The query located here: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L1029-L1032] > is not hitting an index. The index contains CAT_NAME whereas this query does > not. This was a change made in Hive 3.0, I think. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27165) PART_COL_STATS metastore query not hitting the index
[ https://issues.apache.org/jira/browse/HIVE-27165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongdan Zhu resolved HIVE-27165. Resolution: Fixed > PART_COL_STATS metastore query not hitting the index > > > Key: HIVE-27165 > URL: https://issues.apache.org/jira/browse/HIVE-27165 > Project: Hive > Issue Type: Improvement >Reporter: Hongdan Zhu >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The query located here: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L1029-L1032] > is not hitting an index. The index contains CAT_NAME whereas this query does > not. This was a change made in Hive 3.0, I think. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27165) PART_COL_STATS metastore query not hitting the index
[ https://issues.apache.org/jira/browse/HIVE-27165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongdan Zhu reassigned HIVE-27165: -- Assignee: Hongdan Zhu > PART_COL_STATS metastore query not hitting the index > > > Key: HIVE-27165 > URL: https://issues.apache.org/jira/browse/HIVE-27165 > Project: Hive > Issue Type: Improvement >Reporter: Hongdan Zhu >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The query located here: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L1029-L1032] > is not hitting an index. The index contains CAT_NAME whereas this query does > not. This was a change made in Hive 3.0, I think. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27165) PART_COL_STATS metastore query not hitting the index
[ https://issues.apache.org/jira/browse/HIVE-27165?focusedWorklogId=854893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854893 ] ASF GitHub Bot logged work on HIVE-27165: - Author: ASF GitHub Bot Created on: 04/Apr/23 20:47 Start Date: 04/Apr/23 20:47 Worklog Time Spent: 10m Work Description: nrg4878 merged PR #4141: URL: https://github.com/apache/hive/pull/4141 Issue Time Tracking --- Worklog Id: (was: 854893) Time Spent: 1.5h (was: 1h 20m) > PART_COL_STATS metastore query not hitting the index > > > Key: HIVE-27165 > URL: https://issues.apache.org/jira/browse/HIVE-27165 > Project: Hive > Issue Type: Improvement >Reporter: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The query located here: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L1029-L1032] > is not hitting an index. The index contains CAT_NAME whereas this query does > not. This was a change made in Hive 3.0, I think. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?focusedWorklogId=854892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854892 ] ASF GitHub Bot logged work on HIVE-27184: - Author: ASF GitHub Bot Created on: 04/Apr/23 20:43 Start Date: 04/Apr/23 20:43 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4196: URL: https://github.com/apache/hive/pull/4196#issuecomment-1496581868 LGTM +1 Issue Time Tracking --- Worklog Id: (was: 854892) Time Spent: 0.5h (was: 20m) > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance, pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27206) Backport of HIVE-20179 Some Tez jar-s are not on classpath so HS2 keeps too long to start
[ https://issues.apache.org/jira/browse/HIVE-27206?focusedWorklogId=854887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854887 ] ASF GitHub Bot logged work on HIVE-27206: - Author: ASF GitHub Bot Created on: 04/Apr/23 19:50 Start Date: 04/Apr/23 19:50 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4182: URL: https://github.com/apache/hive/pull/4182#issuecomment-1496510319 @vihangk1 Can you please merge this. Issue Time Tracking --- Worklog Id: (was: 854887) Time Spent: 40m (was: 0.5h) > Backport of HIVE-20179 Some Tez jar-s are not on classpath so HS2 keeps too > long to start > - > > Key: HIVE-27206 > URL: https://issues.apache.org/jira/browse/HIVE-27206 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HIVE-20179 was already part of Hive 3.1.3 release so it makes sense to > backport this to branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?focusedWorklogId=854885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854885 ] ASF GitHub Bot logged work on HIVE-27184: - Author: ASF GitHub Bot Created on: 04/Apr/23 19:36 Start Date: 04/Apr/23 19:36 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4196: URL: https://github.com/apache/hive/pull/4196#issuecomment-1496495193 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4196) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=VULNERABILITY) [1 Vulnerability](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4196&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4196&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4196&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4196&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4196&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4196&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854885) Time Spent: 20m (was: 10m) > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27184 started by Dmitriy Fingerman. > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?focusedWorklogId=854882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854882 ] ASF GitHub Bot logged work on HIVE-27217: - Author: ASF GitHub Bot Created on: 04/Apr/23 19:03 Start Date: 04/Apr/23 19:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4197: URL: https://github.com/apache/hive/pull/4197#issuecomment-1496459452 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4197) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4197&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4197&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4197&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4197&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4197&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4197&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854882) Time Spent: 20m (was: 10m) > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27127) Upgrade jackson version to 2.13.5
[ https://issues.apache.org/jira/browse/HIVE-27127?focusedWorklogId=854881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854881 ] ASF GitHub Bot logged work on HIVE-27127: - Author: ASF GitHub Bot Created on: 04/Apr/23 19:01 Start Date: 04/Apr/23 19:01 Worklog Time Spent: 10m Work Description: shreeyasand commented on PR #4102: URL: https://github.com/apache/hive/pull/4102#issuecomment-1496456416 > Same question do you need help to merge this? :) Sure, that would be great. Issue Time Tracking --- Worklog Id: (was: 854881) Time Spent: 1h (was: 50m) > Upgrade jackson version to 2.13.5 > - > > Key: HIVE-27127 > URL: https://issues.apache.org/jira/browse/HIVE-27127 > Project: Hive > Issue Type: Improvement >Reporter: Sand Shreeya >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27217: -- Labels: pull-request-available (was: ) > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?focusedWorklogId=854867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854867 ] ASF GitHub Bot logged work on HIVE-27217: - Author: ASF GitHub Bot Created on: 04/Apr/23 17:14 Start Date: 04/Apr/23 17:14 Worklog Time Spent: 10m Work Description: jfsii opened a new pull request, #4197: URL: https://github.com/apache/hive/pull/4197 …atch ### What changes were proposed in this pull request? Rethrow an unhandled exception in addWriteNotificationLogInBatch to make failures much more visible. ### Why are the changes needed? If we don't re-throw the exception, a failure to write to the notification log could be hard to notice. It is very hard to debug after the fact. ### Does this PR introduce _any_ user-facing change? In very rare cases addWriteNotificationLogInBatch failures will become visible when they were not. ### How was this patch tested? Local dev environment against a HMS implementation that throw an TApplicationException that addWriteNotificationLogInBatch did not handle. Issue Time Tracking --- Worklog Id: (was: 854867) Remaining Estimate: 0h Time Spent: 10m > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[ https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854865 ] ASF GitHub Bot logged work on HIVE-27187: - Author: ASF GitHub Bot Created on: 04/Apr/23 17:08 Start Date: 04/Apr/23 17:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4166: URL: https://github.com/apache/hive/pull/4166#issuecomment-1496319047 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4166) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4166&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4166&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4166&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4166&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4166&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4166&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854865) Time Spent: 3h 20m (was: 3h 10m) > Incremental rebuild of materialized view having aggregate and stored by > iceberg > --- > > Key: HIVE-27187 > URL: https://issues.apache.org/jira/browse/HIVE-27187 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration, Materialized views >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently incremental rebuild of materialized view stored by iceberg which > definition query contains aggregate operator is transformed to an insert > overwrite statement which contains a union operator if the source tables > contains insert operations only. One branch of the union scans the view the > other produces the delta. > This can be improved further: transform the statement to a multi insert > statement representing a merge statement to insert new aggregations and
[jira] [Assigned] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-27217: --- > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27189) Remove duplicate debug log in Hive.isSubDIr
[
https://issues.apache.org/jira/browse/HIVE-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shuyouZZ updated HIVE-27189:
Description:
In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
{{isSubDir}} will print twice
{code:java}
LOG.debug("The source path is " + fullF1 + " and the destination path is " +
fullF2);{code}
we should remove the duplicate debug log.
was:
In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
{{isSubDir}} will print twice
{code:java}
LOG.debug("The source path is " + fullF1 + " and the destination path is " +
fullF2);{code}
we should remove the duplicate info log.
> Remove duplicate debug log in Hive.isSubDIr
> ---
>
> Key: HIVE-27189
> URL: https://issues.apache.org/jira/browse/HIVE-27189
> Project: Hive
> Issue Type: Improvement
>Reporter: shuyouZZ
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
> {{isSubDir}} will print twice
> {code:java}
> LOG.debug("The source path is " + fullF1 + " and the destination path is " +
> fullF2);{code}
> we should remove the duplicate debug log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27189) Remove duplicate debug log in Hive.isSubDIr
[
https://issues.apache.org/jira/browse/HIVE-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shuyouZZ updated HIVE-27189:
Summary: Remove duplicate debug log in Hive.isSubDIr (was: Remove
duplicate info log in Hive.isSubDIr)
> Remove duplicate debug log in Hive.isSubDIr
> ---
>
> Key: HIVE-27189
> URL: https://issues.apache.org/jira/browse/HIVE-27189
> Project: Hive
> Issue Type: Improvement
>Reporter: shuyouZZ
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
> {{isSubDir}} will print twice
> {code:java}
> LOG.debug("The source path is " + fullF1 + " and the destination path is " +
> fullF2);{code}
> we should remove the duplicate info log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27189) Remove duplicate info log in Hive.isSubDIr
[
https://issues.apache.org/jira/browse/HIVE-27189?focusedWorklogId=854859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854859
]
ASF GitHub Bot logged work on HIVE-27189:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:39
Start Date: 04/Apr/23 16:39
Worklog Time Spent: 10m
Work Description: shuyouZZ commented on PR #4167:
URL: https://github.com/apache/hive/pull/4167#issuecomment-1496283477
> LGTM The title of the PR is wrong, it is not `info` log it is `debug` log.
Change it to-> `Remove duplicate debug log in Hive.isSubDIr`
Fixed, thanks for your review!
Issue Time Tracking
---
Worklog Id: (was: 854859)
Time Spent: 1h 20m (was: 1h 10m)
> Remove duplicate info log in Hive.isSubDIr
> --
>
> Key: HIVE-27189
> URL: https://issues.apache.org/jira/browse/HIVE-27189
> Project: Hive
> Issue Type: Improvement
>Reporter: shuyouZZ
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
> {{isSubDir}} will print twice
> {code:java}
> LOG.debug("The source path is " + fullF1 + " and the destination path is " +
> fullF2);{code}
> we should remove the duplicate info log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-22383) `alterPartitions` is invoked twice during dynamic partition load causing runtime delay
[
https://issues.apache.org/jira/browse/HIVE-22383?focusedWorklogId=854857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854857
]
ASF GitHub Bot logged work on HIVE-22383:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:37
Start Date: 04/Apr/23 16:37
Worklog Time Spent: 10m
Work Description: ayushtkn merged PR #4161:
URL: https://github.com/apache/hive/pull/4161
Issue Time Tracking
---
Worklog Id: (was: 854857)
Time Spent: 2h (was: 1h 50m)
> `alterPartitions` is invoked twice during dynamic partition load causing
> runtime delay
> --
>
> Key: HIVE-22383
> URL: https://issues.apache.org/jira/browse/HIVE-22383
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Dmitriy Fingerman
>Priority: Major
> Labels: performance, pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> First invocation in {{Hive::loadDynamicPartitions}}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2978
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2638
> Second invocation in {{BasicStatsTask::aggregateStats}}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java#L335
> This leads to good amount of delay in dynamic partition loading.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27184: -- Labels: performance pull-request-available (was: performance) > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-22383) `alterPartitions` is invoked twice during dynamic partition load causing runtime delay
[
https://issues.apache.org/jira/browse/HIVE-22383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17708495#comment-17708495
]
Ayush Saxena commented on HIVE-22383:
-
Committed to master.
Thanx [~difin] for the contribution and [~rajesh.balamohan] for the review!!!
> `alterPartitions` is invoked twice during dynamic partition load causing
> runtime delay
> --
>
> Key: HIVE-22383
> URL: https://issues.apache.org/jira/browse/HIVE-22383
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Dmitriy Fingerman
>Priority: Major
> Labels: performance, pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> First invocation in {{Hive::loadDynamicPartitions}}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2978
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2638
> Second invocation in {{BasicStatsTask::aggregateStats}}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java#L335
> This leads to good amount of delay in dynamic partition loading.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27184) Add class name profiling option in ProfileServlet
[ https://issues.apache.org/jira/browse/HIVE-27184?focusedWorklogId=854858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854858 ] ASF GitHub Bot logged work on HIVE-27184: - Author: ASF GitHub Bot Created on: 04/Apr/23 16:37 Start Date: 04/Apr/23 16:37 Worklog Time Spent: 10m Work Description: difin opened a new pull request, #4196: URL: https://github.com/apache/hive/pull/4196 ### What changes were proposed in this pull request? Adding support for method name profiling to Hive's ProfileServlet. ### Why are the changes needed? Hive has ProfileServlet. It makes use of async-profiler and allows profiling specific events. Currently profileServlet supports events like cpu, alloc, lock etc. It will be good to enhance to support method name profiling as well. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Automated pre-commit tests; Manual testing Issue Time Tracking --- Worklog Id: (was: 854858) Remaining Estimate: 0h Time Spent: 10m > Add class name profiling option in ProfileServlet > - > > Key: HIVE-27184 > URL: https://issues.apache.org/jira/browse/HIVE-27184 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Dmitriy Fingerman >Priority: Major > Labels: performance > Time Spent: 10m > Remaining Estimate: 0h > > With async-profiler "-e classame.method", it is possible to profile specific > events. Currently profileServlet supports events like cpu, alloc, lock etc. > It will be good to enhance to support method name profiling as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[ https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854854&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854854 ] ASF GitHub Bot logged work on HIVE-27185: - Author: ASF GitHub Bot Created on: 04/Apr/23 16:30 Start Date: 04/Apr/23 16:30 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4165: URL: https://github.com/apache/hive/pull/4165#issuecomment-1496272763 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4165) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4165&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4165&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4165&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4165&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4165&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4165&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854854) Time Spent: 2h 50m (was: 2h 40m) > Iceberg: Cache iceberg table while loading for stats > > > Key: HIVE-27185 > URL: https://issues.apache.org/jira/browse/HIVE-27185 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Presently iceberg for stats loads the iceberg table multiple times for stats > via different routes. > Cache it to avoid reading/loading the iceberg table multiple times. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-25799) Maven should stop early for unsupported java versions
[ https://issues.apache.org/jira/browse/HIVE-25799?focusedWorklogId=854850&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854850 ] ASF GitHub Bot logged work on HIVE-25799: - Author: ASF GitHub Bot Created on: 04/Apr/23 16:17 Start Date: 04/Apr/23 16:17 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #2866: URL: https://github.com/apache/hive/pull/2866#issuecomment-1496255102 @ayushtkn @abstractdog any feedback? Issue Time Tracking --- Worklog Id: (was: 854850) Time Spent: 3h 10m (was: 3h) > Maven should stop early for unsupported java versions > - > > Key: HIVE-25799 > URL: https://issues.apache.org/jira/browse/HIVE-25799 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Trivial > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > I made the mistake of using my default java compiler (which apparently is 17) > to build hive, and it failed. If hive only supports a certain version (or > ranges of versions) of java, it would be incredibly helpful if maven reported > this eagerly. > This is easily achievable using the maven enforcer system (a number of > enforcers are already deployed in the top level pom): > [https://maven.apache.org/enforcer/enforcer-rules/requireJavaVersion.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[
https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=854847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854847
]
ASF GitHub Bot logged work on HIVE-26400:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:05
Start Date: 04/Apr/23 16:05
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157465223
##
packaging/src/docker/README.md:
##
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide
the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+ Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading,
writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of
the dependent, as well as build from source.
+
+# Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+# Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop
+-tez
+-hive
+```
+If the version is not provided, it will read the version from current
`pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez
respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version`
and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local
`apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't
exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by
default, the image is tagged by the provided Hive version.
+
+ Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION`
first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate
-Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name
metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive
table's schema and data, start the container with an external Postgres and
Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
@dengzhhu653 where is the driver (jar) coming from?
I think would be great to have a separate docker image for HMS as it's an
independent component (Data Catalog) that could be used by external apps. Note:
that could be done in another PR.
Issue Time Tracking
---
Worklog Id: (was: 854847)
Time Spent: 13h (was: 12h 50m)
> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
> Issue Type: Sub-task
> Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
> Labels: hive-4.0.0-must, pull-request-available
> Time Spent: 13h
> Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed
> mode, with MySQL/Derby as its back database, provide the following:
> * Quick-start/Debugging/Prepare a test env for Hive;
> * Tools to build target image with specified version of Hive and its
> dependencies;
> * Images can be used as the basis for the Kubernetes operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854845
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:02
Start Date: 04/Apr/23 16:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157469710
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
Renamed
Issue Time Tracking
---
Worklog Id: (was: 854845)
Time Spent: 3h 10m (was: 3h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854844
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 16:02
Start Date: 04/Apr/23 16:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157194454
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
I think this name emphasizes that it is used in case of native a storage
format. The same convention is used in case of `ColumnAppender`.
Issue Time Tracking
---
Worklog Id: (was: 854844)
Time Spent: 3h (was: 2h 50m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[
https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=854842&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854842
]
ASF GitHub Bot logged work on HIVE-26400:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:59
Start Date: 04/Apr/23 15:59
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157465223
##
packaging/src/docker/README.md:
##
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide
the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+ Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading,
writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of
the dependent, as well as build from source.
+
+# Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+# Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop
+-tez
+-hive
+```
+If the version is not provided, it will read the version from current
`pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez
respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version`
and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local
`apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't
exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by
default, the image is tagged by the provided Hive version.
+
+ Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION`
first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate
-Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name
metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive
table's schema and data, start the container with an external Postgres and
Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
where is the driver coming from?
Issue Time Tracking
---
Worklog Id: (was: 854842)
Time Spent: 12h 50m (was: 12h 40m)
> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
> Issue Type: Sub-task
> Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
> Labels: hive-4.0.0-must, pull-request-available
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed
> mode, with MySQL/Derby as its back database, provide the following:
> * Quick-start/Debugging/Prepare a test env for Hive;
> * Tools to build target image with specified version of Hive and its
> dependencies;
> * Images can be used as the basis for the Kubernetes operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-25799) Maven should stop early for unsupported java versions
[ https://issues.apache.org/jira/browse/HIVE-25799?focusedWorklogId=854839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854839 ] ASF GitHub Bot logged work on HIVE-25799: - Author: ASF GitHub Bot Created on: 04/Apr/23 15:37 Start Date: 04/Apr/23 15:37 Worklog Time Spent: 10m Work Description: slachiewicz commented on PR #2866: URL: https://github.com/apache/hive/pull/2866#issuecomment-1496197169 This PR is to block possibility to compile Hive with any Java except 8. Issue Time Tracking --- Worklog Id: (was: 854839) Time Spent: 3h (was: 2h 50m) > Maven should stop early for unsupported java versions > - > > Key: HIVE-25799 > URL: https://issues.apache.org/jira/browse/HIVE-25799 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Trivial > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > I made the mistake of using my default java compiler (which apparently is 17) > to build hive, and it failed. If hive only supports a certain version (or > ranges of versions) of java, it would be incredibly helpful if maven reported > this eagerly. > This is easily achievable using the maven enforcer system (a number of > enforcers are already deployed in the top level pom): > [https://maven.apache.org/enforcer/enforcer-rules/requireJavaVersion.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[ https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=854836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854836 ] ASF GitHub Bot logged work on HIVE-26659: - Author: ASF GitHub Bot Created on: 04/Apr/23 15:32 Start Date: 04/Apr/23 15:32 Worklog Time Spent: 10m Work Description: scarlin-cloudera commented on code in PR #4190: URL: https://github.com/apache/hive/pull/4190#discussion_r1157432947 ## ql/src/test/queries/clientpositive/antijoin2.q: ## @@ -0,0 +1,41 @@ +set hive.merge.nway.joins=false; Review Comment: Can we also add the explain cbo plan and explain plan here? Issue Time Tracking --- Worklog Id: (was: 854836) Time Spent: 40m (was: 0.5h) > TPC-DS query 16, 69, 94 return wrong results. > - > > Key: HIVE-26659 > URL: https://issues.apache.org/jira/browse/HIVE-26659 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0-alpha-2 >Reporter: Sungwoo Park >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join > is set to true. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854835
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:29
Start Date: 04/Apr/23 15:29
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157429628
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -120,7 +120,7 @@ public class HiveIcebergMetaHook implements HiveMetaHook {
static final EnumSet SUPPORTED_ALTER_OPS = EnumSet.of(
AlterTableType.ADDCOLS, AlterTableType.REPLACE_COLUMNS,
AlterTableType.RENAME_COLUMN,
AlterTableType.ADDPROPS, AlterTableType.DROPPROPS,
AlterTableType.SETPARTITIONSPEC,
- AlterTableType.UPDATE_COLUMNS, AlterTableType.SETPARTITIONSPEC,
AlterTableType.EXECUTE);
+ AlterTableType.UPDATE_COLUMNS, AlterTableType.RENAME,
AlterTableType.EXECUTE);
Review Comment:
👍
Issue Time Tracking
---
Worklog Id: (was: 854835)
Time Spent: 2.5h (was: 2h 20m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854834
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:28
Start Date: 04/Apr/23 15:28
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157427849
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
##
@@ -253,11 +253,13 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
boolean renamedTranslatedToExternalTable = rename &&
MetaStoreUtils.isTranslatedToExternalTable(oldt)
&& MetaStoreUtils.isTranslatedToExternalTable(newt);
+ boolean isRenameIcebergTable = rename &&
"ICEBERG".equalsIgnoreCase(newt.getParameters().get("table_type"));
Review Comment:
could we please introduce an enum for the table type
Issue Time Tracking
---
Worklog Id: (was: 854834)
Time Spent: 2h 20m (was: 2h 10m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854833
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:27
Start Date: 04/Apr/23 15:27
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157426593
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaHook.java:
##
@@ -55,6 +55,10 @@ public interface HiveMetaHook {
// if this flag is set to true, the HMS call from
HiveMetaStoreClient#alter_table() will be skipped
String SKIP_METASTORE_ALTER = "skip_metastore_alter";
+ String PRE_RENAME_TABLE_NAME = "pre_rename_table_name";
Review Comment:
maybe "OLD_" prefix would be more generic
Issue Time Tracking
---
Worklog Id: (was: 854833)
Time Spent: 2h 10m (was: 2h)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854832
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:26
Start Date: 04/Apr/23 15:26
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157424903
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##
@@ -286,6 +282,14 @@ public void checkValidity(Configuration conf) throws
HiveException {
validateColumns(getCols(), getPartCols());
}
+ public void validateName(Configuration conf) throws HiveException {
+String name = tTable.getTableName();
+if (null == name || name.length() == 0
Review Comment:
StringUtils.isBlank(name)
Issue Time Tracking
---
Worklog Id: (was: 854832)
Time Spent: 2h (was: 1h 50m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854831&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854831
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:25
Start Date: 04/Apr/23 15:25
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157417211
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -875,10 +876,17 @@ public void alterTable(String catName, String dbName,
String tblName, Table newT
if (newTbl.getParameters() != null) {
newTbl.getParameters().remove(hive_metastoreConstants.DDL_TIME);
}
- newTbl.checkValidity(conf);
if (environmentContext == null) {
environmentContext = new EnvironmentContext();
}
+ if (environmentContext.isSetProperties() &&
AlterTableType.RENAME.toString()
Review Comment:
could we do something like AlterTableType.valueOf(..) =
AlterTableType.RENAME? also why do we need to check
`environmentContext.isSetProperties()`?
Issue Time Tracking
---
Worklog Id: (was: 854831)
Time Spent: 1h 50m (was: 1h 40m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854830
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:20
Start Date: 04/Apr/23 15:20
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157417211
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -875,10 +876,17 @@ public void alterTable(String catName, String dbName,
String tblName, Table newT
if (newTbl.getParameters() != null) {
newTbl.getParameters().remove(hive_metastoreConstants.DDL_TIME);
}
- newTbl.checkValidity(conf);
if (environmentContext == null) {
environmentContext = new EnvironmentContext();
}
+ if (environmentContext.isSetProperties() &&
AlterTableType.RENAME.toString()
Review Comment:
could we do something like AlterTableType.valueOf(..) =
AlterTableType.RENAME?
Issue Time Tracking
---
Worklog Id: (was: 854830)
Time Spent: 1h 40m (was: 1.5h)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854829&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854829
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:18
Start Date: 04/Apr/23 15:18
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157415122
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -875,10 +876,17 @@ public void alterTable(String catName, String dbName,
String tblName, Table newT
if (newTbl.getParameters() != null) {
newTbl.getParameters().remove(hive_metastoreConstants.DDL_TIME);
}
- newTbl.checkValidity(conf);
Review Comment:
is it safe to remove? Hive class method could be used by external clients
Issue Time Tracking
---
Worklog Id: (was: 854829)
Time Spent: 1.5h (was: 1h 20m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854828
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:15
Start Date: 04/Apr/23 15:15
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157410687
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -120,7 +120,7 @@ public class HiveIcebergMetaHook implements HiveMetaHook {
static final EnumSet SUPPORTED_ALTER_OPS = EnumSet.of(
AlterTableType.ADDCOLS, AlterTableType.REPLACE_COLUMNS,
AlterTableType.RENAME_COLUMN,
AlterTableType.ADDPROPS, AlterTableType.DROPPROPS,
AlterTableType.SETPARTITIONSPEC,
- AlterTableType.UPDATE_COLUMNS, AlterTableType.SETPARTITIONSPEC,
AlterTableType.EXECUTE);
+ AlterTableType.UPDATE_COLUMNS, AlterTableType.RENAME,
AlterTableType.EXECUTE);
Review Comment:
It is there in the above line:
` AlterTableType.ADDPROPS, AlterTableType.DROPPROPS,
AlterTableType.SETPARTITIONSPEC,`
It was duplicate in the second line, so just replaced with the new one
Issue Time Tracking
---
Worklog Id: (was: 854828)
Time Spent: 1h 20m (was: 1h 10m)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27208) Iceberg: Add support for rename table
[
https://issues.apache.org/jira/browse/HIVE-27208?focusedWorklogId=854827&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854827
]
ASF GitHub Bot logged work on HIVE-27208:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:12
Start Date: 04/Apr/23 15:12
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4185:
URL: https://github.com/apache/hive/pull/4185#discussion_r1157407192
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -120,7 +120,7 @@ public class HiveIcebergMetaHook implements HiveMetaHook {
static final EnumSet SUPPORTED_ALTER_OPS = EnumSet.of(
AlterTableType.ADDCOLS, AlterTableType.REPLACE_COLUMNS,
AlterTableType.RENAME_COLUMN,
AlterTableType.ADDPROPS, AlterTableType.DROPPROPS,
AlterTableType.SETPARTITIONSPEC,
- AlterTableType.UPDATE_COLUMNS, AlterTableType.SETPARTITIONSPEC,
AlterTableType.EXECUTE);
+ AlterTableType.UPDATE_COLUMNS, AlterTableType.RENAME,
AlterTableType.EXECUTE);
Review Comment:
why remove SETPARTITIONSPEC?
Issue Time Tracking
---
Worklog Id: (was: 854827)
Time Spent: 1h 10m (was: 1h)
> Iceberg: Add support for rename table
> -
>
> Key: HIVE-27208
> URL: https://issues.apache.org/jira/browse/HIVE-27208
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Add support for renaming iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27177) Add alter table...Convert to Iceberg command
[
https://issues.apache.org/jira/browse/HIVE-27177?focusedWorklogId=854826&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854826
]
ASF GitHub Bot logged work on HIVE-27177:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 15:08
Start Date: 04/Apr/23 15:08
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on PR #4155:
URL: https://github.com/apache/hive/pull/4155#issuecomment-1496142392
Could we update the migration q tests and also check the explain output?
Issue Time Tracking
---
Worklog Id: (was: 854826)
Time Spent: 1h 50m (was: 1h 40m)
> Add alter table...Convert to Iceberg command
>
>
> Key: HIVE-27177
> URL: https://issues.apache.org/jira/browse/HIVE-27177
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Add an alter table convert to Iceberg [TBLPROPERTIES('','')] to
> convert exiting external tables to iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27177) Add alter table...Convert to Iceberg command
[
https://issues.apache.org/jira/browse/HIVE-27177?focusedWorklogId=854822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854822
]
ASF GitHub Bot logged work on HIVE-27177:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:59
Start Date: 04/Apr/23 14:59
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4155:
URL: https://github.com/apache/hive/pull/4155#discussion_r1157388147
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/convert/AlterTableConvertOperation.java:
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.table.convert;
+
+
+import org.apache.hadoop.hive.ql.ddl.DDLOperationContext;
+import org.apache.hadoop.hive.ql.ddl.table.AbstractAlterTableOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Partition;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+
+import java.util.Map;
+
+/**
+ * Operation process of ALTER TABLE ... CONVERT command
+ */
+public class AlterTableConvertOperation extends
AbstractAlterTableOperation {
+
+ private enum ConversionFormats {
+ICEBERG("org.apache.iceberg.mr.hive.HiveIcebergStorageHandler");
+
+private final String className;
+
+ConversionFormats(String className) {
+ this.className = className;
+}
+
+public String className() {
+ return className;
+}
+ }
+
+ public AlterTableConvertOperation(DDLOperationContext context,
AlterTableConvertDesc desc) {
+super(context, desc);
+ }
+
+ @Override
+ protected void doAlteration(Table table, Partition partition) throws
HiveException {
+// Add the covert type
+String convertType = desc.getConvertSpec().getTargetType();
+ConversionFormats format = ConversionFormats.valueOf(convertType);
+if
(format.className().equalsIgnoreCase(table.getParameters().get("storage_handler")))
{
Review Comment:
could we please use constant instead of string literal
Issue Time Tracking
---
Worklog Id: (was: 854822)
Time Spent: 1h 40m (was: 1.5h)
> Add alter table...Convert to Iceberg command
>
>
> Key: HIVE-27177
> URL: https://issues.apache.org/jira/browse/HIVE-27177
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Add an alter table convert to Iceberg [TBLPROPERTIES('','')] to
> convert exiting external tables to iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27177) Add alter table...Convert to Iceberg command
[
https://issues.apache.org/jira/browse/HIVE-27177?focusedWorklogId=854821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854821
]
ASF GitHub Bot logged work on HIVE-27177:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:57
Start Date: 04/Apr/23 14:57
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4155:
URL: https://github.com/apache/hive/pull/4155#discussion_r1157386059
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/convert/AlterTableConvertOperation.java:
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.table.convert;
+
+
+import org.apache.hadoop.hive.ql.ddl.DDLOperationContext;
+import org.apache.hadoop.hive.ql.ddl.table.AbstractAlterTableOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Partition;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+
+import java.util.Map;
+
+/**
+ * Operation process of ALTER TABLE ... CONVERT command
+ */
+public class AlterTableConvertOperation extends
AbstractAlterTableOperation {
+
+ private enum ConversionFormats {
+ICEBERG("org.apache.iceberg.mr.hive.HiveIcebergStorageHandler");
Review Comment:
can we add ACID here as well?
Issue Time Tracking
---
Worklog Id: (was: 854821)
Time Spent: 1.5h (was: 1h 20m)
> Add alter table...Convert to Iceberg command
>
>
> Key: HIVE-27177
> URL: https://issues.apache.org/jira/browse/HIVE-27177
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Add an alter table convert to Iceberg [TBLPROPERTIES('','')] to
> convert exiting external tables to iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[
https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=854819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854819
]
ASF GitHub Bot logged work on HIVE-26659:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:54
Start Date: 04/Apr/23 14:54
Worklog Time Spent: 10m
Work Description: scarlin-cloudera commented on code in PR #4190:
URL: https://github.com/apache/hive/pull/4190#discussion_r1157379902
##
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java:
##
@@ -556,6 +556,9 @@ public void process(Object row, int tag) throws
HiveException {
}
} else {
storage[pos] = emptyList;
+ if (pos != 0 && condn[pos - 1].getType() == JoinDesc.ANTI_JOIN) {
Review Comment:
Can we add a comment here? Thanks!
Issue Time Tracking
---
Worklog Id: (was: 854819)
Time Spent: 0.5h (was: 20m)
> TPC-DS query 16, 69, 94 return wrong results.
> -
>
> Key: HIVE-26659
> URL: https://issues.apache.org/jira/browse/HIVE-26659
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sungwoo Park
>Assignee: Seonggon Namgung
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join
> is set to true.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[
https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854804
]
ASF GitHub Bot logged work on HIVE-27185:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:07
Start Date: 04/Apr/23 14:07
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4165:
URL: https://github.com/apache/hive/pull/4165#discussion_r1157306903
##
ql/src/java/org/apache/hadoop/hive/ql/session/SessionStateUtil.java:
##
@@ -115,7 +115,7 @@ public static boolean addCommitInfo(Configuration conf,
String tableName, String
return addResource(conf, COMMIT_INFO_PREFIX + tableName, newCommitInfoMap);
}
- private static Optional getQueryState(Configuration conf) {
+ public static Optional getQueryState(Configuration conf) {
return Optional.ofNullable(SessionState.get())
Review Comment:
could we refactor to:
return Optional.ofNullable(SessionState.get())
.map(ss -> ss.getQueryState(HiveConf.getQueryId(conf)));
HiveConf
public static String getQuertyId(Configuration conf) {
return getVar(conf, HiveConf.ConfVars.HIVEQUERYID, "");
}
see getQueryString
Issue Time Tracking
---
Worklog Id: (was: 854804)
Time Spent: 2h 40m (was: 2.5h)
> Iceberg: Cache iceberg table while loading for stats
>
>
> Key: HIVE-27185
> URL: https://issues.apache.org/jira/browse/HIVE-27185
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Presently iceberg for stats loads the iceberg table multiple times for stats
> via different routes.
> Cache it to avoid reading/loading the iceberg table multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[
https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854803
]
ASF GitHub Bot logged work on HIVE-27185:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:06
Start Date: 04/Apr/23 14:06
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4165:
URL: https://github.com/apache/hive/pull/4165#discussion_r1157306903
##
ql/src/java/org/apache/hadoop/hive/ql/session/SessionStateUtil.java:
##
@@ -115,7 +115,7 @@ public static boolean addCommitInfo(Configuration conf,
String tableName, String
return addResource(conf, COMMIT_INFO_PREFIX + tableName, newCommitInfoMap);
}
- private static Optional getQueryState(Configuration conf) {
+ public static Optional getQueryState(Configuration conf) {
return Optional.ofNullable(SessionState.get())
Review Comment:
could we refactor to:
return Optional.ofNullable(SessionState.get())
.map(ss -> ss.getQueryState(HiveConf.getQueryString(conf)));
HiveConf
public static String getQuertyId(Configuration conf) {
return getVar(conf, HiveConf.ConfVars.HIVEQUERYID, "");
}
see getQueryString
Issue Time Tracking
---
Worklog Id: (was: 854803)
Time Spent: 2.5h (was: 2h 20m)
> Iceberg: Cache iceberg table while loading for stats
>
>
> Key: HIVE-27185
> URL: https://issues.apache.org/jira/browse/HIVE-27185
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Presently iceberg for stats loads the iceberg table multiple times for stats
> via different routes.
> Cache it to avoid reading/loading the iceberg table multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[
https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854801
]
ASF GitHub Bot logged work on HIVE-27185:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 14:05
Start Date: 04/Apr/23 14:05
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4165:
URL: https://github.com/apache/hive/pull/4165#discussion_r1157306903
##
ql/src/java/org/apache/hadoop/hive/ql/session/SessionStateUtil.java:
##
@@ -115,7 +115,7 @@ public static boolean addCommitInfo(Configuration conf,
String tableName, String
return addResource(conf, COMMIT_INFO_PREFIX + tableName, newCommitInfoMap);
}
- private static Optional getQueryState(Configuration conf) {
+ public static Optional getQueryState(Configuration conf) {
return Optional.ofNullable(SessionState.get())
Review Comment:
could we refactor to:
Issue Time Tracking
---
Worklog Id: (was: 854801)
Time Spent: 2h 20m (was: 2h 10m)
> Iceberg: Cache iceberg table while loading for stats
>
>
> Key: HIVE-27185
> URL: https://issues.apache.org/jira/browse/HIVE-27185
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Presently iceberg for stats loads the iceberg table multiple times for stats
> via different routes.
> Cache it to avoid reading/loading the iceberg table multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-25799) Maven should stop early for unsupported java versions
[ https://issues.apache.org/jira/browse/HIVE-25799?focusedWorklogId=854800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854800 ] ASF GitHub Bot logged work on HIVE-25799: - Author: ASF GitHub Bot Created on: 04/Apr/23 14:00 Start Date: 04/Apr/23 14:00 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #2866: URL: https://github.com/apache/hive/pull/2866#issuecomment-1496027053 Yes, we had a discussion about it. Because Java 8 is bit old and not sexy anymore. On the other hand the industry still use it and for some reason the oracle extended the support till 2030 (?) I would also go ahead personally, but the big companies who uses hive are way more conservatives. To move even to JDK 11 for compile it would be make some big pain on them. Even if you are able to somehow compile it (especially for 11 I think it is possible) I don't think the community would accept it. Still to many ppl and enterprises use the java8. Issue Time Tracking --- Worklog Id: (was: 854800) Time Spent: 2h 50m (was: 2h 40m) > Maven should stop early for unsupported java versions > - > > Key: HIVE-25799 > URL: https://issues.apache.org/jira/browse/HIVE-25799 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Trivial > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > I made the mistake of using my default java compiler (which apparently is 17) > to build hive, and it failed. If hive only supports a certain version (or > ranges of versions) of java, it would be incredibly helpful if maven reported > this eagerly. > This is easily achievable using the maven enforcer system (a number of > enforcers are already deployed in the top level pom): > [https://maven.apache.org/enforcer/enforcer-rules/requireJavaVersion.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[
https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854797
]
ASF GitHub Bot logged work on HIVE-27185:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 13:46
Start Date: 04/Apr/23 13:46
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4165:
URL: https://github.com/apache/hive/pull/4165#discussion_r1157282843
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##
@@ -349,6 +351,17 @@ public Map getBasicStatistics(Partish
partish) {
return stats;
}
+ private Table getTable(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+Table table;
+final Optional queryState =
SessionStateUtil.getQueryState(conf);
+if (!queryState.isPresent() || queryState.get().getNumModifiedRows() != 0)
{
Review Comment:
`queryState.get().getNumModifiedRows() > 0`
Issue Time Tracking
---
Worklog Id: (was: 854797)
Time Spent: 2h 10m (was: 2h)
> Iceberg: Cache iceberg table while loading for stats
>
>
> Key: HIVE-27185
> URL: https://issues.apache.org/jira/browse/HIVE-27185
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Presently iceberg for stats loads the iceberg table multiple times for stats
> via different routes.
> Cache it to avoid reading/loading the iceberg table multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27185) Iceberg: Cache iceberg table while loading for stats
[
https://issues.apache.org/jira/browse/HIVE-27185?focusedWorklogId=854793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854793
]
ASF GitHub Bot logged work on HIVE-27185:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 13:40
Start Date: 04/Apr/23 13:40
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4165:
URL: https://github.com/apache/hive/pull/4165#discussion_r1157274432
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##
@@ -315,8 +316,9 @@ public boolean canProvideBasicStatistics() {
@Override
public Map getBasicStatistics(Partish partish) {
org.apache.hadoop.hive.ql.metadata.Table hmsTable = partish.getTable();
-TableDesc tableDesc = Utilities.getTableDesc(hmsTable);
-Table table = Catalogs.loadTable(conf, tableDesc.getProperties());
+Table table;
+// For write queries where rows got modified, don't fetch from cache as
values could have changed.
+table = getTable(hmsTable);
Review Comment:
can we inline the declaration and assignment?
Issue Time Tracking
---
Worklog Id: (was: 854793)
Time Spent: 2h (was: 1h 50m)
> Iceberg: Cache iceberg table while loading for stats
>
>
> Key: HIVE-27185
> URL: https://issues.apache.org/jira/browse/HIVE-27185
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Presently iceberg for stats loads the iceberg table multiple times for stats
> via different routes.
> Cache it to avoid reading/loading the iceberg table multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27032) Introduce liquibase for HMS schema evolution
[ https://issues.apache.org/jira/browse/HIVE-27032?focusedWorklogId=854787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854787 ] ASF GitHub Bot logged work on HIVE-27032: - Author: ASF GitHub Bot Created on: 04/Apr/23 13:03 Start Date: 04/Apr/23 13:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4060: URL: https://github.com/apache/hive/pull/4060#issuecomment-1495935984 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [12 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854787) Time Spent: 1.5h (was: 1h 20m) > Introduce liquibase for HMS schema evolution > > > Key: HIVE-27032 > URL: https://issues.apache.org/jira/browse/HIVE-27032 > Project: Hive > Issue Type: Improvement >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Introduce liquibase, and replace current upgrade procedure with it. > The Schematool CLI API should remain untouched, while under the hood, > liquibase should be used for HMS schema evolution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854783&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854783
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:46
Start Date: 04/Apr/23 12:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157194454
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
I think this name emphasizes that it is used in case of native a storage
format. The same convention is used in case of `ColumnAppender`.
Issue Time Tracking
---
Worklog Id: (was: 854783)
Time Spent: 2h 50m (was: 2h 40m)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27187) Incremental rebuild of materialized view having aggregate and stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-27187?focusedWorklogId=854782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854782
]
ASF GitHub Bot logged work on HIVE-27187:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:36
Start Date: 04/Apr/23 12:36
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4166:
URL: https://github.com/apache/hive/pull/4166#discussion_r1157183277
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/NonNativeMaterializedViewASTBuilder.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild;
+
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.singletonList;
+
+public class NonNativeMaterializedViewASTBuilder extends
MaterializedViewASTBuilder {
Review Comment:
what about FullAcidMaterializedViewASTBuilder? Should it be
`NativeAcidMaterializedViewASTBuilder`?
Issue Time Tracking
---
Worklog Id: (was: 854782)
Time Spent: 2h 40m (was: 2.5h)
> Incremental rebuild of materialized view having aggregate and stored by
> iceberg
> ---
>
> Key: HIVE-27187
> URL: https://issues.apache.org/jira/browse/HIVE-27187
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently incremental rebuild of materialized view stored by iceberg which
> definition query contains aggregate operator is transformed to an insert
> overwrite statement which contains a union operator if the source tables
> contains insert operations only. One branch of the union scans the view the
> other produces the delta.
> This can be improved further: transform the statement to a multi insert
> statement representing a merge statement to insert new aggregations and
> update existing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27135) AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in HDFS
[
https://issues.apache.org/jira/browse/HIVE-27135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-27135.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in
> HDFS
> ---
>
> Key: HIVE-27135
> URL: https://issues.apache.org/jira/browse/HIVE-27135
> Project: Hive
> Issue Type: Bug
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> AcidUtils#getHdfsDirSnapshots() throws FileNotFoundException when a directory
> is removed in HDFS while fetching HDFS Snapshots.
> Below testcode can be used to reproduce this issue.
> {code:java}
> @Test
> public void
> testShouldNotThrowFNFEWhenHiveStagingDirectoryIsRemovedWhileFetchingHDFSSnapshots()
> throws Exception {
> MockFileSystem fs = new MockFileSystem(new HiveConf(),
> new MockFile("mock:/tbl/part1/.hive-staging_dir/-ext-10002", 500, new
> byte[0]),
> new MockFile("mock:/tbl/part2/.hive-staging_dir", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/_tmp_space.db", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/delta_1_1/bucket--", 500, new
> byte[0]));
> Path path = new MockPath(fs, "/tbl");
> Path stageDir = new MockPath(fs, "mock:/tbl/part1/.hive-staging_dir");
> FileSystem mockFs = spy(fs);
> Mockito.doThrow(new
> FileNotFoundException("")).when(mockFs).listLocatedStatus(eq(stageDir));
> try {
> Map hdfsDirSnapshots =
> AcidUtils.getHdfsDirSnapshots(mockFs, path);
> Assert.assertEquals(1, hdfsDirSnapshots.size());
> }
> catch (FileNotFoundException fnf) {
> fail("Should not throw FileNotFoundException when a directory is
> removed while fetching HDFSSnapshots");
> }
> }{code}
> This issue got fixed as a part of HIVE-26481 but here its not fixed
> completely.
> [Here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1541]
> FileUtils.listFiles() API which returns a RemoteIterator.
> So while iterating over, it checks if it is a directory and recursive listing
> then it will try to list files from that directory but if that directory is
> removed by other thread/task then it throws FileNotFoundException. Here the
> directory which got removed is the .staging directory which needs to be
> excluded through by using passed filter.
>
> So here we can use same logic written in
> _org.apache.hadoop.hive.ql.io.AcidUtils#getHdfsDirSnapshotsForCleaner()_ API
> to avoid FileNotFoundException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27135) AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in HDFS
[
https://issues.apache.org/jira/browse/HIVE-27135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17708406#comment-17708406
]
Denys Kuzmenko commented on HIVE-27135:
---
Merged to master.
[~Dayakar], thank you for the patch and [~lvegh], [~sbadhya] for the review!
> AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in
> HDFS
> ---
>
> Key: HIVE-27135
> URL: https://issues.apache.org/jira/browse/HIVE-27135
> Project: Hive
> Issue Type: Bug
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> AcidUtils#getHdfsDirSnapshots() throws FileNotFoundException when a directory
> is removed in HDFS while fetching HDFS Snapshots.
> Below testcode can be used to reproduce this issue.
> {code:java}
> @Test
> public void
> testShouldNotThrowFNFEWhenHiveStagingDirectoryIsRemovedWhileFetchingHDFSSnapshots()
> throws Exception {
> MockFileSystem fs = new MockFileSystem(new HiveConf(),
> new MockFile("mock:/tbl/part1/.hive-staging_dir/-ext-10002", 500, new
> byte[0]),
> new MockFile("mock:/tbl/part2/.hive-staging_dir", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/_tmp_space.db", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/delta_1_1/bucket--", 500, new
> byte[0]));
> Path path = new MockPath(fs, "/tbl");
> Path stageDir = new MockPath(fs, "mock:/tbl/part1/.hive-staging_dir");
> FileSystem mockFs = spy(fs);
> Mockito.doThrow(new
> FileNotFoundException("")).when(mockFs).listLocatedStatus(eq(stageDir));
> try {
> Map hdfsDirSnapshots =
> AcidUtils.getHdfsDirSnapshots(mockFs, path);
> Assert.assertEquals(1, hdfsDirSnapshots.size());
> }
> catch (FileNotFoundException fnf) {
> fail("Should not throw FileNotFoundException when a directory is
> removed while fetching HDFSSnapshots");
> }
> }{code}
> This issue got fixed as a part of HIVE-26481 but here its not fixed
> completely.
> [Here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1541]
> FileUtils.listFiles() API which returns a RemoteIterator.
> So while iterating over, it checks if it is a directory and recursive listing
> then it will try to list files from that directory but if that directory is
> removed by other thread/task then it throws FileNotFoundException. Here the
> directory which got removed is the .staging directory which needs to be
> excluded through by using passed filter.
>
> So here we can use same logic written in
> _org.apache.hadoop.hive.ql.io.AcidUtils#getHdfsDirSnapshotsForCleaner()_ API
> to avoid FileNotFoundException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27164) Create Temp Txn Table As Select is failing at tablePath validation
[
https://issues.apache.org/jira/browse/HIVE-27164?focusedWorklogId=854781&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854781
]
ASF GitHub Bot logged work on HIVE-27164:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:32
Start Date: 04/Apr/23 12:32
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on PR #4176:
URL: https://github.com/apache/hive/pull/4176#issuecomment-1495892917
@VenuReddy2103 The sole test failure is probably due to your change. Can you
try to reproduce it locally?
I have seen this test failures in your other runs as well.
`Client Execution succeeded but contained differences (error code = 1) after
executing cte_mat_4.q
132a133
> 1 A
Stacktrace
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing cte_mat_4.q
132a133
> 1 A`
Issue Time Tracking
---
Worklog Id: (was: 854781)
Time Spent: 2h 40m (was: 2.5h)
> Create Temp Txn Table As Select is failing at tablePath validation
> --
>
> Key: HIVE-27164
> URL: https://issues.apache.org/jira/browse/HIVE-27164
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Reporter: Naresh P R
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Attachments: mm_cttas.q
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> After HIVE-25303, every CTAS goes for
> HiveMetaStore$HMSHandler#translate_table_dryrun() call to fetch table
> location for CTAS queries which fails with following exception for temp
> tables if MetastoreDefaultTransformer is set.
> {code:java}
> 2023-03-17 16:41:23,390 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-196]: Starting translation for CreateTable for processor
> HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD,
> HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS,
> HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT,
> HIVEONLYMQTWRITE] on table test_temp
> 2023-03-17 16:41:23,392 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-196]:
> MetaException(message:Illegal location for managed table, it has to be within
> database's managed location)
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.validateTablePaths(MetastoreDefaultTransformer.java:886)
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:666)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) {code}
> I am able to repro this issue using attached testcase. [^mm_cttas.q]
>
> There are multiple ways to fix this issue
> * Have temp txn table path under db's managed location path. This will help
> with encryption zone paths as well.
> * Skip location check for temp tables at
> MetastoreDefaultTransformer#validateTablePaths()
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27135) AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in HDFS
[
https://issues.apache.org/jira/browse/HIVE-27135?focusedWorklogId=854780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854780
]
ASF GitHub Bot logged work on HIVE-27135:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 12:31
Start Date: 04/Apr/23 12:31
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #4114:
URL: https://github.com/apache/hive/pull/4114
Issue Time Tracking
---
Worklog Id: (was: 854780)
Time Spent: 7.5h (was: 7h 20m)
> AcidUtils#getHdfsDirSnapshots() throws FNFE when a directory is removed in
> HDFS
> ---
>
> Key: HIVE-27135
> URL: https://issues.apache.org/jira/browse/HIVE-27135
> Project: Hive
> Issue Type: Bug
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> AcidUtils#getHdfsDirSnapshots() throws FileNotFoundException when a directory
> is removed in HDFS while fetching HDFS Snapshots.
> Below testcode can be used to reproduce this issue.
> {code:java}
> @Test
> public void
> testShouldNotThrowFNFEWhenHiveStagingDirectoryIsRemovedWhileFetchingHDFSSnapshots()
> throws Exception {
> MockFileSystem fs = new MockFileSystem(new HiveConf(),
> new MockFile("mock:/tbl/part1/.hive-staging_dir/-ext-10002", 500, new
> byte[0]),
> new MockFile("mock:/tbl/part2/.hive-staging_dir", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/_tmp_space.db", 500, new byte[0]),
> new MockFile("mock:/tbl/part1/delta_1_1/bucket--", 500, new
> byte[0]));
> Path path = new MockPath(fs, "/tbl");
> Path stageDir = new MockPath(fs, "mock:/tbl/part1/.hive-staging_dir");
> FileSystem mockFs = spy(fs);
> Mockito.doThrow(new
> FileNotFoundException("")).when(mockFs).listLocatedStatus(eq(stageDir));
> try {
> Map hdfsDirSnapshots =
> AcidUtils.getHdfsDirSnapshots(mockFs, path);
> Assert.assertEquals(1, hdfsDirSnapshots.size());
> }
> catch (FileNotFoundException fnf) {
> fail("Should not throw FileNotFoundException when a directory is
> removed while fetching HDFSSnapshots");
> }
> }{code}
> This issue got fixed as a part of HIVE-26481 but here its not fixed
> completely.
> [Here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1541]
> FileUtils.listFiles() API which returns a RemoteIterator.
> So while iterating over, it checks if it is a directory and recursive listing
> then it will try to list files from that directory but if that directory is
> removed by other thread/task then it throws FileNotFoundException. Here the
> directory which got removed is the .staging directory which needs to be
> excluded through by using passed filter.
>
> So here we can use same logic written in
> _org.apache.hadoop.hive.ql.io.AcidUtils#getHdfsDirSnapshotsForCleaner()_ API
> to avoid FileNotFoundException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-25799) Maven should stop early for unsupported java versions
[ https://issues.apache.org/jira/browse/HIVE-25799?focusedWorklogId=854778&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854778 ] ASF GitHub Bot logged work on HIVE-25799: - Author: ASF GitHub Bot Created on: 04/Apr/23 12:27 Start Date: 04/Apr/23 12:27 Worklog Time Spent: 10m Work Description: jsoref commented on PR #2866: URL: https://github.com/apache/hive/pull/2866#issuecomment-1495887185 Not waste time trying to build with a different version./ Get a prettier error message that makes it clear which compiler is supported. Personally, I'd rather it work with 11 or 17, but that wasn't the case and it was an expensive error. Issue Time Tracking --- Worklog Id: (was: 854778) Time Spent: 2h 40m (was: 2.5h) > Maven should stop early for unsupported java versions > - > > Key: HIVE-25799 > URL: https://issues.apache.org/jira/browse/HIVE-25799 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Trivial > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > I made the mistake of using my default java compiler (which apparently is 17) > to build hive, and it failed. If hive only supports a certain version (or > ranges of versions) of java, it would be incredibly helpful if maven reported > this eagerly. > This is easily achievable using the maven enforcer system (a number of > enforcers are already deployed in the top level pom): > [https://maven.apache.org/enforcer/enforcer-rules/requireJavaVersion.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[ https://issues.apache.org/jira/browse/HIVE-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshat Mathur updated HIVE-27203: - Affects Version/s: 4.0.0 > Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, > and combination Tables > -- > > Key: HIVE-27203 > URL: https://issues.apache.org/jira/browse/HIVE-27203 > Project: Hive > Issue Type: Test >Affects Versions: 4.0.0 >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Improve Qtest Coverage for Compaction use cases for ACID Tables: > # Partitioned Tables( Major & Minor ) > # Insert-Only Clustered( Major & Minor ) > # Insert-Only Partitioned( Major & Minor ) > # Insert-Only Clustered and Partitioned( Major & Minor ) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[ https://issues.apache.org/jira/browse/HIVE-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27203 started by Akshat Mathur. > Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, > and combination Tables > -- > > Key: HIVE-27203 > URL: https://issues.apache.org/jira/browse/HIVE-27203 > Project: Hive > Issue Type: Test >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Improve Qtest Coverage for Compaction use cases for ACID Tables: > # Partitioned Tables( Major & Minor ) > # Insert-Only Clustered( Major & Minor ) > # Insert-Only Partitioned( Major & Minor ) > # Insert-Only Clustered and Partitioned( Major & Minor ) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27203) Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, and combination Tables
[ https://issues.apache.org/jira/browse/HIVE-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshat Mathur updated HIVE-27203: - Status: Patch Available (was: In Progress) > Add compaction pending Qtest for Insert-only, Partitioned, Clustered ACID, > and combination Tables > -- > > Key: HIVE-27203 > URL: https://issues.apache.org/jira/browse/HIVE-27203 > Project: Hive > Issue Type: Test >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Improve Qtest Coverage for Compaction use cases for ACID Tables: > # Partitioned Tables( Major & Minor ) > # Insert-Only Clustered( Major & Minor ) > # Insert-Only Partitioned( Major & Minor ) > # Insert-Only Clustered and Partitioned( Major & Minor ) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27164) Create Temp Txn Table As Select is failing at tablePath validation
[
https://issues.apache.org/jira/browse/HIVE-27164?focusedWorklogId=854771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854771
]
ASF GitHub Bot logged work on HIVE-27164:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 11:25
Start Date: 04/Apr/23 11:25
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4176:
URL: https://github.com/apache/hive/pull/4176#issuecomment-149589
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4176)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4176&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4176&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4176&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=CODE_SMELL)
[4 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4176&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4176&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4176&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 854771)
Time Spent: 2.5h (was: 2h 20m)
> Create Temp Txn Table As Select is failing at tablePath validation
> --
>
> Key: HIVE-27164
> URL: https://issues.apache.org/jira/browse/HIVE-27164
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Reporter: Naresh P R
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Attachments: mm_cttas.q
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> After HIVE-25303, every CTAS goes for
> HiveMetaStore$HMSHandler#translate_table_dryrun() call to fetch table
> location for CTAS queries which fails with following exception for temp
> tables if MetastoreDefaultTransformer is set.
> {code:java}
> 2023-03-17 16:41:23,390 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-196]: Starting translation for CreateTable for processor
> HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD,
> H
[jira] [Work logged] (HIVE-26984) Deprecate public HiveConf constructors
[ https://issues.apache.org/jira/browse/HIVE-26984?focusedWorklogId=854757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854757 ] ASF GitHub Bot logged work on HIVE-26984: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:59 Start Date: 04/Apr/23 10:59 Worklog Time Spent: 10m Work Description: abstractdog commented on PR #3983: URL: https://github.com/apache/hive/pull/3983#issuecomment-1495767703 @TuroczyX : actually deprecating some constructors is just a prerequisite for HIVE-26985, which I'm discovering to be solved with AspectJ, so this change is not urgent anymore in general, hiding public constructor might make sense in some circumstances, but now I'm going a different way, so it's not a big deal if we don't ship this in 4.0 Issue Time Tracking --- Worklog Id: (was: 854757) Time Spent: 2h (was: 1h 50m) > Deprecate public HiveConf constructors > -- > > Key: HIVE-26984 > URL: https://issues.apache.org/jira/browse/HIVE-26984 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > From time to time we investigate configuration object problems that are hard > to investigate. We can improve this area, e.g. with HIVE-26985, but first, we > need to introduce a public static factory method to hook into the creation > process. I can see this pattern in another projects as well, like: > HBaseConfiguration. > Creating custom HiveConf subclasses can be useful because putting optional > (say: if else branches or whatever) stuff into the original HiveConf object's > hot codepaths can turn it less performant instantly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27192) Use normal import instead of shaded import in TestSchemaToolCatalogOps.java
[ https://issues.apache.org/jira/browse/HIVE-27192?focusedWorklogId=854751&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854751 ] ASF GitHub Bot logged work on HIVE-27192: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:33 Start Date: 04/Apr/23 10:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4169: URL: https://github.com/apache/hive/pull/4169#issuecomment-1495730038 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4169) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4169&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4169&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4169&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4169&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4169&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4169&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 854751) Time Spent: 2h 20m (was: 2h 10m) > Use normal import instead of shaded import in TestSchemaToolCatalogOps.java > --- > > Key: HIVE-27192 > URL: https://issues.apache.org/jira/browse/HIVE-27192 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27180) Remove JsonSerde from hcatalog, Upgrade should update changed FQN for JsonSerDe in HMS DB
[
https://issues.apache.org/jira/browse/HIVE-27180?focusedWorklogId=854750&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854750
]
ASF GitHub Bot logged work on HIVE-27180:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 10:30
Start Date: 04/Apr/23 10:30
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4159:
URL: https://github.com/apache/hive/pull/4159#issuecomment-1495726477
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4159)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4159&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4159&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4159&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4159&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4159&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4159&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 854750)
Time Spent: 2h 20m (was: 2h 10m)
> Remove JsonSerde from hcatalog, Upgrade should update changed FQN for
> JsonSerDe in HMS DB
> --
>
> Key: HIVE-27180
> URL: https://issues.apache.org/jira/browse/HIVE-27180
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> As Hcatalog JsonSerDe uses the "serde2" version as a back end, Remove
> o{*}rg.apache.hive.hcatalog.data.JsonSerDe{*} from hive-hcatalog. Fix tests
> to use the new Serde class org.apache.hadoop.hive.serde2.JsonSerDe. Hive
> Upgrade schema script can update the SERDES table to alter the class name to
> the new class name, the old tables would work automatically.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27145) Use StrictMath for remaining Math functions as followup of HIVE-23133
[
https://issues.apache.org/jira/browse/HIVE-27145?focusedWorklogId=854749&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854749
]
ASF GitHub Bot logged work on HIVE-27145:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 10:29
Start Date: 04/Apr/23 10:29
Worklog Time Spent: 10m
Work Description: TuroczyX commented on PR #4122:
URL: https://github.com/apache/hive/pull/4122#issuecomment-1495724690
Imho the precision is more important in big data world then performance.
Maybe I think wrongly, but let's imagine a dataset with full of decimals where
I as a customer would like to process a cos or degrees calculation on a TB
scale. I would rather spend more CPU times on get precise value then to have
just a good enough value. I would sacrifice the performance here.
Issue Time Tracking
---
Worklog Id: (was: 854749)
Time Spent: 50m (was: 40m)
> Use StrictMath for remaining Math functions as followup of HIVE-23133
> -
>
> Key: HIVE-27145
> URL: https://issues.apache.org/jira/browse/HIVE-27145
> Project: Hive
> Issue Type: Task
> Components: UDF
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> [HIVE-23133|https://issues.apache.org/jira/browse/HIVE-23133] started using
> {{StrictMath}} for {{cos, exp, log}} UDFs to fix QTests failing as results
> vary based on hardware when using Math library.
> Follow it up by using {{StrictMath}} for other Math functions that can have
> same impact of underlying hardware namely, {{sin, tan, asin, acos, atan,
> sqrt, pow, cbrt}}.
> [JDK-4477961|https://bugs.openjdk.org/browse/JDK-4477961] (in Java 9) changed
> radians and degrees calculation leading to Q Test failures when tests are run
> on Java 9+, fix such tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27127) Upgrade jackson version to 2.13.5
[ https://issues.apache.org/jira/browse/HIVE-27127?focusedWorklogId=854747&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854747 ] ASF GitHub Bot logged work on HIVE-27127: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:23 Start Date: 04/Apr/23 10:23 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4102: URL: https://github.com/apache/hive/pull/4102#issuecomment-1495717734 Same question do you need help to merge this? :) Issue Time Tracking --- Worklog Id: (was: 854747) Time Spent: 50m (was: 40m) > Upgrade jackson version to 2.13.5 > - > > Key: HIVE-27127 > URL: https://issues.apache.org/jira/browse/HIVE-27127 > Project: Hive > Issue Type: Improvement >Reporter: Sand Shreeya >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27110) Upgrade com.esotericsoftware:Kyro version to 5.2.0
[
https://issues.apache.org/jira/browse/HIVE-27110?focusedWorklogId=854745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854745
]
ASF GitHub Bot logged work on HIVE-27110:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 10:18
Start Date: 04/Apr/23 10:18
Worklog Time Spent: 10m
Work Description: TuroczyX commented on PR #4087:
URL: https://github.com/apache/hive/pull/4087#issuecomment-1495712053
Do you need any help with this?
Issue Time Tracking
---
Worklog Id: (was: 854745)
Time Spent: 0.5h (was: 20m)
> Upgrade com.esotericsoftware:Kyro version to 5.2.0
> --
>
> Key: HIVE-27110
> URL: https://issues.apache.org/jira/browse/HIVE-27110
> Project: Hive
> Issue Type: Task
>Reporter: Indhumathi Muthumurugesh
>Assignee: Indhumathi Muthumurugesh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The kryo package (5.0.3) is vulnerable to Uncaught
> Exceptionconditions([https://cwe.mitre.org/data/definitions/248.html)|https://cwe.mitre.org/data/definitions/248.html).]
> The `{*}readReferenceOrNull(){*}` method in the kryo class fails to process
> exceptions that may occur due to undefined ArrayList references in the
> `{*}RefernceResolver`{*} interface while processing input data during
> serialization or deserialization, which may lead to application crash or
> uninformative exceptions.
>
> The above CWE-248 can be avoided by upgrading the kryo version to 5.2.0,
> which has addressed the issue.
> For Reference:
> [https://github.com/EsotericSoftware/kryo/blob/kryo-parent-5.2.0/src/com/esotericsoftware/kryo/Kryo.java]
> Issue: https://github.com/EsotericSoftware/kryo/issues/829
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24706) Spark SQL access hive on HBase table access exception
[ https://issues.apache.org/jira/browse/HIVE-24706?focusedWorklogId=854744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854744 ] ASF GitHub Bot logged work on HIVE-24706: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:14 Start Date: 04/Apr/23 10:14 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4063: URL: https://github.com/apache/hive/pull/4063#issuecomment-1495706964 I don't get it. As I see there is a test failure in the javadoc part, which seems related. So I would say yes it should be fixed. Issue Time Tracking --- Worklog Id: (was: 854744) Time Spent: 40m (was: 0.5h) > Spark SQL access hive on HBase table access exception > - > > Key: HIVE-24706 > URL: https://issues.apache.org/jira/browse/HIVE-24706 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: zhangzhanchang >Priority: Major > Labels: pull-request-available > Attachments: image-2021-01-30-15-51-58-665.png > > Time Spent: 40m > Remaining Estimate: 0h > > Hivehbasetableinputformat relies on two versions of inputformat，one is > org.apache.hadoop.mapred.InputFormat, the other is > org.apache.hadoop.mapreduce.InputFormat,Causes > spark 3.0(https://github.com/apache/spark/pull/31302) both conditions to be > true: > # classOf[oldInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true > # classOf[newInputClass[_, _]].isAssignableFrom(inputFormatClazz) is true > !image-2021-01-30-15-51-58-665.png|width=430,height=137! > Hivehbasetableinputformat relies on inputformat to be changed to > org.apache.hadoop.mapreduce or org.apache.hadoop.mapred? > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27077) upgrade hive grammar to Antlr4
[ https://issues.apache.org/jira/browse/HIVE-27077?focusedWorklogId=854743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854743 ] ASF GitHub Bot logged work on HIVE-27077: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:11 Start Date: 04/Apr/23 10:11 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4058: URL: https://github.com/apache/hive/pull/4058#issuecomment-1495702194 Sorry for dumb question but where do we use antlr? It is in the query analyzer or in the parser? Also what is the purpose of this library here? (I'm not familiar with that and I'm asking because I'm curious :) ) Does this change introduce any breaking change? As I see from the antlr page it is 20% faster the 4 then the previous which is nice. Also in the hive 4 (4.1?) we are able to introduce if it won't introduce breaking change. But It would be great to have an understating of the risks and motivation. Issue Time Tracking --- Worklog Id: (was: 854743) Time Spent: 1h 50m (was: 1h 40m) > upgrade hive grammar to Antlr4 > -- > > Key: HIVE-27077 > URL: https://issues.apache.org/jira/browse/HIVE-27077 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Introducing new module parser-v4 that hosts hive grammar defined using Antlr4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27068) Use Spark Accumulator V2 instead of the deprecated Accumulator V1
[ https://issues.apache.org/jira/browse/HIVE-27068?focusedWorklogId=854742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854742 ] ASF GitHub Bot logged work on HIVE-27068: - Author: ASF GitHub Bot Created on: 04/Apr/23 10:05 Start Date: 04/Apr/23 10:05 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4052: URL: https://github.com/apache/hive/pull/4052#issuecomment-1495693883 Hi @nineinfra I'm not familiar with Accumulator. (I guess it is a serialization mechanism or belongs to that topic) I have two question with this fix: 1. If we allow to upgrade to AC2 that old version of spark (like spark 2) can use it or it will be incompatible? What would see the user if they start to use? 2. If we do not upgrade to AC2 would be a problem with Spark3? It would be nice if it would be clear that what could be the impact of this change cc: @ayushtkn @veghlaci05 Issue Time Tracking --- Worklog Id: (was: 854742) Time Spent: 20m (was: 10m) > Use Spark Accumulator V2 instead of the deprecated Accumulator V1 > - > > Key: HIVE-27068 > URL: https://issues.apache.org/jira/browse/HIVE-27068 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 3.0.0 >Reporter: Wei Yan >Priority: Major > Labels: pull-request-available > Fix For: 3.1.4 > > Time Spent: 20m > Remaining Estimate: 0h > > The Accumulator V1 has been deprecated since the Spark version 2.0.0 > But in hive 3 series and below versions，the Accumulator V1 is still used. > There will be compatibility problems when the new version of spark is used > with hive, > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27023) Add setting to disable hive session being started during Hive Cli startup
[
https://issues.apache.org/jira/browse/HIVE-27023?focusedWorklogId=854739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854739
]
ASF GitHub Bot logged work on HIVE-27023:
-
Author: ASF GitHub Bot
Created on: 04/Apr/23 09:46
Start Date: 04/Apr/23 09:46
Worklog Time Spent: 10m
Work Description: TuroczyX commented on PR #4015:
URL: https://github.com/apache/hive/pull/4015#issuecomment-1495668957
@abstractdog @ramesh0201 So in this case with this config you can avoid to
have a tez session? Is this safe? I can imagine a use case for it. What do you
think?
Issue Time Tracking
---
Worklog Id: (was: 854739)
Time Spent: 0.5h (was: 20m)
> Add setting to disable hive session being started during Hive Cli startup
> -
>
> Key: HIVE-27023
> URL: https://issues.apache.org/jira/browse/HIVE-27023
> Project: Hive
> Issue Type: Improvement
>Reporter: Alagappan Maruthappan
>Assignee: Alagappan Maruthappan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Added a setting disable Tez session being opened during the startup of Hive
> cli. This is particularly needed when Hive action is being launched by Oozie
> to perform DML only operations. In that cases before the tez session thread
> could complete the launcher gets completed and clears all the scratch
> directories. This leads to a failure with following stack trace:
> {code:java}
> Application application_1667416881396_24229473 failed 3 times due to AM
> Container for appattempt_1667416881396_24229473_03 exited with exitCode:
> -1000
> Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not
> exist:
> hdfs:///tmp//_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException:
> File does not exist:
> hdfs:///tmp//_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat
>
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at
>
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at
> org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)at
> org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)at
> org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)at
> java.security.AccessController.doPrivileged(Native Method)at
> javax.security.auth.Subject.doAs(Subject.java:422)at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2136)at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2116)at
> org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:249)at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:242)at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:230)at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at
> java.lang.Thread.run(Thread.java:748) {code}
> Also opening up of tez session in such cases is not needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26832) Implement SHOW PARTITIONS for Iceberg
[ https://issues.apache.org/jira/browse/HIVE-26832?focusedWorklogId=854738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854738 ] ASF GitHub Bot logged work on HIVE-26832: - Author: ASF GitHub Bot Created on: 04/Apr/23 09:41 Start Date: 04/Apr/23 09:41 Worklog Time Spent: 10m Work Description: simhadri-g commented on PR #3849: URL: https://github.com/apache/hive/pull/3849#issuecomment-1495661835 Yes Attila. Issue Time Tracking --- Worklog Id: (was: 854738) Time Spent: 3h 10m (was: 3h) > Implement SHOW PARTITIONS for Iceberg > - > > Key: HIVE-26832 > URL: https://issues.apache.org/jira/browse/HIVE-26832 > Project: Hive > Issue Type: New Feature >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Show partition command for iceberg tables should reflect the partition info > from the iceberg.partition metadata table based on the default-spec-id . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26984) Deprecate public HiveConf constructors
[ https://issues.apache.org/jira/browse/HIVE-26984?focusedWorklogId=854737&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854737 ] ASF GitHub Bot logged work on HIVE-26984: - Author: ASF GitHub Bot Created on: 04/Apr/23 09:34 Start Date: 04/Apr/23 09:34 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #3983: URL: https://github.com/apache/hive/pull/3983#issuecomment-1495652389 @abstractdog Seems like this ticket is abandoned for a while. I think it is always good to remove deprecated stuff especially before the hive 4 release. Issue Time Tracking --- Worklog Id: (was: 854737) Time Spent: 1h 50m (was: 1h 40m) > Deprecate public HiveConf constructors > -- > > Key: HIVE-26984 > URL: https://issues.apache.org/jira/browse/HIVE-26984 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > From time to time we investigate configuration object problems that are hard > to investigate. We can improve this area, e.g. with HIVE-26985, but first, we > need to introduce a public static factory method to hook into the creation > process. I can see this pattern in another projects as well, like: > HBaseConfiguration. > Creating custom HiveConf subclasses can be useful because putting optional > (say: if else branches or whatever) stuff into the original HiveConf object's > hot codepaths can turn it less performant instantly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26832) Implement SHOW PARTITIONS for Iceberg
[ https://issues.apache.org/jira/browse/HIVE-26832?focusedWorklogId=854736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854736 ] ASF GitHub Bot logged work on HIVE-26832: - Author: ASF GitHub Bot Created on: 04/Apr/23 09:30 Start Date: 04/Apr/23 09:30 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #3849: URL: https://github.com/apache/hive/pull/3849#issuecomment-1495647351 @simhadri-g Do you have a plan to address this comments? Issue Time Tracking --- Worklog Id: (was: 854736) Time Spent: 3h (was: 2h 50m) > Implement SHOW PARTITIONS for Iceberg > - > > Key: HIVE-26832 > URL: https://issues.apache.org/jira/browse/HIVE-26832 > Project: Hive > Issue Type: New Feature >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Show partition command for iceberg tables should reflect the partition info > from the iceberg.partition metadata table based on the default-spec-id . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=854735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854735 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 04/Apr/23 09:28 Start Date: 04/Apr/23 09:28 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1495644501 Yes, it should be unrelated. But it would be also great if we have a green build to follow the industrial standard. (Even logically does not have relation) Btw we need to think about as a community to push hive images into docker hub. That would be super cool. Issue Time Tracking --- Worklog Id: (was: 854735) Time Spent: 12h 40m (was: 12.5h) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 12h 40m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-25799) Maven should stop early for unsupported java versions
[ https://issues.apache.org/jira/browse/HIVE-25799?focusedWorklogId=854734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854734 ] ASF GitHub Bot logged work on HIVE-25799: - Author: ASF GitHub Bot Created on: 04/Apr/23 09:27 Start Date: 04/Apr/23 09:27 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #2866: URL: https://github.com/apache/hive/pull/2866#issuecomment-1495641994 @jsoref With this change what would you like to achieve? Yes, the hive is currently only build with JDK8. But from runtime perspective it is possible to use 11 or even 17th. (Hopefully 20th will come soon) cc: @InvisibleProgrammer Would you like to build with 11? I'm not familiar with this rules, but I can tell we haven't tried to build it with 11. @akshat0395 Issue Time Tracking --- Worklog Id: (was: 854734) Time Spent: 2.5h (was: 2h 20m) > Maven should stop early for unsupported java versions > - > > Key: HIVE-25799 > URL: https://issues.apache.org/jira/browse/HIVE-25799 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Trivial > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > I made the mistake of using my default java compiler (which apparently is 17) > to build hive, and it failed. If hive only supports a certain version (or > ranges of versions) of java, it would be incredibly helpful if maven reported > this eagerly. > This is easily achievable using the maven enforcer system (a number of > enforcers are already deployed in the top level pom): > [https://maven.apache.org/enforcer/enforcer-rules/requireJavaVersion.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
