[jira] [Updated] (HIVE-26667) Incompatible expression deserialization against latest HMS
[
https://issues.apache.org/jira/browse/HIVE-26667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-26667:
---
Description:
When an old Metastore client requests listPartitionsByExpr against the lastest
HMS, an exception would be thrown:
{noformat}
MetaException(message:Unable to find class:
)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_expr(ThriftHiveMetastore.java:3273)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_expr(ThriftHiveMetastore.java:3260)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByExpr(HiveMetaStoreClient.java:1488){noformat}
This was caused by a gap between old client and server on (de)serializing the
expression. Old client doesn't stream the expression’s class type into bytes,
while HMS reads the class type from serialized bytes first, which makes the
trouble. Other APIs that need to (de)serialize expression may be suffered as
well.
was:
When an old Hive Metastore client issues listPartitionsByExpr against the
lastest HMS, an exception would be thrown:
{noformat}
MetaException(message:Unable to find class:
)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_expr(ThriftHiveMetastore.java:3273)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_expr(ThriftHiveMetastore.java:3260)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByExpr(HiveMetaStoreClient.java:1488){noformat}
This was caused by a gap between old client and server on (de)serializing the
expression. In old client, we don’t stream the expression’s class type into
bytes, while the server should read the class type from serialized bytes
firstly, which makes the trouble. Other APIs that need to (de)serialize
expression may be suffered as well.
> Incompatible expression deserialization against latest HMS
> --
>
> Key: HIVE-26667
> URL: https://issues.apache.org/jira/browse/HIVE-26667
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0-alpha-1
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When an old Metastore client requests listPartitionsByExpr against the
> lastest HMS, an exception would be thrown:
>
> {noformat}
> MetaException(message:Unable to find class:
> )
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_expr(ThriftHiveMetastore.java:3273)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_expr(ThriftHiveMetastore.java:3260)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByExpr(HiveMetaStoreClient.java:1488){noformat}
> This was caused by a gap between old client and server on (de)serializing the
> expression. Old client doesn't stream the expression’s
[jira] [Resolved] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks
[
https://issues.apache.org/jira/browse/HIVE-25483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-25483.
-
Resolution: Duplicate
> TxnHandler::acquireLock should close the DB conn to avoid connection leaks
> --
>
> Key: HIVE-25483
> URL: https://issues.apache.org/jira/browse/HIVE-25483
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: KIRTI RUGE
>Priority: Major
>
> TxnHandler::acquireLock should close DB connection on exiting the function.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740]
> If there are any exceptions downstream, this connection isn't closed
> cleanly. In a corner case, hikari connection leak detector reported the
> following
> {noformat}
> 2021-08-26 09:19:18,102 WARN com.zaxxer.hikari.pool.ProxyLeakTask:
> [HikariPool-4 housekeeper]: Connection leak detection triggered for
> org.postgresql.jdbc.PgConnection@77f76747, stack trace follows
> java.lang.Exception: Apparent connection leak detected
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:5135)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:107)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks
[
https://issues.apache.org/jira/browse/HIVE-25483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reopened HIVE-25483:
-
> TxnHandler::acquireLock should close the DB conn to avoid connection leaks
> --
>
> Key: HIVE-25483
> URL: https://issues.apache.org/jira/browse/HIVE-25483
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: KIRTI RUGE
>Priority: Major
>
> TxnHandler::acquireLock should close DB connection on exiting the function.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740]
> If there are any exceptions downstream, this connection isn't closed
> cleanly. In a corner case, hikari connection leak detector reported the
> following
> {noformat}
> 2021-08-26 09:19:18,102 WARN com.zaxxer.hikari.pool.ProxyLeakTask:
> [HikariPool-4 housekeeper]: Connection leak detection triggered for
> org.postgresql.jdbc.PgConnection@77f76747, stack trace follows
> java.lang.Exception: Apparent connection leak detected
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:5135)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:107)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26656) Remove hsqldb dependency in hive due to CVE-2022-41853
[ https://issues.apache.org/jira/browse/HIVE-26656?focusedWorklogId=824820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824820 ] ASF GitHub Bot logged work on HIVE-26656: - Author: ASF GitHub Bot Created on: 10/Nov/22 05:49 Start Date: 10/Nov/22 05:49 Worklog Time Spent: 10m Work Description: saihemanth-cloudera merged PR #3740: URL: https://github.com/apache/hive/pull/3740 Issue Time Tracking --- Worklog Id: (was: 824820) Time Spent: 1h 40m (was: 1.5h) > Remove hsqldb dependency in hive due to CVE-2022-41853 > -- > > Key: HIVE-26656 > URL: https://issues.apache.org/jira/browse/HIVE-26656 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-21599) Remove predicate on partition columns from Table Scan operator
[ https://issues.apache.org/jira/browse/HIVE-21599?focusedWorklogId=824814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824814 ] ASF GitHub Bot logged work on HIVE-21599: - Author: ASF GitHub Bot Created on: 10/Nov/22 04:34 Start Date: 10/Nov/22 04:34 Worklog Time Spent: 10m Work Description: amansinha100 commented on PR #3742: URL: https://github.com/apache/hive/pull/3742#issuecomment-1309760717 A note about the commit message: the summary says 'remove predicate ..' which is not true any more with this patch. So best to reword it. Also remove reference to virtual columns since the patch is not making any changes for that. Issue Time Tracking --- Worklog Id: (was: 824814) Time Spent: 1h 40m (was: 1.5h) > Remove predicate on partition columns from Table Scan operator > -- > > Key: HIVE-21599 > URL: https://issues.apache.org/jira/browse/HIVE-21599 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21599.1.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Filter predicates are pushed to Table Scan (to be pushed to and used by > storage handler/input format). Such predicates could consist of partition > columns which are of no use to storage handler or input formats. Therefore > it should be removed from TS filter expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-21599) Remove predicate on partition columns from Table Scan operator
[
https://issues.apache.org/jira/browse/HIVE-21599?focusedWorklogId=824813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824813
]
ASF GitHub Bot logged work on HIVE-21599:
-
Author: ASF GitHub Bot
Created on: 10/Nov/22 04:32
Start Date: 10/Nov/22 04:32
Worklog Time Spent: 10m
Work Description: amansinha100 commented on code in PR #3742:
URL: https://github.com/apache/hive/pull/3742#discussion_r1018639310
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java:
##
@@ -209,6 +214,22 @@ public FilterCompat.Filter setFilter(final JobConf conf,
MessageType schema) {
}
}
+ private MessageType getSchemaWithoutPartitionColumns(JobConf conf,
MessageType schema) {
+String partCols = conf.get(IOConstants.PARTITION_COLUMNS);
+if (partCols != null && partCols.length() > 0) {
+ Set partitionColumns = new
HashSet<>(Arrays.asList(partCols.split(",")));
+ List newFields = new ArrayList<>();
+
+ for (Type field: schema.getFields()) {
+if(!partitionColumns.contains(field.getName())) {
Review Comment:
Could you confirm if this comparison works in a case-insensitive way ? i.e
if the partition columns is defined as 'part_col' and the query has predicate
on 'PART_COL'.
##
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration
conf,
LOG.info("schema.evolution.columns and schema.evolution.columns.types
not available");
}
}
+ public static void setPartitionColumnsToConf(Configuration conf,
TableScanOperator tableScanOp) {
+TableScanDesc scanDesc = tableScanOp.getConf();
+if (scanDesc != null && scanDesc.getTableMetadata() != null) {
+ List partitionColsList =
scanDesc.getTableMetadata().getPartColNames();
+ if (!partitionColsList.isEmpty()) {
+conf.set(IOConstants.PARTITION_COLUMNS, String.join(",",
partitionColsList));
+ }
+} else {
+ LOG.info(IOConstants.PARTITION_COLUMNS + " not available");
Review Comment:
Is this else condition placed correctly ? Seems like the INFO message for
partition columns not available should be logged when
partitionColsList.isEmpty() returns true.
##
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration
conf,
LOG.info("schema.evolution.columns and schema.evolution.columns.types
not available");
}
}
+ public static void setPartitionColumnsToConf(Configuration conf,
TableScanOperator tableScanOp) {
Review Comment:
nit: pls add a comment for this method. Also, the unset method is named
'InConf' whereas this one has 'ToConf'. Can both set and unset method names
use the same suffix ?
Issue Time Tracking
---
Worklog Id: (was: 824813)
Time Spent: 1.5h (was: 1h 20m)
> Remove predicate on partition columns from Table Scan operator
> --
>
> Key: HIVE-21599
> URL: https://issues.apache.org/jira/browse/HIVE-21599
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21599.1.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Filter predicates are pushed to Table Scan (to be pushed to and used by
> storage handler/input format). Such predicates could consist of partition
> columns which are of no use to storage handler or input formats. Therefore
> it should be removed from TS filter expression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects
[
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631412#comment-17631412
]
Naveen Gangam commented on HIVE-26220:
--
[~ayushtkn] I have marked this jira a must-fix for 4.0.0. Would you have cycles
to take a look at this? Thank you
> Shade & relocate dependencies in hive-exec to avoid conflicting with
> downstream projects
>
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
> Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade
> & relocate dependencies in {{hive-exec}} that conflict with the ones they
> brought in by themselves. However, with {{hive-exec}} this is no longer
> possible, since it is a fat jar that shade those dependencies but do not
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need
> to make sure all the dependencies in {{hive-exec}} are properly shaded and
> relocated, so they won't cause conflicts with those from the downstream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects
[
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-26220:
-
Target Version/s: 4.0.0
> Shade & relocate dependencies in hive-exec to avoid conflicting with
> downstream projects
>
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
> Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade
> & relocate dependencies in {{hive-exec}} that conflict with the ones they
> brought in by themselves. However, with {{hive-exec}} this is no longer
> possible, since it is a fat jar that shade those dependencies but do not
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need
> to make sure all the dependencies in {{hive-exec}} are properly shaded and
> relocated, so they won't cause conflicts with those from the downstream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects
[
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-26220:
-
Labels: hive-4.0.0-must (was: )
> Shade & relocate dependencies in hive-exec to avoid conflicting with
> downstream projects
>
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
> Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade
> & relocate dependencies in {{hive-exec}} that conflict with the ones they
> brought in by themselves. However, with {{hive-exec}} this is no longer
> possible, since it is a fat jar that shade those dependencies but do not
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need
> to make sure all the dependencies in {{hive-exec}} are properly shaded and
> relocated, so they won't cause conflicts with those from the downstream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=824805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824805
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 10/Nov/22 02:19
Start Date: 10/Nov/22 02:19
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3745:
URL: https://github.com/apache/hive/pull/3745#issuecomment-1309677424
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3745)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3745=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3745=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 824805)
Time Spent: 3h 10m (was: 3h)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-26676) Count distinct in subquery returning wrong results
[
https://issues.apache.org/jira/browse/HIVE-26676?focusedWorklogId=824795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824795
]
ASF GitHub Bot logged work on HIVE-26676:
-
Author: ASF GitHub Bot
Created on: 10/Nov/22 01:15
Start Date: 10/Nov/22 01:15
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3734:
URL: https://github.com/apache/hive/pull/3734#issuecomment-1309633160
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3734)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3734=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3734=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 824795)
Time Spent: 1h (was: 50m)
> Count distinct in subquery returning wrong results
> --
>
> Key: HIVE-26676
> URL: https://issues.apache.org/jira/browse/HIVE-26676
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The following select query should return 1 row
> {code:java}
> CREATE TABLE t_test(
> a tinyint
> );
> INSERT INTO t_test VALUES
> (0), (1), (2), (3), (4), (5), (6), (7), (8), (9);
> select 1 from (select count(distinct a) from t_test) x;
> {code}
> But it's returning 10 rows
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-21599) Remove predicate on partition columns from Table Scan operator
[ https://issues.apache.org/jira/browse/HIVE-21599?focusedWorklogId=824773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824773 ] ASF GitHub Bot logged work on HIVE-21599: - Author: ASF GitHub Bot Created on: 09/Nov/22 22:02 Start Date: 09/Nov/22 22:02 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3742: URL: https://github.com/apache/hive/pull/3742#issuecomment-1309435881 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3742) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3742=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3742=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 824773) Time Spent: 1h 20m (was: 1h 10m) > Remove predicate on partition columns from Table Scan operator > -- > > Key: HIVE-21599 > URL: https://issues.apache.org/jira/browse/HIVE-21599 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21599.1.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Filter predicates are pushed to Table Scan (to be pushed to and used by > storage handler/input format). Such predicates could consist of partition > columns which are of no use to storage handler or input formats. Therefore > it should be removed from TS filter expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26692) Check for the expected thrift version before compiling
[ https://issues.apache.org/jira/browse/HIVE-26692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631311#comment-17631311 ] Ayush Saxena commented on HIVE-26692: - That makes sense to me, a wrong version can sometimes lead to unnecessary dev efforts, it would be good if we could fail fast > Check for the expected thrift version before compiling > -- > > Key: HIVE-26692 > URL: https://issues.apache.org/jira/browse/HIVE-26692 > Project: Hive > Issue Type: Task > Components: Thrift API >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Priority: Major > > At the moment we don't check for the thrift version before launching thrift, > the error messages are often cryptic upon mismatches. > An explicit check with a clear error message would be nice, like what parquet > does: > [https://github.com/apache/parquet-mr/blob/master/parquet-thrift/pom.xml#L247-L268] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26662) FAILED: SemanticException [Error 10072]: Database does not exist: global_temp
[
https://issues.apache.org/jira/browse/HIVE-26662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-26662.
-
Resolution: Invalid
Vendor specific question, please reach out to their support
> FAILED: SemanticException [Error 10072]: Database does not exist: global_temp
> -
>
> Key: HIVE-26662
> URL: https://issues.apache.org/jira/browse/HIVE-26662
> Project: Hive
> Issue Type: Bug
>Reporter: Mahmood Abu Awwad
>Priority: Blocker
>
> while running our batches using Apache Spark with Hive on EMR cluster, as
> we're using AWS glue as a MetaStore, it seems there is an issue occurs, which
> is
> {code:java}
> EntityNotFoundException ,Database global_temp not found {code}
> {code:java}
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Completed compiling
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736);
> Time taken: 0.02 seconds
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: reexec.ReExecDriver (:()) - Execution #1 of query
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Executing
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736):
> show views
> 2022-10-09T10:36:31,263 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Starting task [Stage-0:DDL] in serial mode
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Completed executing
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736);
> Time taken: 1.008 seconds
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - OK
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: exec.ListSinkOperator (:()) - RECORDS_OUT_INTERMEDIATE:0,
> RECORDS_OUT_OPERATOR_LIST_SINK_0:0,
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: CliDriver (:()) - Time taken: 1.028 seconds
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5104)) - Using the
> default value passed in for log id: 573c4ce0-f73c-439b-829d-1f0b25db45ec
> 2022-10-09T10:36:32,272 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: session.SessionState (SessionState.java:resetThreadName(452)) -
> Resetting thread name to main
> 2022-10-09T10:36:46,512 INFO [main([])]: conf.HiveConf
> (HiveConf.java:getLogIdVar(5104)) - Using the default value passed in for log
> id: 573c4ce0-f73c-439b-829d-1f0b25db45ec
> 2022-10-09T10:36:46,513 INFO [main([])]: session.SessionState
> (SessionState.java:updateThreadName(441)) - Updating thread name to
> 573c4ce0-f73c-439b-829d-1f0b25db45ec main
> 2022-10-09T10:36:46,515 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Compiling
> command(queryId=hadoop_20221009103646_f390a868-07d7-49f1-b620-70d40e5e2cff):
> use global_temp
> 2022-10-09T10:36:46,530 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:46,666 ERROR [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - FAILED: SemanticException [Error 10072]:
> Database does not exist: global_temp
> org.apache.hadoop.hive.ql.parse.SemanticException: Database does not exist:
> global_temp
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2171)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeSwitchDatabase(DDLSemanticAnalyzer.java:1413)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:516)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
>
[jira] [Commented] (HIVE-26700) alter view set owner is throwing exception
[
https://issues.apache.org/jira/browse/HIVE-26700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631302#comment-17631302
]
Ayush Saxena commented on HIVE-26700:
-
try via TBLPROPERTIES setting the value of owner, this command isn't supported
now
> alter view set owner is throwing exception
> --
>
> Key: HIVE-26700
> URL: https://issues.apache.org/jira/browse/HIVE-26700
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Priority: Major
>
> Tested in 3.1, we can alter the view with alter table command itself
> desc formatted test_view_owner;
> {noformat}
> +---++---+
> | col_name | data_type
> | comment |
> +---++---+
> | # col_name | data_type
> | comment |
> | id | int
> | |
> | name | string
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | OwnerType: | USER
> | NULL |
> | Owner: | hive
> | NULL |
> | CreateTime: | Tue Oct 25 20:40:47 GMT+08:00 2022
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Retention: | 0
> | NULL |
> | Table Type: | VIRTUAL_VIEW
> | NULL |
> | Table Parameters: | NULL
> | NULL |
> | | bucketing_version
> | 2 |
> | | transient_lastDdlTime
> | 1666701647 |
> | | NULL
> | NULL |
> | # Storage Information | NULL
> | NULL |
> | SerDe Library: | null
> | NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
> | NULL |
> | Num Buckets: | -1
> | NULL |
> | Bucket Columns: | []
> | NULL |
> | Sort Columns: | []
> | NULL |
> | | NULL
> | NULL |
> | # View Information | NULL
> | NULL |
> | Original Query: | select * from test_owner_hive
> | NULL |
> | Expanded Query: | select `test_owner_hive`.`id`,
> `test_owner_hive`.`name` from `default`.`test_owner_hive` | NULL
> |
> +---++---+{noformat}
> alter table test_view_owner set owner user tarak;
> desc formatted test_view_owner;
> {noformat}
> +---++---+
> | col_name | data_type
> | comment |
>
[jira] [Commented] (HIVE-26592) TestMiniLlapLocalCliDriver test runs throw NoSuchMethodError since log4j 2.18 upgrade
[
https://issues.apache.org/jira/browse/HIVE-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631258#comment-17631258
]
Hankó Gergely commented on HIVE-26592:
--
[~Aggarwal_Raghav], I usually build with
{noformat}
mvn clean install (...) -rf :hive-it-druid{noformat}
but hive-it-druid is used from mvn cache this way and it is not rebuilt. So I
had to rebuild it without the
{code:java}
-rf :hive-it-druid {code}
part then it started to work again.
It can be skipped again afterwards because the new build is installed to the
mvn cache.
> TestMiniLlapLocalCliDriver test runs throw NoSuchMethodError since log4j 2.18
> upgrade
> -
>
> Key: HIVE-26592
> URL: https://issues.apache.org/jira/browse/HIVE-26592
> Project: Hive
> Issue Type: Bug
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
>
> The issue exists since
> [https://github.com/apache/hive/commit/c9e7f5dd6191636232921279acc1a5dd5a6fcaff]
> {code:java}
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.03
> s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR] org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver Time elapsed:
> 8.028 s <<< ERROR!
> java.lang.NoSuchMethodError:
> org.apache.logging.log4j.util.StackLocatorUtil.getCallerClassLoader(I)Ljava/lang/ClassLoader;
> at org.apache.log4j.Logger.getLogger(Logger.java:35)
> at
> org.apache.hadoop.hive.ql.udf.esri.ST_GeometryRelational.(ST_GeometryRelational.java:36)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:83)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:193)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerFunction(Registry.java:128)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerFunction(Registry.java:115)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:689)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:345)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:325)
> at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:551)
> at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:443)
> at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:430)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:386)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:291)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.(BaseSemanticAnalyzer.java:269)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.(SemanticAnalyzer.java:477)
> at org.apache.hadoop.hive.ql.QTestUtil.postInit(QTestUtil.java:565)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:88)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)[INFO]
>
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR]
> TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> » NoSuchMethod
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks
[
https://issues.apache.org/jira/browse/HIVE-25483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631199#comment-17631199
]
Ayush Saxena commented on HIVE-25483:
-
the fix type should be duplicate in that case and please link the Jira as well
> TxnHandler::acquireLock should close the DB conn to avoid connection leaks
> --
>
> Key: HIVE-25483
> URL: https://issues.apache.org/jira/browse/HIVE-25483
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: KIRTI RUGE
>Priority: Major
>
> TxnHandler::acquireLock should close DB connection on exiting the function.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740]
> If there are any exceptions downstream, this connection isn't closed
> cleanly. In a corner case, hikari connection leak detector reported the
> following
> {noformat}
> 2021-08-26 09:19:18,102 WARN com.zaxxer.hikari.pool.ProxyLeakTask:
> [HikariPool-4 housekeeper]: Connection leak detection triggered for
> org.postgresql.jdbc.PgConnection@77f76747, stack trace follows
> java.lang.Exception: Apparent connection leak detected
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:5135)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:107)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824717
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 18:04
Start Date: 09/Nov/22 18:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018259841
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
Review Comment:
how can we be sure that this is a "create MV" operation and not select from
MV?
Issue Time Tracking
---
Worklog Id: (was: 824717)
Time Spent: 3h 10m (was: 3h)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824716
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 18:01
Start Date: 09/Nov/22 18:01
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018257262
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7917,17 +7929,13 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destinationPath, currentTableId, destTableIsFullAcid,
destTableIsTemporary,//this was 1/4 acid
destTableIsMaterialization, queryTmpdir, rsCtx, dpCtx, lbCtx, fsRS,
canBeMerged, destinationTable, writeId, isMmCreate, destType, qb,
isDirectInsert, acidOperation, moveTaskId);
-if (isMmCreate) {
+if (isMmCreate || ((qb.isCTAS() || qb.isMaterializedView()) &&
isDirectInsert)) {
Review Comment:
unneeded extra brackets:
(isMmCreate || (qb.isCTAS() || qb.isMaterializedView()) && isDirectInsert)
Issue Time Tracking
---
Worklog Id: (was: 824716)
Time Spent: 3h (was: 2h 50m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824714
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:58
Start Date: 09/Nov/22 17:58
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018254056
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7990,19 +7998,29 @@ protected boolean enableColumnStatsCollecting() {
return true;
}
- private Path getCtasLocation(CreateTableDesc tblDesc, boolean
createTableWithSuffix) throws SemanticException {
+ private Path getCtasOrCMVLocation(CreateTableDesc tblDesc,
CreateMaterializedViewDesc viewDesc,
+ boolean createTableWithSuffix) throws
SemanticException {
Path location;
+String protoName;
+String[] names;
+Table tbl;
try {
- String protoName = tblDesc.getDbTableName();
- String[] names = Utilities.getDbTableName(protoName);
-
- // Handle table translation initially and if not present
- // use default table path.
- // Property modifications of the table is handled later.
- // We are interested in the location if it has changed
- // due to table translation.
- Table tbl = tblDesc.toTable(conf);
- tbl = db.getTranslateTableDryrun(tbl.getTTable());
+ if (tblDesc != null) {
+protoName = tblDesc.getDbTableName();
+names = Utilities.getDbTableName(protoName);
+
+// Handle table translation initially and if not present
+// use default table path.
+// Property modifications of the table is handled later.
+// We are interested in the location if it has changed
+// due to table translation.
+tbl = tblDesc.toTable(conf);
+tbl = db.getTranslateTableDryrun(tbl.getTTable());
Review Comment:
shouldn't we do translation for MV as well?
Issue Time Tracking
---
Worklog Id: (was: 824714)
Time Spent: 2h 40m (was: 2.5h)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824715
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:58
Start Date: 09/Nov/22 17:58
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018254056
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7990,19 +7998,29 @@ protected boolean enableColumnStatsCollecting() {
return true;
}
- private Path getCtasLocation(CreateTableDesc tblDesc, boolean
createTableWithSuffix) throws SemanticException {
+ private Path getCtasOrCMVLocation(CreateTableDesc tblDesc,
CreateMaterializedViewDesc viewDesc,
+ boolean createTableWithSuffix) throws
SemanticException {
Path location;
+String protoName;
+String[] names;
+Table tbl;
try {
- String protoName = tblDesc.getDbTableName();
- String[] names = Utilities.getDbTableName(protoName);
-
- // Handle table translation initially and if not present
- // use default table path.
- // Property modifications of the table is handled later.
- // We are interested in the location if it has changed
- // due to table translation.
- Table tbl = tblDesc.toTable(conf);
- tbl = db.getTranslateTableDryrun(tbl.getTTable());
+ if (tblDesc != null) {
+protoName = tblDesc.getDbTableName();
+names = Utilities.getDbTableName(protoName);
+
+// Handle table translation initially and if not present
+// use default table path.
+// Property modifications of the table is handled later.
+// We are interested in the location if it has changed
+// due to table translation.
+tbl = tblDesc.toTable(conf);
+tbl = db.getTranslateTableDryrun(tbl.getTTable());
Review Comment:
shouldn't we do the translation for MV as well?
Issue Time Tracking
---
Worklog Id: (was: 824715)
Time Spent: 2h 50m (was: 2h 40m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824712
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:57
Start Date: 09/Nov/22 17:57
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018254056
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7990,19 +7998,29 @@ protected boolean enableColumnStatsCollecting() {
return true;
}
- private Path getCtasLocation(CreateTableDesc tblDesc, boolean
createTableWithSuffix) throws SemanticException {
+ private Path getCtasOrCMVLocation(CreateTableDesc tblDesc,
CreateMaterializedViewDesc viewDesc,
+ boolean createTableWithSuffix) throws
SemanticException {
Path location;
+String protoName;
+String[] names;
+Table tbl;
try {
- String protoName = tblDesc.getDbTableName();
- String[] names = Utilities.getDbTableName(protoName);
-
- // Handle table translation initially and if not present
- // use default table path.
- // Property modifications of the table is handled later.
- // We are interested in the location if it has changed
- // due to table translation.
- Table tbl = tblDesc.toTable(conf);
- tbl = db.getTranslateTableDryrun(tbl.getTTable());
+ if (tblDesc != null) {
+protoName = tblDesc.getDbTableName();
+names = Utilities.getDbTableName(protoName);
+
+// Handle table translation initially and if not present
+// use default table path.
+// Property modifications of the table is handled later.
+// We are interested in the location if it has changed
+// due to table translation.
+tbl = tblDesc.toTable(conf);
+tbl = db.getTranslateTableDryrun(tbl.getTTable());
Review Comment:
shouldn't we do this translation for MV as well?
Issue Time Tracking
---
Worklog Id: (was: 824712)
Time Spent: 2.5h (was: 2h 20m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=824707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824707
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:47
Start Date: 09/Nov/22 17:47
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1018243935
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7990,19 +7998,29 @@ protected boolean enableColumnStatsCollecting() {
return true;
}
- private Path getCtasLocation(CreateTableDesc tblDesc, boolean
createTableWithSuffix) throws SemanticException {
+ private Path getCtasOrCMVLocation(CreateTableDesc tblDesc,
CreateMaterializedViewDesc viewDesc,
+ boolean createTableWithSuffix) throws
SemanticException {
Path location;
+String protoName;
+String[] names;
+Table tbl;
try {
- String protoName = tblDesc.getDbTableName();
- String[] names = Utilities.getDbTableName(protoName);
-
- // Handle table translation initially and if not present
- // use default table path.
- // Property modifications of the table is handled later.
- // We are interested in the location if it has changed
- // due to table translation.
- Table tbl = tblDesc.toTable(conf);
- tbl = db.getTranslateTableDryrun(tbl.getTTable());
+ if (tblDesc != null) {
+protoName = tblDesc.getDbTableName();
+names = Utilities.getDbTableName(protoName);
Review Comment:
can't we extract `names = Utilities.getDbTableName(protoName);`?
Issue Time Tracking
---
Worklog Id: (was: 824707)
Time Spent: 2h 20m (was: 2h 10m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=824705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824705
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:43
Start Date: 09/Nov/22 17:43
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3745:
URL: https://github.com/apache/hive/pull/3745#issuecomment-1309114346
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3745)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3745=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3745=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3745=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3745=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 824705)
Time Spent: 3h (was: 2h 50m)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-21599) Remove predicate on partition columns from Table Scan operator
[ https://issues.apache.org/jira/browse/HIVE-21599?focusedWorklogId=824701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824701 ] ASF GitHub Bot logged work on HIVE-21599: - Author: ASF GitHub Bot Created on: 09/Nov/22 17:34 Start Date: 09/Nov/22 17:34 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3742: URL: https://github.com/apache/hive/pull/3742#issuecomment-1309103415 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3742) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3742=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3742=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 824701) Time Spent: 1h 10m (was: 1h) > Remove predicate on partition columns from Table Scan operator > -- > > Key: HIVE-21599 > URL: https://issues.apache.org/jira/browse/HIVE-21599 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21599.1.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Filter predicates are pushed to Table Scan (to be pushed to and used by > storage handler/input format). Such predicates could consist of partition > columns which are of no use to storage handler or input formats. Therefore > it should be removed from TS filter expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824700
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:33
Start Date: 09/Nov/22 17:33
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
If you want to create a column name mapping function, please keep it
localized (i.e. ShowCompactionsAnalyzer) + add mapping for "SCHEMA" ->
"DATABASE".
I don't think you need to define an enum, simply append "CC_"+ provided
value.
Note: users should be able to order by any of the columns from `show
compactions` output
Issue Time Tracking
---
Worklog Id: (was: 824700)
Time Spent: 6h (was: 5h 50m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824699
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:32
Start Date: 09/Nov/22 17:32
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
If you want to create a column name mapping function, please keep it
localized (i.e. ShowCompactionsAnalyzer) + add mapping for "SCHEMA" ->
"DATABASE". I don't think you need to define an enum, simply append "CC_".
Note: users should be able to order by any of the columns from `show
compactions` output
Issue Time Tracking
---
Worklog Id: (was: 824699)
Time Spent: 5h 50m (was: 5h 40m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824697
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:30
Start Date: 09/Nov/22 17:30
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018203932
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -4013,6 +4018,11 @@ private String
getShowCompactFilterClause(ShowCompactRequest request) {
" WHERE " + StringUtils.join(" AND ", params) : EMPTY;
}
+ private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
+String sortingOrder = request.getOrder();
+return isNotBlank(sortingOrder) ? String.join("\t"," ORDER BY
",sortingOrder) : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
Review Comment:
reformat, spaces are missing
also, what is String. join is doing here? can we simply "ORDER BY
"+sortingOrder
Issue Time Tracking
---
Worklog Id: (was: 824697)
Time Spent: 5h 40m (was: 5.5h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824694
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:26
Start Date: 09/Nov/22 17:26
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018224069
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
+TBNAME("CC_TABLE"),
+TYPE("CC_TYPE"),
+STATE("CC_STATE");
+private final String colVal;
+CompactionColumn(String colVal) {
+ this.colVal = colVal;
+}
+public String getValue() {
Review Comment:
@Override toString()
Issue Time Tracking
---
Worklog Id: (was: 824694)
Time Spent: 5.5h (was: 5h 20m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824692
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:26
Start Date: 09/Nov/22 17:26
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
If you want to create a column name mapping function, please keep it
localized (i.e. ShowCompactionsAnalyzer) + add mapping for "SCHEMA" ->
"DATABASE"
"TABLE" -> "CC_TABLE"
"PARTITION" -> "CC_PARTITION"
Issue Time Tracking
---
Worklog Id: (was: 824692)
Time Spent: 5h 20m (was: 5h 10m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824690
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:20
Start Date: 09/Nov/22 17:20
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
If you want to create a column mapping function, please keep it localized
(i.e. ShowCompactionsAnalyzer) + add mapping for "SCHEMA" -> "DATABASE"
Issue Time Tracking
---
Worklog Id: (was: 824690)
Time Spent: 5h 10m (was: 5h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824681
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:07
Start Date: 09/Nov/22 17:07
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018203932
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -4013,6 +4018,11 @@ private String
getShowCompactFilterClause(ShowCompactRequest request) {
" WHERE " + StringUtils.join(" AND ", params) : EMPTY;
}
+ private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
+String sortingOrder = request.getOrder();
+return isNotBlank(sortingOrder) ? String.join("\t"," ORDER BY
",sortingOrder) : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
Review Comment:
reformat, spaces are missing
Issue Time Tracking
---
Worklog Id: (was: 824681)
Time Spent: 5h (was: 4h 50m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824678
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:05
Start Date: 09/Nov/22 17:05
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018202359
##
standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
##
@@ -1351,7 +1351,9 @@ struct ShowCompactRequest {
4: required string tablename,
5: optional string partitionname,
6: required CompactionType type,
-7: required string state
+7: required string state,
+8: optional i64 limit,
+9: optional string order
Review Comment:
it's not a map, but map serialized into string
Issue Time Tracking
---
Worklog Id: (was: 824678)
Time Spent: 4h 50m (was: 4h 40m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824673
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:00
Start Date: 09/Nov/22 17:00
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018189155
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3438,4 +3452,14 @@ public static CompactionState
compactionStateStr2Enum(String inputValue) throws
throw new SemanticException("Unexpected compaction state " + inputValue);
}
}
+
+ public static CompactionColumn compactionColStr2Enum(String inputValue)
throws SemanticException {
+try {
+ return CompactionColumn.valueOf(inputValue.toUpperCase());
+}catch (IllegalArgumentException e) {
+throw new SemanticException("Unexpected sorting column name" +
inputValue);
Review Comment:
space at the end
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3438,4 +3452,14 @@ public static CompactionState
compactionStateStr2Enum(String inputValue) throws
throw new SemanticException("Unexpected compaction state " + inputValue);
}
}
+
+ public static CompactionColumn compactionColStr2Enum(String inputValue)
throws SemanticException {
Review Comment:
not needed
Issue Time Tracking
---
Worklog Id: (was: 824673)
Time Spent: 4.5h (was: 4h 20m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=824674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824674
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:01
Start Date: 09/Nov/22 17:01
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request, #3745:
URL: https://github.com/apache/hive/pull/3745
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Issue Time Tracking
---
Worklog Id: (was: 824674)
Time Spent: 2h 50m (was: 2h 40m)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:522)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:474)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:439)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:433)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:200)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:126)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:727)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:697)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:114)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestIcebergLlapLocalCliDriver.testCliDriver(TestIcebergLlapLocalCliDriver.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
>
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824675
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:01
Start Date: 09/Nov/22 17:01
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018198739
##
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##
@@ -1984,4 +1991,27 @@ public ParseContext getParseContext() {
return pCtx;
}
+ public PTFInvocationSpec.OrderSpec processOrderSpec(ASTNode sortNode) {
Review Comment:
why not keep it in ShowCompactionsAnalyzer?
Issue Time Tracking
---
Worklog Id: (was: 824675)
Time Spent: 4h 40m (was: 4.5h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824672
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 17:00
Start Date: 09/Nov/22 17:00
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
redundant, just add `CC_ to the column.toUpperCase()
Issue Time Tracking
---
Worklog Id: (was: 824672)
Time Spent: 4h 20m (was: 4h 10m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824670
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:57
Start Date: 09/Nov/22 16:57
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018194153
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
Review Comment:
redundant, just add cc_ to the column name.
Issue Time Tracking
---
Worklog Id: (was: 824670)
Time Spent: 4h 10m (was: 4h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824669
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:56
Start Date: 09/Nov/22 16:56
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018185710
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
Review Comment:
it's single
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
Review Comment:
DB_NAME
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
+TBNAME("CC_TABLE"),
Review Comment:
TABLE_NAME
Issue Time Tracking
---
Worklog Id: (was: 824669)
Time Spent: 4h (was: 3h 50m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824666
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:53
Start Date: 09/Nov/22 16:53
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018189155
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3438,4 +3452,14 @@ public static CompactionState
compactionStateStr2Enum(String inputValue) throws
throw new SemanticException("Unexpected compaction state " + inputValue);
}
}
+
+ public static CompactionColumn compactionColStr2Enum(String inputValue)
throws SemanticException {
+try {
+ return CompactionColumn.valueOf(inputValue.toUpperCase());
+}catch (IllegalArgumentException e) {
+throw new SemanticException("Unexpected sorting column name" +
inputValue);
Review Comment:
space at the end
Issue Time Tracking
---
Worklog Id: (was: 824666)
Time Spent: 3h 50m (was: 3h 40m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824665
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:52
Start Date: 09/Nov/22 16:52
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018188264
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
+TBNAME("CC_TABLE"),
+TYPE("CC_TYPE"),
+STATE("CC_STATE");
+private final String colVal;
+CompactionColumn(String colVal) {
+ this.colVal = colVal;
+}
+public String getValue() {
Review Comment:
toString()
Issue Time Tracking
---
Worklog Id: (was: 824665)
Time Spent: 3h 40m (was: 3.5h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824664
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:51
Start Date: 09/Nov/22 16:51
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018186960
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
Review Comment:
DB_NAME
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
+DBNAME("CC_DATABASE"),
+TBNAME("CC_TABLE"),
Review Comment:
TABLE_NAME
Issue Time Tracking
---
Worklog Id: (was: 824664)
Time Spent: 3.5h (was: 3h 20m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824663
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:50
Start Date: 09/Nov/22 16:50
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018185710
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -230,6 +230,20 @@ public boolean accept(Path path) {
}
};
+ public enum CompactionColumn {
+PARTITIONS("CC_PARTITION"),
Review Comment:
it's single
Issue Time Tracking
---
Worklog Id: (was: 824663)
Time Spent: 3h 20m (was: 3h 10m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824662
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:49
Start Date: 09/Nov/22 16:49
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018184988
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
##
@@ -99,20 +99,26 @@ private ShowCompactRequest
getShowCompactioRequest(ShowCompactionsDesc desc) thr
if (isNotEmpty(desc.getPartSpec())) {
request.setPartitionname(AcidUtils.getPartitionName(desc.getPartSpec()));
}
-if(desc.getCompactionId()>0){
+if(desc.getCompactionId() > 0){
Review Comment:
nit: space
Issue Time Tracking
---
Worklog Id: (was: 824662)
Time Spent: 3h 10m (was: 3h)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[
https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=824660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824660
]
ASF GitHub Bot logged work on HIVE-26580:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 16:44
Start Date: 09/Nov/22 16:44
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3708:
URL: https://github.com/apache/hive/pull/3708#discussion_r1018178679
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
##
@@ -99,20 +99,26 @@ private ShowCompactRequest
getShowCompactioRequest(ShowCompactionsDesc desc) thr
if (isNotEmpty(desc.getPartSpec())) {
request.setPartitionname(AcidUtils.getPartitionName(desc.getPartSpec()));
}
-if(desc.getCompactionId()>0){
+if(desc.getCompactionId() > 0){
request.setId(desc.getCompactionId());
}
+if (desc.getLimit() > -1) {
+ request.setLimit(desc.getLimit());
+}
+if (isNotBlank(desc.getOrderBy())) {
+ request.setOrder(desc.getOrderBy());
+}
return request;
}
private void writeHeader(DataOutputStream os) throws IOException {
os.writeBytes("CompactionId");
os.write(Utilities.tabCode);
-os.writeBytes("Database");
+os.writeBytes("Dbname");
Review Comment:
do not change existing headers, some of the clients might have scripts
relying on them
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
##
@@ -99,20 +99,26 @@ private ShowCompactRequest
getShowCompactioRequest(ShowCompactionsDesc desc) thr
if (isNotEmpty(desc.getPartSpec())) {
request.setPartitionname(AcidUtils.getPartitionName(desc.getPartSpec()));
}
-if(desc.getCompactionId()>0){
+if(desc.getCompactionId() > 0){
request.setId(desc.getCompactionId());
}
+if (desc.getLimit() > -1) {
+ request.setLimit(desc.getLimit());
+}
+if (isNotBlank(desc.getOrderBy())) {
+ request.setOrder(desc.getOrderBy());
+}
return request;
}
private void writeHeader(DataOutputStream os) throws IOException {
os.writeBytes("CompactionId");
os.write(Utilities.tabCode);
-os.writeBytes("Database");
+os.writeBytes("Dbname");
Review Comment:
do not change the existing headers, some of the clients might have scripts
relying on them
Issue Time Tracking
---
Worklog Id: (was: 824660)
Time Spent: 3h (was: 2h 50m)
> SHOW COMPACTIONS should support ordering and limiting functionality in
> filtering options
>
>
> Key: HIVE-26580
> URL: https://issues.apache.org/jira/browse/HIVE-26580
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> SHOW COMPACTION should provide ordering by defied table . It should also
> support limitation of fetched records
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26676) Count distinct in subquery returning wrong results
[
https://issues.apache.org/jira/browse/HIVE-26676?focusedWorklogId=824644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824644
]
ASF GitHub Bot logged work on HIVE-26676:
-
Author: ASF GitHub Bot
Created on: 09/Nov/22 15:51
Start Date: 09/Nov/22 15:51
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3734:
URL: https://github.com/apache/hive/pull/3734#issuecomment-1308967856
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3734)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3734=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3734=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3734=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3734=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 824644)
Time Spent: 50m (was: 40m)
> Count distinct in subquery returning wrong results
> --
>
> Key: HIVE-26676
> URL: https://issues.apache.org/jira/browse/HIVE-26676
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The following select query should return 1 row
> {code:java}
> CREATE TABLE t_test(
> a tinyint
> );
> INSERT INTO t_test VALUES
> (0), (1), (2), (3), (4), (5), (6), (7), (8), (9);
> select 1 from (select count(distinct a) from t_test) x;
> {code}
> But it's returning 10 rows
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26696) Implement row iterator for VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita resolved HIVE-26696. --- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master. Thanks for reviewing [~lpinter]! > Implement row iterator for VectorizedRowBatch > - > > Key: HIVE-26696 > URL: https://issues.apache.org/jira/browse/HIVE-26696 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In order to utilize the existing DeleteFilter implementation of Iceberg we'll > need to provide a row iterator that iterates through the rows of a > VectorizedRowBatch instance. > This can be used during a read with positional deletes, but it will be > especially required for implementing equality deletes. > A HiveRow concept needs to be created along with a converter class that > provides row data information that's usable as an Iceberg Record. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26717) Query based Rebalance compaction on insert-only tables
[ https://issues.apache.org/jira/browse/HIVE-26717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26717: --- Labels: ACID compaction (was: ) > Query based Rebalance compaction on insert-only tables > -- > > Key: HIVE-26717 > URL: https://issues.apache.org/jira/browse/HIVE-26717 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Priority: Major > Labels: ACID, compaction > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26716) Query based Rebalance compaction on full acid tables
[ https://issues.apache.org/jira/browse/HIVE-26716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26716: --- Labels: ACID compaction (was: ) > Query based Rebalance compaction on full acid tables > > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: ACID, compaction > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26716) Query based Rebalance compaction on full acid tables
[ https://issues.apache.org/jira/browse/HIVE-26716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26716: --- Component/s: Hive > Query based Rebalance compaction on full acid tables > > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26716) Query based Rebalance compaction on full acid tables
[ https://issues.apache.org/jira/browse/HIVE-26716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26716: --- Summary: Query based Rebalance compaction on full acid tables (was: CRUD Rebalance compaction) > Query based Rebalance compaction on full acid tables > > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26696) Implement row iterator for VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-26696?focusedWorklogId=824589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824589 ] ASF GitHub Bot logged work on HIVE-26696: - Author: ASF GitHub Bot Created on: 09/Nov/22 13:51 Start Date: 09/Nov/22 13:51 Worklog Time Spent: 10m Work Description: szlta merged PR #3737: URL: https://github.com/apache/hive/pull/3737 Issue Time Tracking --- Worklog Id: (was: 824589) Time Spent: 1h 20m (was: 1h 10m) > Implement row iterator for VectorizedRowBatch > - > > Key: HIVE-26696 > URL: https://issues.apache.org/jira/browse/HIVE-26696 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In order to utilize the existing DeleteFilter implementation of Iceberg we'll > need to provide a row iterator that iterates through the rows of a > VectorizedRowBatch instance. > This can be used during a read with positional deletes, but it will be > especially required for implementing equality deletes. > A HiveRow concept needs to be created along with a converter class that > provides row data information that's usable as an Iceberg Record. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26696) Implement row iterator for VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-26696?focusedWorklogId=824548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824548 ] ASF GitHub Bot logged work on HIVE-26696: - Author: ASF GitHub Bot Created on: 09/Nov/22 12:07 Start Date: 09/Nov/22 12:07 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3737: URL: https://github.com/apache/hive/pull/3737#issuecomment-1308654352 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3737) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3737=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3737=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3737=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3737=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3737=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3737=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 824548) Time Spent: 1h 10m (was: 1h) > Implement row iterator for VectorizedRowBatch > - > > Key: HIVE-26696 > URL: https://issues.apache.org/jira/browse/HIVE-26696 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In order to utilize the existing DeleteFilter implementation of Iceberg we'll > need to provide a row iterator that iterates through the rows of a > VectorizedRowBatch instance. > This can be used during a read with positional deletes, but it will be > especially required for implementing equality deletes. > A HiveRow concept needs to be created along with a converter class that > provides row data information that's usable as an Iceberg Record. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-21599) Remove predicate on partition columns from Table Scan operator
[ https://issues.apache.org/jira/browse/HIVE-21599?focusedWorklogId=824545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824545 ] ASF GitHub Bot logged work on HIVE-21599: - Author: ASF GitHub Bot Created on: 09/Nov/22 12:03 Start Date: 09/Nov/22 12:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3742: URL: https://github.com/apache/hive/pull/3742#issuecomment-1308647596 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3742) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3742=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3742=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3742=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3742=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 824545) Time Spent: 1h (was: 50m) > Remove predicate on partition columns from Table Scan operator > -- > > Key: HIVE-21599 > URL: https://issues.apache.org/jira/browse/HIVE-21599 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21599.1.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Filter predicates are pushed to Table Scan (to be pushed to and used by > storage handler/input format). Such predicates could consist of partition > columns which are of no use to storage handler or input formats. Therefore > it should be removed from TS filter expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26666) Filter out compactions by id to minimise expense of db operations
[ https://issues.apache.org/jira/browse/HIVE-2?focusedWorklogId=824507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-824507 ] ASF GitHub Bot logged work on HIVE-2: - Author: ASF GitHub Bot Created on: 09/Nov/22 10:36 Start Date: 09/Nov/22 10:36 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3712: URL: https://github.com/apache/hive/pull/3712#issuecomment-1308548599 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3712) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3712=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3712=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 824507) Time Spent: 1.5h (was: 1h 20m) > Filter out compactions by id to minimise expense of db operations > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > At present we use below operations while filtering out compactions in classes > like > AlterTableCompactOperation > cleaner > Use show compaction filter option provided after > https://issues.apache.org/jira/browse/HIVE-13353 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-26716) CRUD Rebalance compaction
[ https://issues.apache.org/jira/browse/HIVE-26716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26716 started by László Végh. -- > CRUD Rebalance compaction > - > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26716) CRUD Rebalance compaction
[ https://issues.apache.org/jira/browse/HIVE-26716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh reassigned HIVE-26716: -- Assignee: László Végh > CRUD Rebalance compaction > - > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26715) CBO failed with JDBCJoinPushDown because of wrong HiveJdbcConverter
[
https://issues.apache.org/jira/browse/HIVE-26715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xorsum updated HIVE-26715:
--
Summary: CBO failed with JDBCJoinPushDown because of wrong
HiveJdbcConverter (was: CBO failed with JDBCJoinPushDown because wrong
HiveJdbcConverter)
> CBO failed with JDBCJoinPushDown because of wrong HiveJdbcConverter
> ---
>
> Key: HIVE-26715
> URL: https://issues.apache.org/jira/browse/HIVE-26715
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0, 3.1.3
>Reporter: xorsum
>Priority: Major
>
> h4. reproduction
>
> {code:java}
> create database tpch100m;
> use tpch100m;
> CREATE EXTERNAL TABLE region (
> r_regionkey int,
> r_name string,
> r_comment string
> ) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.url" =
> "jdbc:mysql://127.0.0.1:3306/tpch100m?serverTimezone=UTC=false",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "123456",
> "hive.sql.table" = "region",
> "hive.sql.dbcp.maxActive" = "1"
> );
> CREATE EXTERNAL TABLE nation (
> n_nationkey int,
> n_name string,
> n_regionkey int,
> n_comment string
> ) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.url" =
> "jdbc:mysql://127.0.0.1:3306/tpch100m?serverTimezone=UTC=false",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "123456",
> "hive.sql.table" = "nation",
> "hive.sql.dbcp.maxActive" = "1"
> );
> select * from nation,region where n_regionkey = r_regionkey; {code}
> h4. log
> {code:java}
> 2022-11-09T16:00:35,181 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> translator.ASTBuilder: The HiveJdbcConverter generated sql message is:
> SELECT *
> FROM (SELECT *
> FROM `nation`
> WHERE `n_regionkey` IS NOT NULL) AS `t`
> INNER JOIN (SELECT *
> FROM `region`
> WHERE `r_regionkey` IS NOT NULL) AS `t0` ON `t`.`n_regionkey` =
> `t0`.`r_regionkey`
> 2022-11-09T16:00:35,181 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> parse.CalcitePlanner: Get metadata for source tables
> 2022-11-09T16:00:35,181 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> metastore.HiveMetaStore: 0: get_table : tbl=hive.tpch100m.nation
> 2022-11-09T16:00:35,181 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> HiveMetaStore.audit: ugi=hbk ip=unknown-ip-addr cmd=get_table :
> tbl=hive.tpch100m.nation
> 2022-11-09T16:00:35,194 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> parse.CalcitePlanner: Get metadata for subqueries
> 2022-11-09T16:00:35,194 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> parse.CalcitePlanner: Get metadata for destination tables
> 2022-11-09T16:00:35,202 INFO [04ebb58a-de45-4f8c-9742-06cac555223b main]
> ql.Context: New scratch dir is
> hdfs://0.0.0.0:9000/tmp/hive/hbk/04ebb58a-de45-4f8c-9742-06cac555223b/hive_2022-11-09_16-00-34_987_5570790922374716113-1
> 2022-11-09T16:00:35,224 ERROR [04ebb58a-de45-4f8c-9742-06cac555223b main]
> jdbc.JdbcSerDe: Caught exception while initializing the SqlSerDe
> org.apache.hadoop.hive.serde2.SerDeException: Expected 7 columns. Table
> definition has 4 columns
> at org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:79)
> ~[hive-jdbc-handler-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:10973)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11329)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11288)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:517)
>
