[hive] branch master updated (076f0ac5d51 -> ec51d3b2d07)
This is an automated email from the ASF dual-hosted git repository.
dkuzmenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
from 076f0ac5d51 HIVE-27597: Implement data connector for Hive to Hive
federation over… (#4720)
add ec51d3b2d07 HIVE-27031: Addendum: Iceberg: Implement Copy-On-Write for
Delete queries (Denys Kuzmenko, reviewed by Krisztian Kasa, Butao Zhang)
No new revisions were added by this update.
Summary of changes:
data/conf/iceberg/llap/tez-site.xml| 16 +-
data/conf/iceberg/tez/tez-site.xml |8 +
.../org/apache/iceberg/mr/hive/FilesForCommit.java | 19 +-
.../mr/hive/HiveIcebergOutputCommitter.java| 30 +-
.../iceberg/mr/hive/HiveIcebergStorageHandler.java | 19 +-
...ava => HiveIcebergCopyOnWriteRecordWriter.java} | 33 +-
.../iceberg/mr/hive/writer/WriterBuilder.java | 11 +-
... => delete_iceberg_copy_on_write_partitioned.q} | 12 +-
...> delete_iceberg_copy_on_write_unpartitioned.q} | 12 +-
.../test/queries/positive/iceberg_copy_on_write.q |8 -
.../delete_iceberg_copy_on_write_partitioned.q.out | 1808
...elete_iceberg_copy_on_write_unpartitioned.q.out | 1808
.../positive/dynamic_semijoin_reduction.q.out |2 +-
.../results/positive/iceberg_copy_on_write.q.out | 101 +-
.../hive/ql/metadata/HiveStorageHandler.java |4 +
.../hive/ql/optimizer/lineage/ExprProcFactory.java |4 +-
.../hive/ql/parse/RewriteSemanticAnalyzer.java | 21 +-
.../ql/parse/UpdateDeleteSemanticAnalyzer.java | 100 +-
18 files changed, 3899 insertions(+), 117 deletions(-)
copy
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/{HiveIcebergRecordWriter.java
=> HiveIcebergCopyOnWriteRecordWriter.java} (59%)
copy
iceberg/iceberg-handler/src/test/queries/positive/{delete_iceberg_partitioned_parquet.q
=> delete_iceberg_copy_on_write_partitioned.q} (76%)
copy
iceberg/iceberg-handler/src/test/queries/positive/{delete_iceberg_unpartitioned_parquet.q
=> delete_iceberg_copy_on_write_unpartitioned.q} (74%)
create mode 100644
iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_partitioned.q.out
create mode 100644
iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_unpartitioned.q.out
[hive] branch branch-3 updated: HIVE-27765: Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966 to branch-3(#4772)
This is an automated email from the ASF dual-hosted git repository.
sankarh pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new f6aa916a171 HIVE-27765: Backport of HIVE-20052, HIVE-20093,
HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966 to
branch-3(#4772)
f6aa916a171 is described below
commit f6aa916a17109d4f64e7dd878b49912b79dd8d75
Author: Aman Raj <104416558+amanraj2...@users.noreply.github.com>
AuthorDate: Tue Oct 10 10:41:32 2023 +0530
HIVE-27765: Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290,
HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966 to branch-3(#4772)
* HIVE-20052: Arrow serde should fill ArrowColumnVector(Decimal) with the
given schema precision/scale
* HIVE-20093: LlapOutputFomatService: Use ArrowBuf with Netty for Accounting
* HIVE-20203: Arrow SerDe leaks a DirectByteBuffer
* HIVE-20290: Lazy initialize ArrowColumnarBatchSerDe so it doesn't
allocate buffers during GetSplits
* HIVE-20300: VectorFileSinkArrowOperator
* HIVE-20312: Allow arrow clients to use their own BufferAllocator with
LlapOutputFormatService
* HIVE-20044: Arrow Serde should pad char values and handle empty strings
correctly
* HIVE-21966: Llap external client - Arrow Serializer throws
ArrayIndexOutOfBoundsException in some cases
-
Co-authored-by: Eric Wohlstadter
Co-authored-by: Nikhil Gupta
Co-authored-by: Teddy Choi
Co-authored-by: Shubham Chaurasia
Signed-off-by: Sankar Hariappan
Closes (#4772)
---
.../java/org/apache/hadoop/hive/conf/HiveConf.java | 8 +-
.../org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java | 45 +-
.../hive/jdbc/TestJdbcWithMiniLlapArrow.java | 7 +-
.../apache/hive/jdbc/TestJdbcWithMiniLlapRow.java | 6 +-
...w.java => TestJdbcWithMiniLlapVectorArrow.java} | 297 ---
.../hive/llap/LlapArrowBatchRecordReader.java | 15 +-
.../hadoop/hive/llap/LlapArrowRowInputFormat.java | 14 +-
.../hadoop/hive/llap/LlapBaseInputFormat.java | 25 +-
.../hadoop/hive/llap/LlapArrowRecordWriter.java| 25 +-
.../hive/llap/WritableByteChannelAdapter.java | 12 +-
.../filesink/VectorFileSinkArrowOperator.java | 180 +
.../hive/ql/io/arrow/ArrowColumnarBatchSerDe.java | 20 +-
.../hive/ql/io/arrow/ArrowWrapperWritable.java | 19 +
.../apache/hadoop/hive/ql/io/arrow/Serializer.java | 865 +++--
.../hive/ql/optimizer/physical/Vectorizer.java | 60 +-
.../ql/io/arrow/TestArrowColumnarBatchSerDe.java | 53 ++
.../ql/exec/vector/expressions/StringExpr.java | 15 +
17 files changed, 1268 insertions(+), 398 deletions(-)
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index 3ec99315a27..bf20a78b588 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -2682,6 +2682,8 @@ public class HiveConf extends Configuration {
// For Arrow SerDe
HIVE_ARROW_ROOT_ALLOCATOR_LIMIT("hive.arrow.root.allocator.limit",
Long.MAX_VALUE,
"Arrow root allocator memory size limitation in bytes."),
+HIVE_ARROW_BATCH_ALLOCATOR_LIMIT("hive.arrow.batch.allocator.limit",
10_000_000_000L,
+"Max bytes per arrow batch. This is a threshold, the memory is not
pre-allocated."),
HIVE_ARROW_BATCH_SIZE("hive.arrow.batch.size", 1000, "The number of rows
sent in one Arrow batch."),
// For Druid storage handler
@@ -3690,7 +3692,11 @@ public class HiveConf extends Configuration {
"internal use only. When false, don't suppress fatal exceptions
like\n" +
"NullPointerException, etc so the query will fail and assure it will
be noticed",
true),
-
+HIVE_VECTORIZATION_FILESINK_ARROW_NATIVE_ENABLED(
+"hive.vectorized.execution.filesink.arrow.native.enabled", true,
+"This flag should be set to true to enable the native vectorization\n"
+
+"of queries using the Arrow SerDe and FileSink.\n" +
+"The default value is true."),
HIVE_TYPE_CHECK_ON_INSERT("hive.typecheck.on.insert", true, "This property
has been extended to control "
+ "whether to check, convert, and normalize partition value to conform
to its column type in "
+ "partition operations including but not limited to insert, such as
alter, describe etc."),
diff --git
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java
b/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java
index 5cf765d8eb8..fbcd229d224 100644
---
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java
+++
b/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java
@@ -102,38 +102,20 @@ import org.apache.hadoop.mapred.Inp
[hive] branch master updated (dec006e81e6 -> 076f0ac5d51)
This is an automated email from the ASF dual-hosted git repository. ngangam pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from dec006e81e6 HIVE-27771: Iceberg: Allow expire snapshot by time range. (#4776). (Ayush Saxena, reviewed by Denys Kuzmenko) add 076f0ac5d51 HIVE-27597: Implement data connector for Hive to Hive federation over… (#4720) No new revisions were added by this update. Summary of changes: .../AbstractDataConnectorProvider.java | 3 + .../DataConnectorProviderFactory.java | 2 + .../dataconnector/IDataConnectorProvider.java | 1 + .../JDBCConnectorProviderFactory.java | 5 ++ .../jdbc/AbstractJDBCConnectorProvider.java| 58 +- .../jdbc/DerbySQLConnectorProvider.java| 17 -- .../jdbc/HiveJDBCConnectorProvider.java| 71 ++ .../dataconnector/jdbc/MSSQLConnectorProvider.java | 22 --- .../dataconnector/jdbc/MySQLConnectorProvider.java | 16 - .../jdbc/OracleConnectorProvider.java | 22 --- .../jdbc/PostgreSQLConnectorProvider.java | 15 - 11 files changed, 124 insertions(+), 108 deletions(-) create mode 100644 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java
[hive] branch master updated: HIVE-27771: Iceberg: Allow expire snapshot by time range. (#4776). (Ayush Saxena, reviewed by Denys Kuzmenko)
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new dec006e81e6 HIVE-27771: Iceberg: Allow expire snapshot by time range.
(#4776). (Ayush Saxena, reviewed by Denys Kuzmenko)
dec006e81e6 is described below
commit dec006e81e657bc6edb0d57d31076f04e1f088c5
Author: Ayush Saxena
AuthorDate: Mon Oct 9 16:48:11 2023 +0530
HIVE-27771: Iceberg: Allow expire snapshot by time range. (#4776). (Ayush
Saxena, reviewed by Denys Kuzmenko)
---
.../iceberg/mr/hive/HiveIcebergStorageHandler.java | 22 +++-
.../mr/hive/TestHiveIcebergExpireSnapshots.java| 17 +++
.../hadoop/hive/ql/parse/AlterClauseParser.g | 2 ++
.../table/execute/AlterTableExecuteAnalyzer.java | 24 +++---
.../hive/ql/parse/AlterTableExecuteSpec.java | 19 -
5 files changed, 75 insertions(+), 9 deletions(-)
diff --git
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
index bb9356df251..4e451403b02 100644
---
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
+++
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
@@ -834,7 +834,10 @@ public class HiveIcebergStorageHandler implements
HiveStoragePredicateHandler, H
LOG.info("Executing expire snapshots on iceberg table {} with {}
threads", icebergTable.name(), numThreads);
deleteExecutorService = getDeleteExecutorService(icebergTable.name(),
numThreads);
}
- if (expireSnapshotsSpec.isExpireByIds()) {
+ if (expireSnapshotsSpec.isExpireByTimestampRange()) {
+expireSnapshotByTimestampRange(icebergTable,
expireSnapshotsSpec.getFromTimestampMillis(),
+expireSnapshotsSpec.getTimestampMillis(), deleteExecutorService);
+ } else if (expireSnapshotsSpec.isExpireByIds()) {
expireSnapshotByIds(icebergTable,
expireSnapshotsSpec.getIdsToExpire(), deleteExecutorService);
} else {
expireSnapshotOlderThanTimestamp(icebergTable,
expireSnapshotsSpec.getTimestampMillis(), deleteExecutorService);
@@ -846,6 +849,23 @@ public class HiveIcebergStorageHandler implements
HiveStoragePredicateHandler, H
}
}
+ private void expireSnapshotByTimestampRange(Table icebergTable, Long
fromTimestamp, Long toTimestamp,
+ ExecutorService deleteExecutorService) {
+ExpireSnapshots expireSnapshots = icebergTable.expireSnapshots();
+for (Snapshot snapshot : icebergTable.snapshots()) {
+ if (snapshot.timestampMillis() >= fromTimestamp &&
snapshot.timestampMillis() <= toTimestamp) {
+expireSnapshots.expireSnapshotId(snapshot.snapshotId());
+LOG.debug("Expiring snapshot on {} with id: {} and timestamp: {}",
icebergTable.name(), snapshot.snapshotId(),
+snapshot.timestampMillis());
+ }
+}
+LOG.info("Expiring snapshot on {} within time range {} -> {}",
icebergTable.name(), fromTimestamp, toTimestamp);
+if (deleteExecutorService != null) {
+ expireSnapshots.executeDeleteWith(deleteExecutorService);
+}
+expireSnapshots.commit();
+ }
+
private void expireSnapshotOlderThanTimestamp(Table icebergTable, Long
timestamp,
ExecutorService deleteExecutorService) {
ExpireSnapshots expireSnapshots =
icebergTable.expireSnapshots().expireOlderThan(timestamp);
diff --git
a/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergExpireSnapshots.java
b/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergExpireSnapshots.java
index 08277fb7892..a851578ee6c 100644
---
a/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergExpireSnapshots.java
+++
b/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergExpireSnapshots.java
@@ -20,6 +20,8 @@
package org.apache.iceberg.mr.hive;
import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
import org.apache.commons.collections4.IterableUtils;
import org.apache.iceberg.Table;
import org.apache.iceberg.catalog.TableIdentifier;
@@ -65,4 +67,19 @@ public class TestHiveIcebergExpireSnapshots extends
HiveIcebergStorageHandlerWit
table.refresh();
Assert.assertEquals(7, IterableUtils.size(table.snapshots()));
}
+
+ @Test
+public void testExpireSnapshotsWithTimestampRange() throws IOException,
InterruptedException {
+TableIdentifier identifier = TableIdentifier.of("default", "source");
+Table table = testTables.createTableWithVersions(shell, identifier.name(),
+HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA, fileFormat,
+HiveIcebe
[hive] branch master updated (f5d089138ee -> 20be17d19c1)
This is an automated email from the ASF dual-hosted git repository. veghlaci05 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from f5d089138ee HIVE-23680 : TestDbNotificationListener is unstable (Kirti Ruge, reviewed by Laszlo Vegh) add 20be17d19c1 Remove Powermock - finishing touches (Zsolt Miskolczi, reviewed by Laszlo Vegh) No new revisions were added by this update. Summary of changes: beeline/pom.xml | 2 +- itests/hive-jmh/pom.xml | 2 +- jdbc-handler/pom.xml| 2 +- llap-client/pom.xml | 2 +- pom.xml | 2 +- ql/pom.xml | 2 +- service/pom.xml | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-)
[hive] branch master updated (6d75f92142c -> f5d089138ee)
This is an automated email from the ASF dual-hosted git repository. veghlaci05 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from 6d75f92142c HIVE-27399: Add support for lateral views with CBO (Stephen Carlin reviewed by Krisztian Kasa, Stamatis Zampetakis) add f5d089138ee HIVE-23680 : TestDbNotificationListener is unstable (Kirti Ruge, reviewed by Laszlo Vegh) No new revisions were added by this update. Summary of changes: hcatalog/webhcat/java-client/pom.xml | 5 + itests/hcatalog-unit/pom.xml | 1 + .../listener/TestDbNotificationCleanup.java| 189 ++ .../listener/TestDbNotificationListener.java | 218 +++-- .../TestTransactionalDbNotificationListener.java | 169 5 files changed, 391 insertions(+), 191 deletions(-) create mode 100644 itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationCleanup.java create mode 100644 itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestTransactionalDbNotificationListener.java
