[hive] branch master updated: HIVE-26172: Upgrade ant to 1.10.12 (Hemanth Boyina, reviewed by Stamatis Zampetakis)
This is an automated email from the ASF dual-hosted git repository. zabetak pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 1fc7a0c410 HIVE-26172: Upgrade ant to 1.10.12 (Hemanth Boyina, reviewed by Stamatis Zampetakis) 1fc7a0c410 is described below commit 1fc7a0c410d9b8a93ebaa8a318f7664490dd7b1b Author: hemanthboyina AuthorDate: Mon Apr 25 22:25:29 2022 +0530 HIVE-26172: Upgrade ant to 1.10.12 (Hemanth Boyina, reviewed by Stamatis Zampetakis) Closes #3242 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 048d0d72c0..dd6ab89a46 100644 --- a/pom.xml +++ b/pom.xml @@ -96,7 +96,7 @@ 3.0.0-M4 1.10.1 -1.10.9 +1.10.12 3.5.2 1.5.7
[hive] branch master updated: HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table (#3255) (Zoltan Haindrich reviewed by Saihemanth Gantasala)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 778ce817fb HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot
query partition data after rename table (#3255) (Zoltan Haindrich reviewed by
Saihemanth Gantasala)
778ce817fb is described below
commit 778ce817fb50ca6dc8896ebaa258e434964d7639
Author: Zoltan Haindrich
AuthorDate: Wed May 11 17:10:51 2022 +0200
HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot query partition
data after rename table (#3255) (Zoltan Haindrich reviewed by Saihemanth
Gantasala)
---
.../clientpositive/translated_external_rename4.q | 21
.../llap/translated_external_rename4.q.out | 126 +
.../hive/metastore/utils/MetaStoreUtils.java | 22 +++-
.../hadoop/hive/metastore/HiveAlterHandler.java| 39 ---
.../metastore/MetastoreDefaultTransformer.java | 3 -
5 files changed, 189 insertions(+), 22 deletions(-)
diff --git a/ql/src/test/queries/clientpositive/translated_external_rename4.q
b/ql/src/test/queries/clientpositive/translated_external_rename4.q
new file mode 100644
index 00..30768d1298
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/translated_external_rename4.q
@@ -0,0 +1,21 @@
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set metastore.metadata.transformer.location.mode=prohibit;
+
+set hive.fetch.task.conversion=none;
+set hive.compute.query.using.stats=false;
+
+set hive.create.as.external.legacy=true;
+
+CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string);
+
+insert into part_test values ("11","th","20220101");
+insert into part_test values ("22","th","20220102");
+
+alter table part_test rename to part_test11;
+
+
+desc formatted part_test11;
+desc formatted part_test11 partition(dat="20220101");
diff --git
a/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
b/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
new file mode 100644
index 00..67c73b49a9
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
@@ -0,0 +1,126 @@
+PREHOOK: query: CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_test
+POSTHOOK: query: CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_test
+PREHOOK: query: insert into part_test values ("11","th","20220101")
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_test
+POSTHOOK: query: insert into part_test values ("11","th","20220101")
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test@dat=20220101
+POSTHOOK: Lineage: part_test PARTITION(dat=20220101).c1 SCRIPT []
+POSTHOOK: Lineage: part_test PARTITION(dat=20220101).c2 SCRIPT []
+PREHOOK: query: insert into part_test values ("22","th","20220102")
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_test
+POSTHOOK: query: insert into part_test values ("22","th","20220102")
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test@dat=20220102
+POSTHOOK: Lineage: part_test PARTITION(dat=20220102).c1 SCRIPT []
+POSTHOOK: Lineage: part_test PARTITION(dat=20220102).c2 SCRIPT []
+PREHOOK: query: alter table part_test rename to part_test11
+PREHOOK: type: ALTERTABLE_RENAME
+PREHOOK: Input: default@part_test
+PREHOOK: Output: default@part_test
+POSTHOOK: query: alter table part_test rename to part_test11
+POSTHOOK: type: ALTERTABLE_RENAME
+POSTHOOK: Input: default@part_test
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test11
+PREHOOK: query: desc formatted part_test11
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@part_test11
+POSTHOOK: query: desc formatted part_test11
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@part_test11
+# col_name data_type comment
+c1 string
+c2 string
+
+# Partition Information
+# col_name data_type comment
+datstring
+
+# Detailed Table Information
+Database: default
+ A masked pattern was here
+Retention: 0
[hive] branch master updated: HIVE-26203: Implement alter iceberg table metadata location (#3270) (Laszlo Pinter, reviewed by Peter Vary)
This is an automated email from the ASF dual-hosted git repository.
lpinter pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new faa3f62923 HIVE-26203: Implement alter iceberg table metadata location
(#3270) (Laszlo Pinter, reviewed by Peter Vary)
faa3f62923 is described below
commit faa3f62923ac0b5a68a2b1e16736f1a32d38bb8e
Author: László Pintér <4102+lcspin...@users.noreply.github.com>
AuthorDate: Wed May 11 13:57:08 2022 +0200
HIVE-26203: Implement alter iceberg table metadata location (#3270) (Laszlo
Pinter, reviewed by Peter Vary)
---
.../iceberg/mr/hive/HiveIcebergMetaHook.java | 25
.../hive/TestHiveIcebergStorageHandlerNoScan.java | 70 ++
2 files changed, 95 insertions(+)
diff --git
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
index a27078581e..13142ce144 100644
---
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
+++
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
@@ -336,6 +336,31 @@ public class HiveIcebergMetaHook implements HiveMetaHook {
// that users can change data types or reorder columns too with this
alter op type, so its name is misleading..)
assertNotMigratedTable(hmsTable.getParameters(), "CHANGE COLUMN");
handleChangeColumn(hmsTable);
+} else if (AlterTableType.ADDPROPS.equals(currentAlterTableOp)) {
+ assertNotCrossTableMetadataLocationChange(hmsTable.getParameters());
+}
+ }
+
+ /**
+ * Perform a check on the current iceberg table whether a metadata change
can be performed. A table is eligible if
+ * the current metadata uuid and the new metadata uuid matches.
+ * @param tblParams hms table properties, must be non-null
+ */
+ private void assertNotCrossTableMetadataLocationChange(Map
tblParams) {
+if
(tblParams.containsKey(BaseMetastoreTableOperations.METADATA_LOCATION_PROP)) {
+ Preconditions.checkArgument(icebergTable != null,
+ "Cannot perform table migration to Iceberg and setting the snapshot
location in one step. " +
+ "Please migrate the table first");
+ String newMetadataLocation =
tblParams.get(BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ FileIO io = ((BaseTable) icebergTable).operations().io();
+ TableMetadata newMetadata = TableMetadataParser.read(io,
newMetadataLocation);
+ TableMetadata currentMetadata = ((BaseTable)
icebergTable).operations().current();
+ if (!currentMetadata.uuid().equals(newMetadata.uuid())) {
+throw new UnsupportedOperationException(
+String.format("Cannot change iceberg table %s metadata location
pointing to another table's metadata %s",
+icebergTable.name(), newMetadataLocation)
+);
+ }
}
}
diff --git
a/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
b/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
index 31b2800d81..0984cc9d10 100644
---
a/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
+++
b/iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
@@ -1456,6 +1456,76 @@ public class TestHiveIcebergStorageHandlerNoScan {
HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS.stream()).collect(Collectors.toList()),
records, 0);
}
+ @Test
+ public void testAlterTableWithMetadataLocation() throws IOException {
+Assume.assumeTrue("Alter table with metadata location is only supported
for Hive Catalog tables",
+testTableType.equals(TestTables.TestTableType.HIVE_CATALOG));
+TableIdentifier tableIdentifier = TableIdentifier.of("default", "source");
+// create a test table with some dummy data
+Table table =
+testTables.createTable(shell, tableIdentifier.name(),
HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+PartitionSpec.unpartitioned(), FileFormat.PARQUET,
Collections.emptyList(), 1, Collections.emptyMap());
+testTables.appendIcebergTable(shell.getHiveConf(), table,
FileFormat.PARQUET, null,
+HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS);
+String firstMetadataLocation = ((BaseTable)
table).operations().current().metadataFileLocation();
+testTables.appendIcebergTable(shell.getHiveConf(), table,
FileFormat.PARQUET, null,
+HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS);
+table.refresh();
+String secondMetadataLocation = ((BaseTable)
table).operations().current().metadataFileLocation();
+Assert.assertNotEquals(firstMetadataLocation, se
[hive] branch master updated: HIVE-26215: Expose the MIN_HISTORY_LEVEL table through Hive sys database (#3275). (Simhadri G, reviewed by Ayush Saxena)
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 7e4220f015 HIVE-26215: Expose the MIN_HISTORY_LEVEL table through Hive sys database (#3275). (Simhadri G, reviewed by Ayush Saxena) 7e4220f015 is described below commit 7e4220f015792451121b9ed39ef52b17e12935af Author: Simhadri Govindappa AuthorDate: Wed May 11 17:13:16 2022 +0530 HIVE-26215: Expose the MIN_HISTORY_LEVEL table through Hive sys database (#3275). (Simhadri G, reviewed by Ayush Saxena) --- .../upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql| 14 ++ .../hive/upgrade-4.0.0-alpha-1-to-4.0.0-alpha-2.hive.sql | 13 + ql/src/test/results/clientpositive/llap/resourceplan.q.out | 4 .../clientpositive/llap/strict_managed_tables_sysdb.q.out | 6 ++ ql/src/test/results/clientpositive/llap/sysdb.q.out| 9 - 5 files changed, 45 insertions(+), 1 deletion(-) diff --git a/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql b/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql index e8728f18d4..61cafd8ca9 100644 --- a/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql +++ b/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql @@ -1466,6 +1466,20 @@ SELECT DISTINCT HL.`HL_BLOCKEDBY_INT_ID` FROM SYS.`HIVE_LOCKS` AS HL; +CREATE EXTERNAL TABLE IF NOT EXISTS `MIN_HISTORY_LEVEL` ( +`MHL_TXNID` bigint, +`MHL_MIN_OPEN_TXNID` bigint +) +STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' +TBLPROPERTIES ( +"hive.sql.database.type" = "METASTORE", +"hive.sql.query" = +"SELECT +\"MHL_TXNID\", +\"MHL_MIN_OPEN_TXNID\", +FROM \"MIN_HISTORY_LEVEL\"" +); + CREATE EXTERNAL TABLE IF NOT EXISTS `REPLICATION_METRICS_ORIG` ( `SCHEDULED_EXECUTION_ID` bigint, `POLICY_NAME` string, diff --git a/metastore/scripts/upgrade/hive/upgrade-4.0.0-alpha-1-to-4.0.0-alpha-2.hive.sql b/metastore/scripts/upgrade/hive/upgrade-4.0.0-alpha-1-to-4.0.0-alpha-2.hive.sql index 32f39c9e07..a81d6881ed 100644 --- a/metastore/scripts/upgrade/hive/upgrade-4.0.0-alpha-1-to-4.0.0-alpha-2.hive.sql +++ b/metastore/scripts/upgrade/hive/upgrade-4.0.0-alpha-1-to-4.0.0-alpha-2.hive.sql @@ -2,5 +2,18 @@ SELECT 'Upgrading MetaStore schema from 4.0.0-alpha-1 to 4.0.0-alpha-2'; USE SYS; +CREATE EXTERNAL TABLE IF NOT EXISTS `MIN_HISTORY_LEVEL` ( +`MHL_TXNID` bigint, +`MHL_MIN_OPEN_TXNID` bigint +) +STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' +TBLPROPERTIES ( +"hive.sql.database.type" = "METASTORE", +"hive.sql.query" = +"SELECT +\"MHL_TXNID\", +\"MHL_MIN_OPEN_TXNID\", +FROM \"MIN_HISTORY_LEVEL\"" +); SELECT 'Finished upgrading MetaStore schema from 4.0.0-alpha-1 to 4.0.0-alpha-2'; diff --git a/ql/src/test/results/clientpositive/llap/resourceplan.q.out b/ql/src/test/results/clientpositive/llap/resourceplan.q.out index 8fcf133a3a..0a07df4baf 100644 --- a/ql/src/test/results/clientpositive/llap/resourceplan.q.out +++ b/ql/src/test/results/clientpositive/llap/resourceplan.q.out @@ -162,6 +162,10 @@ syslocks hive_test_user USER DELETE true-1 hive_test_user syslocks hive_test_user USERINSERT true-1 hive_test_user syslocks hive_test_user USERSELECT true-1 hive_test_user syslocks hive_test_user USERUPDATE true-1 hive_test_user +sysmin_history_level hive_test_user USERDELETE true-1 hive_test_user +sysmin_history_level hive_test_user USERINSERT true-1 hive_test_user +sysmin_history_level hive_test_user USERSELECT true-1 hive_test_user +sysmin_history_level hive_test_user USERUPDATE true-1 hive_test_user sysmv_creation_metadatahive_test_user USERDELETE true-1 hive_test_user sysmv_creation_metadatahive_test_user USERINSERT true-1 hive_test_user sysmv_creation_metadatahive_test_user USERSELECT true-1 hive_test_user diff --git a/ql/src/test/results/clientpositive/llap/strict_managed_tables_sysdb.q.out b/ql/src/test/results/clientpositive/llap/strict_managed_tables_sysdb.q.out index ab08793855..6555050676 100644 --- a/ql/src/test/results/clientpositive/llap/strict_managed_tables_sysdb.q.out +++ b/ql/src/test/results/clientpositive/llap/strict_managed_tables_sysdb.q.out @@ -252,6 +252,10 @@ syslocks hive_test_user USER DELETE true-1 hive_test_user syslocks hive_test_user USERINSERT true-1 hive_test_us
