[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables Table property 'external.table.purge' should not be ignored when creating Iceberg tables, except that when 'iceberg.catalog' is not the Hive Catalog for managed tables, because we need to call 'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS will override 'external.table.purge' to 'TRUE'. Testing: * existing tests * add e2e tests Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Reviewed-on: http://gerrit.cloudera.org:8080/19416 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test 6 files changed, 90 insertions(+), 23 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Feb 2023 17:00:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Feb 2023 11:48:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9020/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Feb 2023 11:48:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 2: Code-Review+2 (1 comment) Thank you for the clarifications! LGTM! http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191 PS1, Line 191: TRUE > There are two scenarios(TL;DR): Thank you for taking time explaining. I see your point, I think you were talking about managed table's perspective while I was focusing on external tables. -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Feb 2023 10:43:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has posted comments on this change. (
http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be
ignored when CREATE Iceberg tables
..
Patch Set 2:
(1 comment)
Thanks for comments!
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
> Thank you for the explanation, I was not aware that the entire external.tab
There are two scenarios(TL;DR):
<-- Dividing line -->
1 execute 'create table' statement
Query1: create table ice_tbl (col_1 int) stored as iceberg;
Query2:
show create table ice_tbl
"CREATE EXTERNAL TABLE xyz.ice_tbl (
col_1 INT NULL
)
STORED AS ICEBERG
LOCATION 'hdfs://localhost:20500/test-warehouse/xyz.db/ice_tbl'
TBLPROPERTIES ('external.table.purge'='TRUE', ...)"
'external.table.purge' will be true by default, because without 'external'
keyword Hive will consider like we are trying for managed table, so Hive set it
to true by default for CREATE TABLE statement without 'external' keyword.
<-- Dividing line -->
2 execute 'create external table' statement
Query1:
create external table ice_tbl_ext
stored as iceberg
location '/data/ice_tbl_ext'
tblproperties('iceberg.catalog' = 'hadoop.tables');
Query2:
show create table ice_v2_update_all_rows
"CREATE EXTERNAL TABLE xyz.ice_tbl_ext (
col_int INT NOT NULL,
col_str STRING NOT NULL
)
STORED AS ICEBERG
LOCATION 'hdfs://localhost:20500/data/ice_tbl_ext'
TBLPROPERTIES (...)"
'external.table.purge' will be empty, but if we delete this table
'ice_tbl_ext', the data under 'hdfs://localhost:20500/data/ice_tbl_ext' will
not be deleted, because "hive.external.table.purge.default" is false
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3660.
<-- Dividing line -->
So we set TBL_PROP_EXTERNAL_TABLE_PURGE_DEFAULT -> TRUE is consistent with
Hive, because it only applies to CREATE TABLE statement without 'external'
keyword.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 01 Feb 2023 11:38:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191 PS1, Line 191: TRUE > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#Languag Thank you for the explanation, I was not aware that the entire external.table.purge property was introduced to handle migrated tables. Hive seems to set it to false by default, should we follow this direction? It sounds more natural to me to keep it false by default and set it to true for edge cases like migrated table or Iceberg HiveCatalog tables. Or we could move this property to IcebergTable. https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3660 -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 01 Feb 2023 09:43:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 2: (1 comment) https://issues.apache.org/jira/browse/HIVE-26959?focusedCommentId=17678103=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17678103 It seems that we can only warn users that the property 'external.table.purge' will be ignored when creating an Iceberg table if the catalog is not the Hive Catalog, because without external keyword Hive will consider like we are trying for managed table, which will set 'external.table.purge' to TRUE. At least, if we warn the user and they won't be confused by explicitly set 'external.table.purge' to FALSE but it is really TRUE. http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191 PS1, Line 191: TRUE > What is the reason behind setting the default to true for external tables? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ExternalTables Since Hive 4.0.0, this is set to true by default, but if we don't want to drop the data when dropping the table, we should set it to false when creating the table or with the 'alter table' statement. -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 28 Jan 2023 07:03:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12251/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 28 Jan 2023 06:57:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables Table property 'external.table.purge' should not be ignored when creating Iceberg tables, except that when 'iceberg.catalog' is not the Hive Catalog for managed tables, because we need to call 'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS will override 'external.table.purge' to 'TRUE'. Testing: * existing tests * add e2e tests Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test 6 files changed, 90 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/19416/2 -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: (1 comment) Thanks for this nice change! I only had one question about the default value. http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191 PS1, Line 191: TRUE What is the reason behind setting the default to true for external tables? -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 18 Jan 2023 10:06:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768 PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not : // the Hive Catalog for managed tables. > Yeah, I just tested the following in Hive: I created a jira to track this: https://issues.apache.org/jira/browse/HIVE-26959 And Impala cannot guarantee that the user is using a fixed version of HMS, so this warn message maybe should be worth keeping here. -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 18 Jan 2023 05:58:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be
ignored when CREATE Iceberg tables
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768
PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When
'iceberg.catalog' is not
: // the Hive Catalog for managed tables.
> Sure! I should create a Jira in the Hive community, right?
Yeah, I just tested the following in Hive:
create table test_parq_hive (i int)
stored as parquet
tblproperties ('external.table.purge'='false');
Then checked the table properties via the DESCRIBE FORMATTED command:
...
external.table.purge | TRUE
...
I think it's worth to raise a Jira for this.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 17 Jan 2023 14:37:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768 PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not : // the Hive Catalog for managed tables. > Do you want to open a Jira for this? Sure! I should create a Jira in the Hive community, right? -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 17 Jan 2023 10:26:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: Code-Review+1 (1 comment) LGTM, thanks for fixing this. http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768 PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not : // the Hive Catalog for managed tables. Do you want to open a Jira for this? -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 16 Jan 2023 18:03:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 ) Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12153/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 12 Jan 2023 11:51:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipeng...@apache.org has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19416 Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables .. IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables Table property 'external.table.purge' should not be ignored when creating Iceberg tables, except that when 'iceberg.catalog' is not the Hive Catalog for managed tables, because we need to call 'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS will override 'external.table.purge' to 'TRUE'. Testing: * existing tests * add e2e tests Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test 6 files changed, 90 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/19416/1 -- To view, visit http://gerrit.cloudera.org:8080/19416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829 Gerrit-Change-Number: 19416 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
