Hello Tamas Mate, Daniel Becker, Noemi Pap-Takacs, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20907 to look at the new patch set (#3). Change subject: IMPALA-11710: Table properties are not updated in Iceberg metadata files ...................................................................... IMPALA-11710: Table properties are not updated in Iceberg metadata files Impala has some troubles with Iceberg tables that don't specify 'external.table.purge'='true'. Schema changes like ADD COLUMN is not working, also table properties that have been set by an ALTER TABLE SET TBLPROPERTIES statement can be reset by subsequent INSERT statements. There was a bug in CatalogOpExecutor.alterIcebergTables(). Its return value determines whether we need to also update the table definition in HMS, or it was already done by the Iceberg library. In fact, the HMS table definition is always updated by the Iceberg library if the table is handled by the HiveCatalog. In every other case we need to update the HMS table definition ourselves (unless the change won't affect HMS). The issue was that CatalogOpExecutor.alterIcebergTables() returned true (which means we need to update HMS) in case of Iceberg tables that didn't specify 'external.table.purge'='true'. This was a problem, because Iceberg already modified the HMS table and set 'metadata_location' to a new metadata file. But then Impala came and modified some properties of the HMS table definition, but also reset 'metadata_location' to the original value. Therefore subsequent operations/refreshes used the earlier state of the Iceberg table and they reset the modifications. Testing: * added e2e tests for Iceberg tables in different catalogs with 'external.table.merge'='false' Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-external.test M tests/query_test/test_iceberg.py 3 files changed, 78 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/20907/3 -- To view, visit http://gerrit.cloudera.org:8080/20907 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20 Gerrit-Change-Number: 20907 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>