Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20907 )
Change subject: IMPALA-11710: Table properties are not updated in Iceberg metadata files ...................................................................... IMPALA-11710: Table properties are not updated in Iceberg metadata files Impala has some troubles with Iceberg tables that don't specify 'external.table.purge'='true'. Schema changes like ADD COLUMN is not working, also table properties that have been set by an ALTER TABLE SET TBLPROPERTIES statement can be reset by subsequent INSERT statements. There was a bug in CatalogOpExecutor.alterIcebergTables(). Its return value determines whether we need to also update the table definition in HMS, or it was already done by the Iceberg library. In fact, the HMS table definition is always updated by the Iceberg library if the table is handled by the HiveCatalog. In every other case we need to update the HMS table definition ourselves (unless the change won't affect HMS). The issue was that CatalogOpExecutor.alterIcebergTables() returned true (which means we need to update HMS) in case of Iceberg tables that didn't specify 'external.table.purge'='true'. This was a problem, because Iceberg already modified the HMS table and set 'metadata_location' to a new metadata file. But then Impala came and modified some properties of the HMS table definition, but also reset 'metadata_location' to the original value. Therefore subsequent operations/refreshes used the earlier state of the Iceberg table and they reset the modifications. Testing: * added e2e tests for Iceberg tables in different catalogs with 'external.table.merge'='false' Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20 Reviewed-on: http://gerrit.cloudera.org:8080/20907 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-external.test M tests/query_test/test_iceberg.py 3 files changed, 78 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20907 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20 Gerrit-Change-Number: 20907 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
