Hello Tamas Mate, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20907
to look at the new patch set (#2).
Change subject: IMPALA-11710: Table properties are not updated in Iceberg
metadata files
......................................................................
IMPALA-11710: Table properties are not updated in Iceberg metadata files
Impala has some troubles with Iceberg tables that don't specify
'external.table.purge'='true'. Schema changes like ADD COLUMN is not
working, also table properties that have been set by an ALTER TABLE
SET TBLPROPERTIES statement can be reset by subsequent INSERT
statements.
There was a bug in CatalogOpExecutor.alterIcebergTables(). Its return
value determines whether we need to also update the table definition in
HMS, or it was already done by the Iceberg library. In fact, the HMS
table definition is always updated by the Iceberg library if the table
is handled by the HiveCatalog. In every other case we need to update
the HMS table definition ourselves.
The issue was that CatalogOpExecutor.alterIcebergTables() returned true
(which means we need to update HMS) in case of Iceberg tables that
didn't specify 'external.table.purge'='true'. This was a problem,
because Iceberg already modified the HMS table and set
'metadata_location' to a new metadata file. But then Impala came and
modified some properties of the HMS table definition, but also reset
'metadata_location' to the original value. Therefore subsequent
operations/refreshes used the earlier state of the Iceberg table and
they reset the modifications.
Testing:
* added e2e tests for Iceberg tables in different catalogs with
'external.table.merge'='false'
Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-external.test
M tests/query_test/test_iceberg.py
3 files changed, 75 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/20907/2
--
To view, visit http://gerrit.cloudera.org:8080/20907
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20
Gerrit-Change-Number: 20907
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tma...@apache.org>
