Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18995 )
Change subject: IMPALA-11583: Use Iceberg API to update stats ...................................................................... IMPALA-11583: Use Iceberg API to update stats Before this patch we used HMS API alter_table() to update an Iceberg table's statistics. 'alter_table()' API calls are unsafe for Iceberg tables as they overwrite the whole HMS table, including the table property 'metadata_location' which must always point to the latest snapshot. Hence concurrent modification to the same table could be reverted by COMPUTE STATS. In this patch we are using Iceberg API to update Iceberg tables. Also, table-level stats (e.g. numRows, totalSize, totalFiles) are not set as Iceberg keeps them up-to-date. COMPUTE INCREMENTAL STATS without partition clause is the same as plain COMPUTE STATS for Iceberg tables. This behavior is aligned with current behavior on non-partitioned tables: https://impala.apache.org/docs/build/html/topics/impala_compute_stats.html COMPUTE INCREMENTAL STATS .. PARTITION raises an error. DROP STATS has been also modified to not drop table-level stats for HMS-integrated Iceberg tables. Testing: * added e2e tests for COMPUTE STATS * added e2e tests for DROP STATS * manually tested concurrent Hive INSERT and Impala COMPUTE STATS using latest Hive * opened IMPALA-11590 to add automated interop tests with Hive Change-Id: I46b6e0a5a65e18e5aaf2a007ec0242b28e0fed92 Reviewed-on: http://gerrit.cloudera.org:8080/18995 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-compute-stats.test M tests/query_test/test_iceberg.py 4 files changed, 448 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I46b6e0a5a65e18e5aaf2a007ec0242b28e0fed92 Gerrit-Change-Number: 18995 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Gergely Fürnstáhl <gfurnst...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
