Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18226
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests ...................................................................... IMPALA-10737: Optimize the number of Iceberg API Metadata requests Iceberg stores the table metadata next to the data files, when this is accessed through the Iceberg API a filesystem call is executed (HDFS, S3, ADLS). This patch moves the individual Iceberg API load calls to the CatalogD and replaces them with metatada requests from the CatalogD. Impala daemons no longer load files from the filesystem and the Catalog daemon caches the Iceberg metadata similar to other tables. As a result REFRESH/INVALIDATE METADATA is required to reload the Iceberg metadata. This improvement is more significant for smaller queries, where the metadata request has larger impact on the query time. Testing: - Passed Iceberg E2E tests. Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test 16 files changed, 215 insertions(+), 151 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18226/1 -- To view, visit http://gerrit.cloudera.org:8080/18226 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62 Gerrit-Change-Number: 18226 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
