Peter Rozsa has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/25039 )
Change subject: IMPALA-15493: Coordinator-local DDL execution for Iceberg REST catalogs ...................................................................... IMPALA-15493: Coordinator-local DDL execution for Iceberg REST catalogs DDL on a table loaded from an Iceberg REST catalog has always been built as a TDdlExecRequest and sent to CatalogD, which does not know the table in hybrid mode and does not exist in standalone mode. This adds the DDL counterpart of the coordinator-side DML finalization introduced in IMPALA-15310. TCatalogOpRequest carries an optional rest_catalog_name. When it is set, CatalogOpExecutor hands the DDL to the coordinator frontend through the new TIcebergDdlExecRequest JNI call instead of CatalogD, and ClientRequestState skips the catalog update wait, since nothing is cached on the coordinator for REST tables. The frontend routes a DDL this way when its analyzed target table was loaded from a uniquely named REST catalog. When there is no loaded table to route by (IF EXISTS on a missing table, or a table that failed to load), the DDL is routed by its database instead: MetaProvider gains getIcebergRestCatalogNameForDb(), which MultiMetaProvider answers from the first provider that lists the database, and FeDb exposes it. The new IcebergDdlExecutor executes the operation through the named catalog without loading the table first, so tables with unreadable metadata can still be dropped. DROP TABLE [IF EXISTS] [PURGE] is the first operation carried by the new path: IcebergRESTCatalog.dropTable() is implemented and the analyzer lets DROP TABLE through for REST tables whose catalog has a unique name. DROP VIEW IF EXISTS routed to a REST catalog is a no-op, as REST catalogs serve no views. Other DDL is still rejected at analysis. Since the catalog name now routes DDL as well, the DML-specific names are generalized: getIcebergDmlCatalogName() becomes getIcebergRestCatalogName(), and FeCatalogManager resolves a REST catalog name once, in getCatalogProvider(), from which getCatalog() (for loading tables) and getIcebergCatalog() (for DROP TABLE) derive. Testing: - New IcebergDdlExecutorTest and an AnalyzerCapabilityTest case cover routing by table and by database, execution, and the error paths without a live catalog. - New custom cluster tests drop REST tables with and without CatalogD, across two REST catalogs, and with IF EXISTS on missing tables; test_iceberg_rest_catalog.py passes. - Updated the REST catalog documentation. Change-Id: I932a211b25ea4e24a607d047c35da186d8922d02 Assisted-by: Claude Fable 5.1 (Claude Code) Co-Authored-By: Claude Fable 5.1 <[email protected]> --- M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M docs/topics/impala_iceberg_rest_catalog.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/service/IcebergDdlExecutor.java M fe/src/main/java/org/apache/impala/service/IcebergDmlFinalizer.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/catalogmanager/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/catalogmanager/LocalImpl.java M fe/src/main/java/org/apache/impala/service/catalogmanager/TestImpl.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerCapabilityTest.java A fe/src/test/java/org/apache/impala/service/IcebergDdlExecutorTest.java M fe/src/test/java/org/apache/impala/service/IcebergRestDmlOperationTest.java M fe/src/test/java/org/apache/impala/service/catalogmanager/LocalImplTest.java M tests/custom_cluster/test_iceberg_rest_catalog.py 31 files changed, 837 insertions(+), 70 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/25039/2 -- To view, visit http://gerrit.cloudera.org:8080/25039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I932a211b25ea4e24a607d047c35da186d8922d02 Gerrit-Change-Number: 25039 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
