Peter Rozsa has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24838 )
Change subject: IMPALA-15145: Propagate credential metadata to Scan Nodes / Execution Plan ...................................................................... IMPALA-15145: Propagate credential metadata to Scan Nodes / Execution Plan Wires the credentials vended by an Iceberg REST catalog (fetched and translated to Hadoop config keys by IMPALA-15144) through to the S3 connections used during planning and execution. Backend: - HdfsTableDescriptor reads TIcebergTable.credentials into a list of CredentialEntry (prefix, fs.s3a.* config, expiry). - New per-query QueryCredentials, owned by QueryState: scan nodes register their table's credentials at fragment init; lookups are a longest-prefix match across the registered tables, where a prefix only covers paths below it (on a path-component boundary, so a table's credential is never applied to a sibling location whose name merely starts with the table name), and hand out shared ownership of immutable entries. An existing entry for a prefix is only replaced by one that expires later. - HdfsFsCache::GetConnection() takes an optional QueryState and resolves the credential for the path: vended entry first, then the process-global S3ConnCredentials, then core-site.xml defaults. For a vended entry it applies the entry's Hadoop config (key material and credential provider) to the connection it builds. Such connections are keyed by the credential's prefix plus a digest of its material, so every query reading a prefix with the same credential shares one connection, a rotated credential gets a new one, and no secret is embedded in a cache key. Missing connections are built outside of the cache lock (concurrent callers for the same key wait for the build rather than starting their own). Vended connections are private instances and do not stay cached forever: HdfsFsCache tracks the queries that obtained each one (released from ~QueryState()) and disconnects it once no live query holds it and its credential has expired or it has been idle for an hour. HdfsConnOptions becomes a plain std::map so options and credentials share one representation. - QueryState is threaded into every GetConnection() call on the query path: scan init, delete-vector/Puffin blob reads, table sinks, DML finalization and the bulk HDFS operations it issues. Spill-to-S3 keeps passing its upload-tuning options per call. Frontend: - New VendedCredentialsFileIO, a HadoopFileIO wrapper set as the catalog's io-impl when vending is enabled (an explicitly configured io-impl is left alone). Iceberg's RESTCatalog instantiates it per loaded table and hands it the table's credentials through SupportsStorageCredentials. Every open resolves the credential whose prefix covers the path and uses a FileSystem instance created with that credential's Hadoop config; Hadoop's FileSystem cache is keyed by bucket and user only, so these instances are created with FileSystem.newInstance() and cached process-wide by bucket and credential identity, and closed once their credential has expired or they have been idle for an hour (which bounds instances of credentials without an expiry once a rotation superseded them). Manifest and stats reads therefore authenticate with the vended keys without any change to Iceberg's planning code, and IcebergFileMetadataLoader lists files through the same instances. Paths not covered by a credential fall through to HadoopFileIO. - Server-returned credentials are ignored when iceberg.rest-catalog.vended-credentials-enabled is false; some catalogs (Lakekeeper) vend them regardless of the access-delegation header. Tests: - tmp-file-mgr-test seeds its fake S3 connection under the same key GetConnection() now computes. - test_iceberg_rest_catalog gains a run against the in-tree HDFS-backed REST server with vended-credentials-enabled=true; that server vends nothing, so this verifies the plumbing is a no-op for the common case. Change-Id: I73592a1521f29374316ed001341f956b40f9c0d5 Assisted-by: Claude Fable 5.1 <[email protected]> --- M be/src/exec/blob-reader.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/iceberg-delete-builder.cc M be/src/exec/puffin/puffin-writer.cc M be/src/exec/table-sink-base.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/coordinator.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/hdfs-fs-cache.cc M be/src/runtime/hdfs-fs-cache.h A be/src/runtime/query-credentials.cc A be/src/runtime/query-credentials.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/hdfs-bulk-ops.cc M be/src/util/hdfs-bulk-ops.h M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/RESTCatalogProperties.java A fe/src/main/java/org/apache/impala/catalog/iceberg/VendedCredentialsFileIO.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/iceberg/TestRESTCatalogProperties.java A testdata/configs/catalog_configs/iceberg_rest_vended_config/rest.properties M tests/custom_cluster/test_iceberg_rest_catalog.py 30 files changed, 1,055 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/24838/6 -- To view, visit http://gerrit.cloudera.org:8080/24838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73592a1521f29374316ed001341f956b40f9c0d5 Gerrit-Change-Number: 24838 Gerrit-PatchSet: 6 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
