Peter Rozsa has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24839 )
Change subject: IMPALA-15146: Extend Impala Minicluster with an S3-compatible object store for testing vended credentials ...................................................................... IMPALA-15146: Extend Impala Minicluster with an S3-compatible object store for testing vended credentials Adds a Docker Compose stack under testdata/bin/minicluster_lakekeeper_s3 that gives the minicluster an S3-compatible object store and a REST catalog that vends per-table credentials, so credential vending can be tested end to end without a cloud account: - RustFS as the S3 endpoint. It is Apache-2.0 licensed and MinIO-compatible, and its STS AssumeRole endpoint is what lets Lakekeeper vend scoped, expiring session credentials. RustFS enforces the session policy, so a credential vended for one table's prefix is rejected for another table in the same bucket. - Lakekeeper as the Iceberg REST catalog (s3-compat storage flavor), backed by Postgres and authenticating against the Keycloak realm shared with the existing minicluster_lakekeeper stack. - A one-shot bootstrap container (bootstrap.py) that creates the test bucket via pyarrow's S3 client (no vendor CLI needed), bootstraps Lakekeeper and creates the warehouse through its management API, and creates and seeds the ice_s3.nation and ice_s3.many_files tables via pyiceberg. run-lakekeeper-s3.sh / kill-lakekeeper-s3.sh start and stop the stack; the start script waits for the bootstrap container to finish so Impala never connects before the warehouse exists. Minicluster config: core-site.xml.py honours S3_ENDPOINT (and S3_CONNECTION_SSL_ENABLED) to point fs.s3a at RustFS with path-style access; it is ignored when S3 is the target filesystem. Two catalog configs are added: iceberg_s3_vended_config with vending enabled and iceberg_s3_novend_config as the negative variant. tests/custom_cluster/test_iceberg_credential_vending.py brings the stack up around the test class (skipped when Docker is unavailable, when S3 is the target filesystem, or when the minicluster config does not point fs.s3a at RustFS) and verifies that - a scan of the S3-backed table succeeds with vended credentials, - vended credentials coexist with, and win over, the process-global --s3a_*_key_cmd credentials for the table's prefix, - the scan fails to authenticate when vending is disabled and no other S3 credentials are configured (Lakekeeper vends regardless of the access-delegation header, so this also covers Impala ignoring them), - two tables in one bucket with different prefix-scoped credentials can be planned and scanned in the same query. Change-Id: I913b43300f8e0c7052b0b1fea609dc0ad50bce9b Assisted-by: Claude Fable 5.1 <[email protected]> --- A testdata/bin/kill-lakekeeper-s3.sh A testdata/bin/minicluster_lakekeeper_s3/Dockerfile A testdata/bin/minicluster_lakekeeper_s3/bootstrap.py A testdata/bin/minicluster_lakekeeper_s3/docker-compose.yaml A testdata/bin/run-lakekeeper-s3.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py A testdata/configs/catalog_configs/iceberg_s3_novend_config/s3-novend.properties A testdata/configs/catalog_configs/iceberg_s3_vended_config/s3-vended.properties A tests/custom_cluster/test_iceberg_credential_vending.py 9 files changed, 760 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/24839/6 -- To view, visit http://gerrit.cloudera.org:8080/24839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I913b43300f8e0c7052b0b1fea609dc0ad50bce9b Gerrit-Change-Number: 24839 Gerrit-PatchSet: 6 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
