mengw15 opened a new pull request, #4988: URL: https://github.com/apache/texera/pull/4988
### What changes were proposed in this PR? `IcebergUtil.createRestCatalog` (Scala) and `iceberg_utils.create_rest_catalog` (Python) no longer hardcode `s3.endpoint`, `s3.region`, `s3.path-style-access`, `s3.access-key-id` or `s3.secret-access-key` at catalog initialization. Both helpers now pass only `warehouse`, the catalog `uri`, and (Scala only) the FileIO implementation hint; the REST catalog server stays the source of truth for S3 settings. | Before | After | | --- | --- | | Both helpers passed system `StorageConfig.s3*` / `StorageConfig.S3_*` at init | Both helpers pass only catalog-side settings | | Hardcoded creds were inert under remote-signing (Lakekeeper returns `s3.remote-signing-enabled: true`, which makes both Iceberg `S3FileIO` and pyiceberg `FsspecFileIO` ignore them) | Server-provided settings are the only ones in play | | Same system bucket assumed for every warehouse | Per-warehouse storage isolation becomes possible | `StorageConfig.s3*` / `StorageConfig.S3_*` are kept — they continue to flow into Python via `PythonWorkflowWorker` and are consumed by `pytexera/storage/large_binary_manager.py` (boto3 client for the non-Iceberg `texera-large-binaries` bucket, used for R UDF large binary support). That bucket is out of scope for this PR. ### Any related issues, documentation, discussions? Closes #4987 ### How was this PR tested? ``` sbt "WorkflowCore/compile" [success] Total time: 10 s ``` Python edits were syntax-checked via `ast.parse` — both files parse cleanly. This is a dead-code removal: the compile step verifies no remaining caller depends on the removed Scala properties; the Python `iceberg_utils.create_rest_catalog` signature change is reflected in its only caller (`iceberg_catalog_instance.py`). Runtime behavior is unchanged because the catalog server's `s3.remote-signing-enabled: true` already caused both clients to ignore the locally-configured credentials. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
