thomas-pfeiffer opened a new issue, #3065: URL: https://github.com/apache/iceberg-python/issues/3065
### Feature Request / Improvement When working with multiple AWS configs / credentials in parallel, AWS profiles are a convenient way to achieve this. Ideally, `PyArrow` backend in `PyIceberg` should therefore also support AWS profiles, which currently only the `Fsspec` backend does support. **Current state (as of writing - pyIceberg v0.11.0):** - For `fsspec` backends, AWS profile support is implemented (see #2841) - For `PyArrow` backend, the AWS profile support is not yet available, but the `PyArrow` team do have an enhancement ticket for it (see https://github.com/apache/arrow/issues/47880). Once AWS profile is supported in `PyArrow` it can be implemented in `PyIceberg` as well, I assume. **(Unverified) Workaround for this feature gap:** ```py session = Session(profile_name="your_aws_profile") credentials = session.get_credentials() if credentials is None: raise ValueError("Could not retrieve credentials for profile") catalog = GlueCatalog( name="your_glue_catalog", **{ "client.access-key-id": credentials.access_key, "client.secret-access-key": credentials.secret_key, "client.session-token": credentials.token, ... }, ) ``` or use `Fsspec` backend instead of `PyArrow` backend for now. **To-Be / Expected Behavior:** Once `PyArrow` supports AWS profile names (see https://github.com/apache/arrow/issues/47880), AWS profile support should be implemented for `PyArrow` backend as well and `client.profile-name` and `s3.profile-name` should be fully supported. Remark: This is a follow up issue for #2841 . Issues possibly related to this issue: #570, #1207, #2657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
