BTheunissen commented on issue #1104:
URL:
https://github.com/apache/iceberg-python/issues/1104#issuecomment-2377397379
@cshenrik Sorry about the lateness, I actually did a small internal fork of
the library and added the following logic to:
```
def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) ->
FileSystem:
if scheme in {"s3", "s3a", "s3n"}:
from pyarrow.fs import S3FileSystem
client_kwargs: Dict[str, Any] = {
"endpoint_override": self.properties.get(S3_ENDPOINT),
"access_key": get_first_property_value(self.properties,
S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties,
S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties,
S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
"region": get_first_property_value(self.properties,
S3_REGION, AWS_REGION),
}
if proxy_uri := self.properties.get(S3_PROXY_URI):
client_kwargs["proxy_options"] = proxy_uri
if connect_timeout := self.properties.get(S3_CONNECT_TIMEOUT):
client_kwargs["connect_timeout"] = float(connect_timeout)
if role_arn := self.properties.get(AWS_ROLE_ARN):
client_kwargs["role_arn"] = role_arn
if session_name := self.properties.get(AWS_SESSION_NAME):
client_kwargs["session_name"] = session_name
return S3FileSystem(**client_kwargs)
```
Passing the role_arn **and** session_name will let the S3 File System
automatically refresh the credentials of the AWS C++ client used by the PyARrow
file system, pretty tedious!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
