Taragolis commented on code in PR #35037:
URL: https://github.com/apache/airflow/pull/35037#discussion_r1364676943
##########
airflow/providers/amazon/aws/hooks/s3.py:
##########
@@ -912,14 +912,27 @@ def get_key(self, key: str, bucket_name: str | None =
None) -> S3ResourceObject:
:param bucket_name: the name of the bucket
:return: the key object from the bucket
"""
+
+ def __sanitize_extra_args() -> dict[str, str]:
+ """Parse extra_args and return a dict with only the args listed in
ALLOWED_DOWNLOAD_ARGS."""
+ return {
+ arg_name: arg_value
+ for (arg_name, arg_value) in self.extra_args.items()
+ if arg_name in S3Transfer(self.conn).ALLOWED_DOWNLOAD_ARGS
+ }
+
s3_resource = self.get_session().resource(
"s3",
endpoint_url=self.conn_config.endpoint_url,
config=self.config,
verify=self.verify,
)
obj = s3_resource.Object(bucket_name, key)
- obj.load()
+
+ # TODO inline this after debugging
+ new_args = __sanitize_extra_args()
+
+ obj.load(**new_args)
Review Comment:
Yeah, I've also test some similar stuff, but some errored stuff when
parameters are invalid
```python
import boto3
session = boto3.session.Session(profile_name=...)
resource = session.resource(service_name="s3")
obj = resource.Object('taragolis-example-bucket', 'setup.cfg')
obj.load(SSECustomerKey="aaaaa", SSECustomerAlgorithm="AES256")
```
```console
Traceback (most recent call last):
File "/Users/taragolis/Library/Application
Support/JetBrains/PyCharm2023.2/scratches/some_stuff.py", line 7, in <module>
obj.load(SSECustomerKey="aaaaa", SSECustomerAlgorithm="AES256")
File
"/Users/taragolis/.pyenv/versions/airflow-dev-env-39/lib/python3.9/site-packages/boto3/resources/factory.py",
line 564, in do_action
response = action(self, *args, **kwargs)
File
"/Users/taragolis/.pyenv/versions/airflow-dev-env-39/lib/python3.9/site-packages/boto3/resources/action.py",
line 88, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File
"/Users/taragolis/.pyenv/versions/airflow-dev-env-39/lib/python3.9/site-packages/botocore/client.py",
line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File
"/Users/taragolis/.pyenv/versions/airflow-dev-env-39/lib/python3.9/site-packages/botocore/client.py",
line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the
HeadObject operation: Bad Request
```
Seems like resource part it abandoned part of `boto3`:
https://github.com/boto/boto3/tree/develop/boto3/data - last changes 6-8 years
ago. So I guess no new features available thought High Level Client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
