yangaws commented on a change in pull request #4091: [AIRFLOW-2524] Update
SageMaker hook, operator and sensor for training, tuning and transform
URL: https://github.com/apache/incubator-airflow/pull/4091#discussion_r229640764
##########
File path: airflow/hooks/S3_hook.py
##########
@@ -401,6 +418,41 @@ def load_bytes(self,
client = self.get_conn()
client.upload_fileobj(filelike_buffer, bucket_name, key,
ExtraArgs=extra_args)
+ def load_file_obj(self,
+ file_obj,
+ key,
+ bucket_name=None,
+ replace=False,
+ encrypt=False):
+ """
+ Loads file object to S3
+
+ :param file_obj: file-like object to set as content for the key.
+ :type file_obj: file-like object
+ :param key: S3 key that will point to the file
+ :type key: str
+ :param bucket_name: Name of the bucket in which to store the file
+ :type bucket_name: str
+ :param replace: A flag to decide whether or not to overwrite the key
+ if it already exists
+ :type replace: bool
+ :param encrypt: If True, the file will be encrypted on the server-side
+ by S3 and will be stored in an encrypted form while at rest in S3.
+ :type encrypt: bool
+ """
+ if not bucket_name:
+ (bucket_name, key) = self.parse_s3_url(key)
+
+ if not replace and self.check_for_key(key, bucket_name):
+ raise ValueError("The key {key} already exists.".format(key=key))
+
+ extra_args = {}
+ if encrypt:
+ extra_args['ServerSideEncryption'] = "AES256"
Review comment:
I agree with you. But other S3 hook APIs (load_file, load_bytes) put
extra_args in the function body and my codes just imitated them. I guess to
make things consistent, having a PR to change them all sounds like a better
idea. Not sure if I only change my codes and leave others unchanged is what you
want.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
