kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273018225
##########
File path: airflow/contrib/hooks/gcs_hook.py
##########
@@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
shutil.copyfileobj(f_in, f_out)
filename = filename_gz
- try:
- if multipart:
- if multipart is True:
- chunksize = 256 * 1024 * 1024
- else:
- chunksize = multipart
-
- if chunksize % (256 * 1024) > 0 or chunksize < 0:
- raise ValueError("Multipart size is not a multiple of
262144 (256KiB)")
-
- media = MediaFileUpload(filename, mimetype=mime_type,
- chunksize=chunksize, resumable=True)
-
- request = service.objects().insert(bucket=bucket, name=object,
media_body=media)
- response = None
- while response is None:
- status, response =
request.next_chunk(num_retries=num_retries)
- if status:
- self.log.info("Upload progress %.1f%%",
status.progress() * 100)
-
- else:
- media = MediaFileUpload(filename, mime_type)
-
- service \
- .objects() \
- .insert(bucket=bucket, name=object, media_body=media) \
- .execute(num_retries=num_retries)
-
- except HttpError as ex:
- if ex.resp['status'] == '404':
- return False
- raise
-
- finally:
- if gzip:
- os.remove(filename)
+ client = self.get_conn()
+ bucket = client.get_bucket(bucket_name=bucket)
+ blob = bucket.blob(blob_name=object)
+ blob.upload_from_filename(filename=filename,
+ content_type=mime_type)
- return True
+ if gzip:
+ os.remove(filename)
+ self.log.info('File %s uploaded to %s in %s bucket', filename, object,
bucket)
# pylint:disable=redefined-builtin
def exists(self, bucket, object):
Review comment:
The maintain intention is so that we can cherry-pick this one in 1.10.4.
If you look at this PR and check for "breaking-changes" - the one's that are
there are not widely used (also in Updating.md).
I wouldn't want to change the name of something like the `object` parameter
(or even `bucket`) and put a note in Updating.md. We should not be added the
2nd PR for 1.10.4 and would target 2.0.
They are fundamentally 2 separate pieces: This PR focuses on "Replacing
discovery api with client api" and not on "updating parameter name". Also more
readable in Changelog.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
