kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273010207
##########
File path: airflow/contrib/hooks/gcs_hook.py
##########
@@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
shutil.copyfileobj(f_in, f_out)
filename = filename_gz
- try:
- if multipart:
- if multipart is True:
- chunksize = 256 * 1024 * 1024
- else:
- chunksize = multipart
-
- if chunksize % (256 * 1024) > 0 or chunksize < 0:
- raise ValueError("Multipart size is not a multiple of
262144 (256KiB)")
-
- media = MediaFileUpload(filename, mimetype=mime_type,
- chunksize=chunksize, resumable=True)
-
- request = service.objects().insert(bucket=bucket, name=object,
media_body=media)
- response = None
- while response is None:
- status, response =
request.next_chunk(num_retries=num_retries)
- if status:
- self.log.info("Upload progress %.1f%%",
status.progress() * 100)
-
- else:
- media = MediaFileUpload(filename, mime_type)
-
- service \
- .objects() \
- .insert(bucket=bucket, name=object, media_body=media) \
- .execute(num_retries=num_retries)
-
- except HttpError as ex:
- if ex.resp['status'] == '404':
- return False
- raise
-
- finally:
- if gzip:
- os.remove(filename)
+ client = self.get_conn()
+ bucket = client.get_bucket(bucket_name=bucket)
+ blob = bucket.blob(blob_name=object)
+ blob.upload_from_filename(filename=filename,
+ content_type=mime_type)
- return True
+ if gzip:
+ os.remove(filename)
+ self.log.info('File %s uploaded to %s in %s bucket', filename, object,
bucket)
# pylint:disable=redefined-builtin
def exists(self, bucket, object):
Review comment:
Ya, I have that PR ready. I am trying to keep the changes in this PR to be
more on a backwards-compatible side.
The next PR will contain some breaking changes which will contain these name
changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
