[
https://issues.apache.org/jira/browse/AIRFLOW-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931294#comment-16931294
]
ASF subversion and git services commented on AIRFLOW-5162:
----------------------------------------------------------
Commit 26aa4ba648697ec582437b1e86ca3191bdcc253c in airflow's branch
refs/heads/master from Steven Freed
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=26aa4ba ]
[AIRFLOW-5162] GCS Hook Upload Method Improvement (#5770)
* [AIRFLOW-5162] add to 'upload' method for gcs file data
Allows for local file uploads (parameter 'filename') and file content
(parameter 'data') uploads to gcs by using Google's gcs api method
'upload_from_string' which allows for uploading files content as a string or
bytes.
* [AIRFLOW-5162] add ValueError exceptions and upload tests
ValueError exceptions are raised when the filename and data params are
provided or neither are provided. This avoids errors of local files and
file data. Tests assert file and data upload variants as well as assert
exceptions.
* [AIRFLOW-5162] fixed unit tests for gcs uploads
added seperate unit tests for uploading string content and byte content
using the upload_from_string gcp method. Prevents assertion being called
twice when tests are seperated as well as organizes testing.
* [AIRFLOW-5162] added type annotations and tests modifications
Added type annotations for static tests and ease of use for users. Tests
modifications get rid of assertions for None responses and add tests for
calling upload functions once.
* [AIRFLOW-5162] add pylint and flake8 fixes for type annotations
type annotation syntax requires a space between '=' sign for default
assignment.
* [AIRFLOW-5162] added called once and exception tests
added called once tests for upload methods for gzip data. This was added
to assert that the upload_from_string method was called correctly.
Assertions for exceptions confirming the correct exception messages.
* [AIRFLOW-5162] added fix to doc build
Fixed indentation in docstrings so that the documentation can build
properly. Added new lines after descriptions of the methods
functionalities.
> GCS Uploads for local files and file data (string or bytes)
> -----------------------------------------------------------
>
> Key: AIRFLOW-5162
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5162
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
> Affects Versions: 1.10.4, 1.10.5
> Reporter: Steven Freed
> Priority: Minor
> Labels: beginner, features, pull-request-available
>
> Allows for gcs uploads to be done by specifying a local file name or by using
> the file's content. File content can be uploaded as a string or bytes. This
> gives developers more alternatives to uploading files where developers may
> have constraints on disk space and prefer to upload via file content in
> memory. This update also helps airflow update to Google's new
> 'upload_from_string' method for gcs.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
