[ https://issues.apache.org/jira/browse/AIRFLOW-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457751#comment-16457751 ]
ASF subversion and git services commented on AIRFLOW-2348: ---------------------------------------------------------- Commit 143886209716b041402e78a1313c54336edb840a in incubator-airflow's branch refs/heads/v1-10-test from [~csoulios] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1438862 ] [AIRFLOW-2348] Strip path prefix from the destination_object when source_object contains a wildcard[] Closes #3247 from csoulios/master (cherry picked from commit 840930b4d0c545adc24d8eb040974f88370e081b) Signed-off-by: Fokko Driesprong <fokkodriespr...@godatadriven.com> > GoogleCloudStorageToGoogleCloudStorageOperator copies full source path > ---------------------------------------------------------------------- > > Key: AIRFLOW-2348 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2348 > Project: Apache Airflow > Issue Type: Bug > Components: gcp > Affects Versions: 2.0.0 > Reporter: Christos Soulios > Priority: Minor > Fix For: 1.10.0 > > > I use the GoogleCloudStorageToGoogleCloudStorageOperator to copy a directory > recursively from source dir to target dir. When files are copied, the full > source dir hierarchy is included. > For example, I want to copy the following files to /path/to/destination > {{/path/to/source/a/file1.txt}} > {{/path/to/source/a/file2.txt}} > {{/path/to/source/b/file3.txt}} > > When running: > {{stage_files = GoogleCloudStorageToGoogleCloudStorageOperator(}} > task_id='task', > source_bucket='gs_bucket', > source_object='/path/to/source/*', > {{ destination_object='/path/to/destination')}} > I get the following result: > {{/path/to/destination/path/to/source/a/file1.txt}} > {{/path/to/destination/path/to/source/a/file2.txt}} > {{/path/to/destination/path/to/source/b/file3.txt}} > > I would expect the the operator behavior is similar to gsutil and would > return the following result: > {{/path/to/destination/a/file1.txt}} > {{/path/to/destination/a/file2.txt}} > {{/path/to/destination/b/file3.txt}} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)