[ https://issues.apache.org/jira/browse/AIRFLOW-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christos Soulios updated AIRFLOW-2348: -------------------------------------- Description: I use the GoogleCloudStorageToGoogleCloudStorageOperator to copy a directory recursively from source dir to target dir. When files are copied, the full source dir hierarchy is included. For example, I want to copy the following files to /path/to/destination {{/path/to/source/a/file1.txt}} {{/path/to/source/a/file2.txt}} {{/path/to/source/b/file3.txt}} When running: {{stage_files = GoogleCloudStorageToGoogleCloudStorageOperator(}} task_id='task', source_bucket='gs_bucket', source_object='/path/to/source/*', {{ destination_object='/path/to/destination')}} I get the following result: {{/path/to/destination/path/to/source/a/file1.txt}} {{/path/to/destination/path/to/source/a/file2.txt}} {{/path/to/destination/path/to/source/b/file3.txt}} I would expect the the operator behavior is similar to gsutil and would return the following result: {{/path/to/destination/a/file1.txt}} {{/path/to/destination/a/file2.txt}} {{/path/to/destination/b/file3.txt}} was: I use the GoogleCloudStorageToGoogleCloudStorageOperator to copy a directory recursively from source dir to target dir. When files are copied, the full source dir hierarchy is included. For example, I want to copy the following files to /path/to/destination {{/path/to/source/a/file1.txt}} {{/path/to/source/a/file2.txt}} {{/path/to/source/b/file3.txt}} When running: {{stage_files = GoogleCloudStorageToGoogleCloudStorageOperator(}} {{ task_id='task',}} {{ source_bucket='gs_bucket',}} {{ source_object='/path/to/source/*',}} {{destination_object='/path/to/destination')}} I get the following result: {{/path/to/destination/path/to/source/a/file1.txt}} {{/path/to/destination/path/to/source/a/file2.txt}} {{/path/to/destination/path/to/source/b/file3.txt}} I would expect the the operator behavior is similar to gsutil and would return the following result: {{/path/to/destination/a/file1.txt}} {{/path/to/destination/a/file2.txt}} {{/path/to/destination/b/file3.txt}} > GoogleCloudStorageToGoogleCloudStorageOperator copies full source path > ---------------------------------------------------------------------- > > Key: AIRFLOW-2348 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2348 > Project: Apache Airflow > Issue Type: Bug > Components: gcp > Affects Versions: 2.0.0 > Reporter: Christos Soulios > Priority: Minor > > I use the GoogleCloudStorageToGoogleCloudStorageOperator to copy a directory > recursively from source dir to target dir. When files are copied, the full > source dir hierarchy is included. > For example, I want to copy the following files to /path/to/destination > {{/path/to/source/a/file1.txt}} > {{/path/to/source/a/file2.txt}} > {{/path/to/source/b/file3.txt}} > > When running: > {{stage_files = GoogleCloudStorageToGoogleCloudStorageOperator(}} > task_id='task', > source_bucket='gs_bucket', > source_object='/path/to/source/*', > {{ destination_object='/path/to/destination')}} > I get the following result: > {{/path/to/destination/path/to/source/a/file1.txt}} > {{/path/to/destination/path/to/source/a/file2.txt}} > {{/path/to/destination/path/to/source/b/file3.txt}} > > I would expect the the operator behavior is similar to gsutil and would > return the following result: > {{/path/to/destination/a/file1.txt}} > {{/path/to/destination/a/file2.txt}} > {{/path/to/destination/b/file3.txt}} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)