turbaszek commented on a change in pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#discussion_r450785525
##########
File path: tests/providers/google/cloud/transfers/test_gcs_to_gcs.py
##########
@@ -86,6 +86,19 @@ def test_execute_no_suffix(self, mock_hook):
TEST_BUCKET, prefix="test_object", delimiter=""
)
+ @mock.patch('airflow.providers.google.cloud.transfers.gcs_to_gcs.GCSHook')
+ def test_execute_wildcard_with_replace_flag_false(self, mock_hook):
+ operator = GCSToGCSOperator(
+ task_id=TASK_ID, source_bucket=TEST_BUCKET,
+ source_object=SOURCE_OBJECT_WILDCARD_SUFFIX,
+ destination_bucket=DESTINATION_BUCKET,
+ replace=False)
+
+ operator.execute(None)
+ mock_hook.return_value.list.assert_called_once_with(
+ TEST_BUCKET, prefix="test_object", delimiter="", replace=False
+ )
Review comment:
This method is called two times during execute: L282 and L288 so we
should do something like
```python
calls = [
mock.Call(TEST_BUCKET, prefix="test_object", delimiter=""),
mock.Call(DESTINATION_BUCKET, prefix="test_object", delimiter=""),
]
mock_hook.return_value.assert_has_calls(calls)
```
##########
File path: airflow/providers/google/cloud/transfers/gcs_to_gcs.py
##########
@@ -276,6 +280,21 @@ def _copy_source_with_wildcard(self, hook, prefix):
self.log.info('Delimiter ignored because wildcard is in prefix')
prefix_, delimiter = prefix.split(WILDCARD, 1)
objects = hook.list(self.source_bucket, prefix=prefix_,
delimiter=delimiter)
+ if not self.replace:
+ # If we are not replacing, list all files in the Destination GCS
bucket
+ # and only keep those files which are present in
+ # Source GCS bucket and not in Destination GCS bucket
+
+ existing_objects = hook.list(self.destination_bucket,
prefix=prefix_, delimiter=delimiter)
+
+ objects = list(set(objects) - set(existing_objects))
Review comment:
```suggestion
objects = set(objects) - set(existing_objects)
```
Is there any particular reason to use list?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
