yuseok89 commented on code in PR #65724:
URL: https://github.com/apache/airflow/pull/65724#discussion_r3189391833
##########
providers/google/src/airflow/providers/google/cloud/transfers/s3_to_gcs.py:
##########
@@ -210,6 +210,9 @@ def _get_files(self, context: Context, gcs_hook: GCSHook)
-> list[str]:
# use the super method to list all the files in an S3 bucket/key
s3_objects = super().execute(context)
+ # Skip S3 folder-marker keys (``.../``); they overlap sibling keys in
Storage Transfer includePrefixes.
+ s3_objects = [obj for obj in s3_objects if not obj.endswith("/")]
Review Comment:
I pushed the patch that matches the proposal in my earlier comment. If
anything here does not fit what you want in production, say the word and I will
roll it back or narrow the scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
