shahar1 commented on code in PR #65724:
URL: https://github.com/apache/airflow/pull/65724#discussion_r3177623486
##########
providers/google/src/airflow/providers/google/cloud/transfers/s3_to_gcs.py:
##########
@@ -210,6 +210,9 @@ def _get_files(self, context: Context, gcs_hook: GCSHook)
-> list[str]:
# use the super method to list all the files in an S3 bucket/key
s3_objects = super().execute(context)
+ # Skip S3 folder-marker keys (``.../``); they overlap sibling keys in
Storage Transfer includePrefixes.
+ s3_objects = [obj for obj in s3_objects if not obj.endswith("/")]
Review Comment:
Technically object names in S3 may end in slash, so treating every key that
ends with / as a folder marker is not safe. This turns the patch into a
behavior change and can silently drop valid objects from both transfer and XCom
output. I would rather keep original key semantics and fix only the deferrable
includePrefixes overlap path, plus add a regression test for a real
slash-suffixed key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
