yuseok89 commented on code in PR #73269:
URL: https://github.com/apache/airflow/pull/73269#discussion_r4079285165
##########
providers/amazon/src/airflow/providers/amazon/aws/transfers/s3_to_ftp.py:
##########
@@ -109,16 +109,25 @@ def execute(self, context: Context):
self.log.info("Getting files in s3://%s/%s", self.s3_bucket,
self.s3_key)
all_keys = s3_hook.list_keys(bucket_name=self.s3_bucket,
prefix=self.s3_key) or []
filenames = [k[len(self.s3_key) :] for k in all_keys]
- if self.s3_filenames == "*":
+ s3_prefix: str = self.s3_filenames
+ if s3_prefix == "*":
files = filenames
else:
- s3_prefix: str = self.s3_filenames
- files = [f for f in filenames if s3_prefix in f]
+ files = [f for f in filenames if f.startswith(s3_prefix)]
Review Comment:
Thanks, good catch. I went with the file name on the S3 side too, for both
matching and renaming, so a nested key keeps its directory.
I was a bit wary that this makes the prefix mean something other than an S3
key prefix. Since these operators bridge S3 to a filesystem, and `s3_key`
already carries the directory while `*_filenames` carries the file name, I
thought the file name reading fits better here. I noted it in the docstring and
the changelog. Happy to revisit if you see it differently. Either way, all four
operators now behave the same on layouts with subdirectories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
