yuseok89 commented on PR #72638: URL: https://github.com/apache/airflow/pull/72638#issuecomment-5835651209
Thanks, this makes sense. I checked it against a real SFTP server, and you're right that writing name/ as a file only comes from the `normpath` change. This PR set out to stop folder markers from breaking the transfer, but I think it works better built around one rule, that an empty folder is never silently dropped, which is also where the S3 side landed. On SFTP that would mean the following. - A trailing-slash key is a folder, so it creates that directory and is never written as a file. - With move_object, the marker is deleted after its directory is created, like any other moved object. With the rule in place, the logic this PR added to guard those corner cases should go away as well. For the switch, I wonder if `create_intermediate_dirs` could serve the same purpose instead of a new parameter. When it is False, the markers would be skipped with a log line and left in the bucket. Settling the rule this way does mean the default differs from what you suggested. I'd also extend its docstring to cover markers. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
