Dawnpool commented on code in PR #47533:
URL: https://github.com/apache/airflow/pull/47533#discussion_r1988239758
##########
providers/sftp/src/airflow/providers/sftp/operators/sftp.py:
##########
@@ -171,9 +185,17 @@ def execute(self, context: Any) -> str | list[str] | None:
file_msg = f"from {_local_filepath} to
{_remote_filepath}"
self.log.info("Starting to transfer file %s", file_msg)
if os.path.isdir(_local_filepath):
- self.sftp_hook.store_directory(
- _remote_filepath, _local_filepath,
confirm=self.confirm
- )
+ if self.concurrency > 1:
+ self.sftp_hook.store_directory_concurrently(
+ _remote_filepath,
+ _local_filepath,
+ confirm=self.confirm,
+ workers=self.concurrency,
+ )
+ elif self.concurrency == 1:
Review Comment:
Hi @Adaverse , Thanks for your comment.
Since the `local_filepath` and `remote_filepath` parameters already accept a
list of strings, users can already pass multiple files at once. Did you mean
adding concurrency not only for directory but also for passing multiple files
as well? If so, I think there should be quite a lot structural changes in the
operator code.
I'd prefer to focus only on directory transfer in this PR. Once this is
merged, I can create a separate PR for your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
