he-gong-sardine opened a new pull request, #36545:
URL: https://github.com/apache/airflow/pull/36545
closes: #34995
---
### Overview:
The current implementation in Airflow allows file transfer from SFTP to GCS
by first downloading the file to the worker's local storage, the
boring-cyborg[bot] commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1874725064
Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution
Lee-W commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1441262932
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -85,13 +106,18 @@ def __init__(
source_path: str,
destination_bucket: str,
dirrao commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1876651595
> * High Memory (Over 100MB): It is recommended to use Google's
upload_from_file method, which is more robust and includes a retry feature.
This method is preferable when sufficient memo
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1443975480
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %s",
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1443975333
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %s",
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1443975333
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %s",
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1445988080
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -85,13 +106,18 @@ def __init__(
source_path: str,
destination_bucket: str,
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1445989162
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1446051467
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %s",
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1500592065
##
tests/providers/google/cloud/transfers/test_sftp_to_gcs.py:
##
@@ -252,3 +254,60 @@ def test_execute_more_than_one_wildcard_exception(self,
sftp_hook, gcs
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1500610358
##
licenses/LICENSES-ui.txt:
##
Review Comment:
thanks. I don't know how this file was generated. removed.
--
This is an automated message from the
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1500610842
##
docs/apache-airflow-providers-google/operators/transfer/sftp_to_gcs.rst:
##
@@ -93,6 +93,33 @@ and ``tests_sftp_hook_dir/subdir/parent-2.bin`` is copied to
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1500613391
##
docs/apache-airflow-providers-google/operators/transfer/sftp_to_gcs.rst:
##
@@ -93,6 +93,33 @@ and ``tests_sftp_hook_dir/subdir/parent-2.bin`` is copied to
he-gong-sardine commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1961757630
Thank you guys. updated. @potiuk @Lee-W @elaye-canopy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
eladkal commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1961856408
doc build is failing on spell check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
eladkal commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1901997539
> Documentation updates are pending. I am currently seeking guidance on
locating the source files for documentation to ensure all necessary updates are
made.
Assuming you want to
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1460688387
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -160,10 +160,8 @@ def execute(self, context: Context):
for file in files:
he-gong-sardine commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1913579619
> > Documentation updates are pending. I am currently seeking guidance on
locating the source files for documentation to ensure all necessary updates are
made.
>
> Assumi
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468859918
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468859918
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %
he-gong-sardine commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1913606077
> > * High Memory (Over 100MB): It is recommended to use Google's
upload_from_file method, which is more robust and includes a retry feature.
This method is preferable when suff
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468866487
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -160,10 +160,8 @@ def execute(self, context: Context):
for file in files:
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468869304
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %
he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468869381
##
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %
potiuk commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1923769973
You will need to rebase to resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
he-gong-sardine commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1928712406
> You will need to rebase to resolve conflicts.
Thanks! Resolved conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
potiuk commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-1929268620
and docs/static checks now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
Lee-W commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1483118035
##
airflow/providers/sftp/provider.yaml:
##
@@ -60,7 +60,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- apache-airflow-providers-ssh>=2.1.0
- - paramiko
elaye-canopy commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1490760502
##
airflow/providers/sftp/provider.yaml:
##
@@ -60,7 +60,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- apache-airflow-providers-ssh>=2.1.0
- - p
github-actions[bot] commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-2126272202
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed in 5 days if no further activity occurs.
Thank you for you
elaye-canopy commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-2135038607
@Lee-W What is still missing? I am also using this feature and have
contributed some of the code to @he-gong-work
Maybe I could take over and finish it if they are busy.
--
T
Lee-W commented on PR #36545:
URL: https://github.com/apache/airflow/pull/36545#issuecomment-2135411519
@elaye-canopy Yep, that would be awesome! I think the major issue we have
now is spelling checks and conflicts. If my memory serves me right, other parts
seem to be good. But I can take a
33 matches
Mail list logo
