[
https://issues.apache.org/jira/browse/AIRFLOW-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17374038#comment-17374038
]
ASF GitHub Bot commented on AIRFLOW-5703:
-----------------------------------------
AngryHelper opened a new pull request #16792:
URL: https://github.com/apache/airflow/pull/16792
closes: AIRFLOW-5703
Added next features: transfer list of files, transfer directory, transfer
files on regex pattern in directories.
<details>
<summary>I writen next dag for tests</summary>
```
from airflow.utils.edgemodifier import Label
from airflow.operators.bash import BashOperator
from airflow import DAG
from airflow.providers.sftp.operators.sftp import SFTPOperator, SFTPOperation
from airflow.providers.ssh.operators.ssh import SSHOperator
from airflow.utils.dates import days_ago
with DAG(
dag_id="test_stfp",
schedule_interval="@once",
start_date=days_ago(1),
max_active_runs=1
) as dag:
create_test_files = BashOperator(
task_id="create_test_files",
bash_command="""
touch /tmp/file.txt & echo "transfer data from local" >>
/tmp/file.txt &
touch /tmp/file1.txt & echo "transfer data from local" >>
/tmp/file1.txt &
touch /tmp/file2.txt & echo "transfer data from local" >>
/tmp/file2.txt &
touch /tmp/file3.txt & echo "transfer data from local" >>
/tmp/file3.txt &
mkdir /tmp/dir_for_remote_transfer &
touch /tmp/dir_for_remote_transfer/file2.txt &
touch /tmp/dir_for_remote_transfer/file3.csv &
ls /tmp """
)
put_file = SFTPOperator(
task_id="put_file",
ssh_conn_id="ssh_default",
local_filepath="/tmp/file.txt",
remote_filepath="/tmp/tmp1/tmp2/file.txt",
operation=SFTPOperation.PUT,
create_intermediate_dirs=True
)
put_file_without_full_remote_path = SFTPOperator(
task_id="put_file_without_full_remote_path",
ssh_conn_id="ssh_default",
local_filepath="/tmp/file3.txt",
remote_filepath="/tmp/tmp1/tmp2/",
operation=SFTPOperation.PUT,
create_intermediate_dirs=True
)
put_dir_txt_files = SFTPOperator(
task_id="put_dir_txt_files",
ssh_conn_id="ssh_default",
local_folder="/tmp/dir_for_remote_transfer/",
remote_folder="/tmp/dir_for_remote_transfer/txt",
regexp_mask=".*.txt",
operation=SFTPOperation.PUT,
create_intermediate_dirs=True
)
put_dir_csv_files = SFTPOperator(
task_id="put_dir_csv_files",
ssh_conn_id="ssh_default",
local_folder="/tmp/dir_for_remote_transfer/",
remote_folder="/tmp/dir_for_remote_transfer/csv/",
regexp_mask=".*.csv",
operation=SFTPOperation.PUT,
create_intermediate_dirs=True
)
put_files = SFTPOperator(
task_id="put_files",
ssh_conn_id="ssh_default",
local_filepath=["/tmp/file1.txt", "/tmp/file2.txt"],
remote_filepath="/tmp/tmp1/tmp2/",
operation=SFTPOperation.PUT,
create_intermediate_dirs=True
)
drop_local_test_files = BashOperator(
task_id="drop_local_test_files",
bash_command="""rm -rf /tmp/file* & rm -rf
/tmp/dir_for_remote_transfer/ & ls /tmp/"""
)
get_file = SFTPOperator(
task_id="get_file",
ssh_conn_id="ssh_default",
local_filepath="/tmp/file.txt",
remote_filepath="/tmp/tmp1/tmp2/file.txt",
operation=SFTPOperation.GET,
create_intermediate_dirs=True
)
get_files = SFTPOperator(
task_id="get_files",
ssh_conn_id="ssh_default",
local_filepath="/tmp/",
remote_filepath=["/tmp/tmp1/tmp2/file1.txt",
"/tmp/tmp1/tmp2/file2.txt"],
operation=SFTPOperation.GET,
create_intermediate_dirs=True
)
get_dir = SFTPOperator(
task_id="get_dir",
ssh_conn_id="ssh_default",
local_folder="/tmp/dir_for_remote_transfer/csv",
remote_folder="/tmp/dir_for_remote_transfer/csv",
operation=SFTPOperation.GET,
create_intermediate_dirs=True
)
drop_remote_files = SSHOperator(
task_id="drop_remote_files",
ssh_conn_id="ssh_default",
command="rm -rf /tmp/tmp1 & ls /tmp/"
)
create_test_files >> Label("transfer data to remote server") >> \
[put_file, put_files, put_file_without_full_remote_path,
put_dir_txt_files, put_dir_csv_files] >> \
drop_local_test_files >> Label("transfer data to local server") >> \
[get_file, get_files, get_dir] >> drop_remote_files
```
</details>
<details>
<summary>Pre-commit Tests</summary>
```
ubuntu@ubuntu-advanced-8-16-10gb:~/airflow$ pre-commit run --files
airflow/providers/sftp/operators/sftp.py
identity.................................................................................Passed
- hook id: identity
- duration: 0.04s
airflow/providers/sftp/operators/sftp.py
Check hooks apply to the repository..................................(no
files to check)Skipped
No-tabs
checker..........................................................................Passed
Add license for all SQL files........................................(no
files to check)Skipped
Add license for all other files......................................(no
files to check)Skipped
Add license for all rst files........................................(no
files to check)Skipped
Add license for all JS/TS/TSX/CSS/PUML files.........................(no
files to check)Skipped
Add license for all JINJA template files.............................(no
files to check)Skipped
Add license for all shell files......................................(no
files to check)Skipped
Add license for all Python
files.........................................................Passed
Add license for all XML files........................................(no
files to check)Skipped
Add license for all YAML files.......................................(no
files to check)Skipped
Add license for all md files.........................................(no
files to check)Skipped
Add license for all mermaid files....................................(no
files to check)Skipped
Add TOC for md files.................................................(no
files to check)Skipped
black....................................................................................Passed
blacken-docs.............................................................................Passed
Check for merge
conflicts................................................................Passed
Debug Statements
(Python)................................................................Passed
Check builtin type constructor
use.......................................................Passed
Detect Private
Key.......................................................................Passed
Fix End of
Files.........................................................................Passed
Mixed line
ending........................................................................Passed
Check that executables have shebangs.................................(no
files to check)Skipped
Check Xml............................................................(no
files to check)Skipped
Trim Trailing
Whitespace.................................................................Passed
Fix python encoding
pragma...............................................................Passed
Pretty format JSON...................................................(no
files to check)Skipped
pyupgrade................................................................................Passed
rst ``code`` is two backticks........................................(no
files to check)Skipped
use
logger.warning(......................................................................Passed
Check YAML files with yamllint.......................................(no
files to check)Skipped
Run isort to sort imports in Python
files................................................Passed
Run
pydocstyle...........................................................................Passed
Strip unnecessary `#
noqa`s..............................................................Passed
Lint OpenAPI using spectral..........................................(no
files to check)Skipped
Lint OpenAPI using openapi-spec-validator............................(no
files to check)Skipped
Lint dockerfile......................................................(no
files to check)Skipped
Check order of dependencies in setup.cfg and setup.py................(no
files to check)Skipped
Checks setup extra packages..........................................(no
files to check)Skipped
Update output of breeze command in BREEZE.rst........................(no
files to check)Skipped
Update mounts in the local yml file..................................(no
files to check)Skipped
Update setup.cfg file with all licenses..............................(no
files to check)Skipped
Build cross-dependencies for providers
packages..........................................Passed
Update extras in documentation.......................................(no
files to check)Skipped
Check for pydevd debug statements accidentally
left......................................Passed
Don't use safe in templates..........................................(no
files to check)Skipped
No providers imports in core example DAGs............................(no
files to check)Skipped
No relative
imports......................................................................Passed
Check for language that we do not accept as
community....................................Passed
Check BaseOperator[Link] core imports................................(no
files to check)Skipped
Check BaseOperator[Link] other
imports...................................................Passed
Check provide_session and create_session
imports.........................................Passed
Make sure LoggingMixin is not used
alone.................................................Passed
Make sure days_ago is imported from
airflow.utils.dates..................................Passed
'start_date' should not be defined in default_args in example_dags...(no
files to check)Skipped
Check if integration list is aligned.................................(no
files to check)Skipped
Check if licenses are OK for Apache..................................(no
files to check)Skipped
Checks for consistency between config.yml and default_config.cfg.....(no
files to check)Skipped
Verify that DB Migrations have been documented.......................(no
files to check)Skipped
Checks for Boring Cyborg configuration
consistency.......................................Passed
Sort INTHEWILD.md alphabetically.....................................(no
files to check)Skipped
Sort alphabetically and uniquify spelling_wordlist.txt...............(no
files to check)Skipped
Lint Helm Chart......................................................(no
files to check)Skipped
Check Shell scripts syntax correctness...............................(no
files to check)Skipped
Run BATS bash tests for changed Breeze bash files....................(no
files to check)Skipped
Run BATS bash tests for changed bash files...........................(no
files to check)Skipped
stylelint............................................................(no
files to check)Skipped
Provider init
file.......................................................................Passed
Validate providers.yaml files........................................(no
files to check)Skipped
Generate mermaid images..............................................(no
files to check)Skipped
Check if pre-commits are described...................................(no
files to check)Skipped
Ensure hook ids are not overly long..................................(no
files to check)Skipped
Checks providers available when declared by extras in
setup.py...........................Passed
Run markdownlint.....................................................(no
files to check)Skipped
Lint JSON Schema files with JSON Schema..............................(no
files to check)Skipped
Lint NodePort Service with JSON Schema...............................(no
files to check)Skipped
Lint Docker compose files with JSON Schema...........................(no
files to check)Skipped
Lint chart/values.schema.json file with JSON Schema..................(no
files to check)Skipped
Lint chart/values.yaml file with JSON Schema.........................(no
files to check)Skipped
Lint airflow/config_templates/config.yml file with JSON Schema.......(no
files to check)Skipped
Convert to f-strings with
flynt..........................................................Passed
ESLint against airflow/ui............................................(no
files to check)Skipped
ESLint against current UI js files...................................(no
files to check)Skipped
Check if image build is
needed...........................................................Passed
Run in container bats tests..........................................(no
files to check)Skipped
Run
mypy.................................................................................Passed
Run mypy for helm chart tests........................................(no
files to check)Skipped
Run mypy for /docs/ folder...........................................(no
files to check)Skipped
Run
flake8...............................................................................Passed
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Airflow - SFTP Operator for multiple files
> ------------------------------------------
>
> Key: AIRFLOW-5703
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5703
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Affects Versions: 1.10.5
> Reporter: Mattia
> Assignee: Andrey Kucherov
> Priority: Major
>
> *AS* User
> *I WANT TO* download / upload multiple files from sftp server
> *SO THAT* i need the possibility to add a list of file instead of single one
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
