ITC00kie opened a new issue, #51151:
URL: https://github.com/apache/airflow/issues/51151
### Apache Airflow Provider(s)
sftp
### Versions of Apache Airflow Providers
$ pip list | grep sftp
apache-airflow-providers-sftp 5.3.0
### Apache Airflow version
Version: v2.11.0 Git Version:
.release:d9ed7b94d033dc61ee1c05aafb9f1c1f7b82cfb2
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Virtualenv installation
### Deployment details
$ python3 --version
Python 3.11.2
### What happened
We you a SFTPSensor to check if a log file exists on the remote and is
therefore ready for download. We switched to Airflow v2.11.0 this week to
prepare for the v3 migration. Our workflow, which uses the SFTPSensor,
sometimes was behaving abnormal since the update to v2.11.0. We investigated
some time in troubleshooting and found that the SFTPSensor only poke one time
and then exists with success whether the file is there or not.
### What you think should happen instead
The SFTPSensor should keep poking until the expected file is available.
### How to reproduce
Create an SFTPSensor with these parameters and let it listen for a remote
file.
task_id='watch_for_log',
sftp_conn_id='my_conn_id',
path='/my_log_file.log',
poke_interval=3
### Anything else
We came up with a fix for this, that works in our environment.
In the file providers/sftp/src/airflow/providers/sftp/sensors/sftp.py in
line 98 to 104 replace the try-block with this code. The bug here is, that the
`OSError` is already excepted in `self.hook.isfile` and therefore never
excepted in the `poke` funtion. The `self.hook.isfile` returns `False` in case
of an error. With this we just have to check if the file exists before passing
the `self.path` to `actual_files_to_check`.
```python
try:
actual_files_to_check = [self.path] if
self.hook.isfile(self.path) else []
except OSError as e:
if e.errno != SFTP_NO_SUCH_FILE:
raise AirflowException from e
actual_files_to_check = []
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
