rustikk opened a new issue #22196:
URL: https://github.com/apache/airflow/issues/22196
### Apache Airflow version
2.2.3
### What happened
This dag is finding files with the FileSensor with connections where the
directory doesn't exist. I originally found this bug from switching from
airflow breeze to Astronomer's astro cli and I forgot to change the filepath of
the connection. Yet even with a directory that doesn't exist the glob character
is finding files.
### What you expected to happen
I expected the FileSensor to fail as the directory the tasks connection is
set to doesn't exist.
### How to reproduce
```
from airflow.models import DAG, Connection
from airflow.sensors.filesystem import FileSensor
from airflow.operators.python import PythonOperator
from airflow.hooks.base import BaseHook
from airflow.exceptions import AirflowNotFoundException
from airflow import settings
from datetime import datetime, timedelta
def add_conn(
conn_id: str,
conn_type: str,
host: str,
username: str,
pw: str,
scheme: str,
port: int
):
try:
found = BaseHook().get_connection(conn_id)
print("The connection has been made previously.")
except AirflowNotFoundException:
found = None
remote_connection = Connection(
conn_id=conn_id,
conn_type=conn_type,
host=host,
login=username,
password=pw,
schema=scheme,
port=port,
)
print(f"The connection that was created is: {remote_connection}")
session = settings.Session()
session.add(remote_connection)
session.commit()
def delete_conn(connection_id):
conn = Connection().get_connection_from_secrets(connection_id)
session = settings.Session()
session.delete(conn)
with DAG(
dag_id="filesystem_sensor_glob_negative",
start_date=datetime(2021, 1, 1),
schedule_interval=timedelta(days=365),
max_active_tasks=1,
max_active_runs=1,
doc_md=docs,
tags=["core", "negative", "file_system_sensor"]
) as dag:
conn_id = "FileSensorGlobNegative"
t0 = PythonOperator(
task_id="add_conn_introspected",
python_callable=add_conn,
op_args=[
conn_id,
"File",
"/opt/airflow/logs",
None,
None,
None,
None,
]
)
t1 = FileSensor(
task_id="sense_all_files",
fs_conn_id=conn_id,
filepath="**",
recursive=True,
timeout=300,
poke_interval=20,
)
t2 = PythonOperator(
task_id="del_conn",
python_callable=delete_conn,
op_args=[conn_id],
)
t0 >> t1 >> t2
```
### Operating System
Docker (debian:buster)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
Astro CLI with this image: quay.io/astronomer/ap-airflow-dev:2.2.3-2
### Anything else
Occurs everytime after multiple `docker system system prune --volumes`
commands
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
