Abhishek1559 commented on issue #36734:
URL: https://github.com/apache/airflow/issues/36734#issuecomment-3160138790
Hi maintainers,
Based on the discussion till now and testing, I have replicated the issue
with FileSensor and S3 sensor DAGs.
Following are the observations I had:
The issue no longer exists in version 3.x.x.
Tested for S3 as well as Filesensor to confirm the behavior for different
sensor dags and found out that dags in standard as well as deferrable mode,
fail as expected and do not go for retry.
The DAGs in both the modes acted as expected and threw an
‘AirflowSensorTimeout’ as expected in both the cases.
The same helps me conclude that the issue no longer exists and this can be
closed now.
Test 1.
While testing for the bug (Airflow 3.1.0) with FileSensor, I followed the
following code snippet:
‘’’
from airflow import DAG
from airflow.sensors.filesystem import FileSensor
from datetime import datetime, timedelta
default_args = {
'owner': 'you',
'start_date': datetime(2025, 7, 1)
}
with DAG(
dag_id="test_sensor_timeout_bug",
default_args=default_args,
catchup=False,
tags=["sensor", "timeout"],
) as dag:
# Poke mode sensor (normal blocking)
poke_sensor = FileSensor(
task_id="poke_sensor",
filepath="/tmp/nonexistent_file.txt",
retries=3,
poke_interval=5,
timeout=10,
mode='reschedule',
deferrable=False,
fs_conn_id="fs_local",
)
# Reschedule (deferrable) sensor — to reproduce the bug
deferred_sensor = FileSensor(
task_id="deferred_sensor",
filepath="/tmp/nonexistent_file.txt",
retries=3,
timeout=10,
deferrable=True,
fs_conn_id="fs_local"
)
‘’’
The DAGs in both the modes acted as expected and threw an
‘AirflowSensorTimeout’ as expected and failed immediately in both the cases as
shown in the below logs.
Poke mode:
<img width="1854" height="964" alt="Image"
src="https://github.com/user-attachments/assets/e362c65d-6a59-4bea-bb70-84cef05bc1c0";
/>
Deferrable mode:
<img width="1854" height="952" alt="Image"
src="https://github.com/user-attachments/assets/743c26ef-9c6d-4033-9c24-88cba8efcb1f";
/>
Test 2:
While testing for the bug (Airflow 3.1.0) with S3, I followed the following
code snippet:
‘’’
from datetime import datetime
from airflow import models
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
with models.DAG(
dag_id='sensor_timeout',
start_date=datetime(2018, 10, 31),
schedule='0 7 * * 4',
catchup=False,
max_active_runs=5,
):
sensor = S3KeySensor(
task_id='sensor_test',
aws_conn_id='aks_aws',
bucket_name='s3-bucket-name', # s3 bucket name
bucket_key='path/to/an/object', #path to object
retries=3,
wildcard_match=True,
poke_interval=2,
timeout=10,
mode='reschedule',
deferrable=False
)
sensor_defer = S3KeySensor(
task_id='sensor_test_defer',
aws_conn_id='aks_aws',
bucket_name='s3-bucket-name', # s3 bucket name
bucket_key='path/to/an/object', #path to object
retries=3,
wildcard_match=True,
timeout=10,
deferrable=True,
)
‘’’
The DAGs in both the modes acted as expected and threw an
‘AirflowSensorTimeout’ as expected and failed immediately in both the cases as
shown in the below logs.
Poke mode:
<img width="1854" height="960" alt="Image"
src="https://github.com/user-attachments/assets/2d9dce71-f013-483a-8e5e-4a6de563e92d";
/>
Deferrable mode:
<img width="1854" height="964" alt="Image"
src="https://github.com/user-attachments/assets/88616a58-92e9-407b-978c-6d49862bee6d";
/>
This confirms the consistent behavior for S3 Sensor Dags in both the modes.
They are performing as expected and failing immediately.
The inconsistency does not exist as shown in the testing. If you find the
research and testing relevant, we can close the issue.
Anything that I have missed/misunderstood or any suggestions or feedback is
welcomed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
