Ferdinanddb opened a new issue, #77255:
URL: https://github.com/apache/airflow/issues/77255

   ### Under which category would you file this issue?
   
   Task SDK
   
   ### Apache Airflow version
   
   3.4.0 (task-sdk 1.4.0, apache-airflow-providers-standard 1.21.0)
   
   ### What happened and how to reproduce it?
   
   The [Timeouts section of the task 
docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts)
 says that a sensor that fails for another reason (a network outage, say) "can 
retry up to 2 times as defined by `retries`. Retrying does not reset the 
`timeout`. It will still have up to 3600 seconds in total for it to succeed."
   
   In Airflow 3 every retry gets a brand new `timeout`, both for sensors in 
`reschedule` mode and for deferrable sensors. Both repros below use 
`timeout=120`, crash on try 1 about 60 s in, and retry once.
   
   **Reschedule mode** (`PythonSensor`): try 2 starts with no reschedule 
history and waits another full 120 s, so the sensor times out 193 s after its 
first poke:
   
   ```
   try=1 reschedules_seen=0 now=18:04:18
   try=1 reschedules_seen=1 now=18:04:48
   try=1 reschedules_seen=2 now=18:05:19   -> RuntimeError: simulated crash, 
up_for_retry
   try=2 reschedules_seen=0 now=18:05:29   <- the try starts with no reschedule 
history
   try=2 reschedules_seen=1 now=18:05:59
   try=2 reschedules_seen=2 now=18:06:30
   try=2 reschedules_seen=3 now=18:07:00
   try=2 reschedules_seen=4 now=18:07:31
   AirflowSensorTimeout: Sensor has timed out; run duration of 121.99157 
seconds exceeds the specified timeout of 120.0.
   ```
   
   **Deferrable mode** (`ExternalTaskSensor`, `deferrable=True`): try 2 defers 
with a deadline of its own defer time + 120 s. From the metadata DB (UTC):
   
   ```
   try_number=1  start_date=17:57:59  -> deferred, RuntimeError: simulated 
crash at 17:59:02, up_for_retry
   try_number=2  start_date=17:59:12  trigger_timeout=18:01:12   <- 120 s after 
try 2 deferred, not after try 1 started
   ```
   
   ---
   
   ### To reproduce:
   
   **Reschedule mode:**
   
   ```python
   from datetime import datetime, timedelta
   
   from airflow.providers.standard.sensors.python import PythonSensor
   from airflow.sdk import DAG
   
   
   def _never_ready(ti, task_reschedule_count, **_):
       # Try 1 crashes on its third poke (~60 s in), like a worker restart 
would.
       print(f"try={ti.try_number} reschedules_seen={task_reschedule_count} 
now={datetime.now():%H:%M:%S}")
       if ti.try_number == 1 and task_reschedule_count >= 2:
           raise RuntimeError("simulated crash")
       return False
   
   
   with DAG("sensor_timeout_reset_on_retry", start_date=datetime(2026, 1, 1), 
schedule=None, catchup=False):
       PythonSensor(
           task_id="wait",
           python_callable=_never_ready,
           mode="reschedule",
           poke_interval=30,
           timeout=120,  # docs: "Retrying does not reset the timeout"
           retries=1,
           retry_delay=timedelta(seconds=10),
       )
   ```
   
   Run `airflow dags test sensor_timeout_reset_on_retry` (a fresh SQLite 
metadata DB is enough) and compare the first `try=1` line with the 
`AirflowSensorTimeout` time.
   
   **Deferrable mode (a real `ExternalTaskSensor`, except that try 1 crashes 
after ~60 s of deferral):**
   
   ```python
   from datetime import datetime, timedelta
   
   from airflow.providers.standard.sensors.external_task import 
ExternalTaskSensor
   from airflow.providers.standard.triggers.temporal import TimeDeltaTrigger
   from airflow.sdk import DAG
   
   
   class CrashOnceExternalTaskSensor(ExternalTaskSensor):
       """A real ExternalTaskSensor, except try 1 crashes after ~60 s of 
deferral."""
   
       def execute(self, context):
           if context["ti"].try_number == 1:
               self.defer(trigger=TimeDeltaTrigger(timedelta(seconds=60)), 
method_name="crash")
           super().execute(context)
   
       def crash(self, context, event=None):
           raise RuntimeError("simulated crash")
   
   
   with DAG("upstream_never_runs", start_date=datetime(2026, 1, 1), 
schedule=None, catchup=False):
       pass
   
   with DAG("deferred_sensor_timeout_reset_on_retry", start_date=datetime(2026, 
1, 1), schedule=None, catchup=False):
       CrashOnceExternalTaskSensor(
           task_id="wait",
           external_dag_id="upstream_never_runs",
           external_task_id=None,
           deferrable=True,
           poke_interval=30,
           timeout=120,  # docs: "Retrying does not reset the timeout"
           retries=1,
           retry_delay=timedelta(seconds=10),
       )
   ```
   
   `airflow dags test` runs triggers inline and does not enforce trigger 
timeouts, so once try 2 has deferred, read its deadline from the metadata DB:
   
   ```sql
   SELECT try_number, start_date, trigger_timeout
   FROM task_instance
   WHERE dag_id = 'deferred_sensor_timeout_reset_on_retry';
   ```
   
   ### What you think should happen instead?
   
   As documented: a retry only gets what is left of `timeout`, counted from the 
first try. In the repros, try 2 should time out around 18:06:18 (reschedule) 
and its `trigger_timeout` should be about 17:59:59 (deferrable).
   
   **Reschedule mode.** The task SDK still intends this: 
`RuntimeTaskInstance.get_first_reschedule_date` computes `first_try_number = 
max_tries - retries + 1` to anchor the timeout to the first try. That anchoring 
can no longer work:
   
   - Each retry is a new task instance row with a new id: 
`TaskInstance.prepare_db_for_next_try` inserts the successor with `id=uuid7()` 
(`airflow-core/src/airflow/models/taskinstance.py`).
   - The execution API's `ti_run` counts reschedules and their first 
`start_date` with `.where(TaskReschedule.ti_id == task_instance_id)` 
(`airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py`),
 i.e. for the current try only.
   - So on a new try `task_reschedule_count` is 0, `get_first_reschedule_date` 
returns `None` before it reaches the `first_try_number` lookup, and 
`BaseSensorOperator.execute` starts the clock again.
   
   **Deferrable mode.** Deferrable sensors pass their timeout to 
`self.defer(timeout=timedelta(seconds=self.timeout), ...)` (e.g. 
`ExternalTaskSensor.execute` in the standard provider), which is measured from 
the moment each try defers. Nothing subtracts the time already spent in earlier 
tries.
   
   ### Operating System
   
   Linux
   
   ### Deployment
   
   Other
   
   ### Apache Airflow Provider(s)
   
   standard
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-standard 1.21.0
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   - Why it matters: `timeout` is the only bound on how long a sensor can wait. 
If each retry gets a fresh one, a sensor with `retries=2` and `timeout=12h` can 
wait up to ~36 h, and a DAG with a cutoff has to set `retries=0` on its 
sensors, giving up protection against worker crashes.
   - A possible fix: record when the first try started waiting (for example the 
first reschedule `start_date` across all tries of the same `(dag_id, run_id, 
task_id, map_index)` since the last clear, or a start time carried over to the 
successor in `prepare_db_for_next_try`), and use it in both paths: 
`BaseSensorOperator.execute` for reschedule/poke mode, and the timeout passed 
to `defer()` for deferrable sensors. Clearing a task should still start a new 
window, as it does today.
   - If the new behaviour is intended instead, the Timeouts section of the docs 
needs updating.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to