kalluripradeep commented on PR #67372:
URL: https://github.com/apache/airflow/pull/67372#issuecomment-4773012973
> This still misses the scheduler executor-event path that triggers the
KubernetesExecutor symptom. A failed executor event for a TI still in
`SCHEDULED`/`QUEUED` enters the state-mismatch path, then calls
`ti.handle_failure()`. But `handle_failure()` only records history when
`ti.state == RUNNING`; otherwise it sets `UP_FOR_RETRY` without
`prepare_db_for_next_try()`. Please add/fix the scheduler executor-event retry
path and add a regression test where a non-running queued/scheduled TI receives
a failed executor event and preserves
`TaskInstanceHistory.hostname`/`start_date`.
Thanks for the detailed review — you're exactly right.
The root cause is in `fetch_handle_failure_context()` (line 1809 in
`taskinstance.py`):
```python
# Before
if ti.state == TaskInstanceState.RUNNING:
ti.prepare_db_for_next_try(session)
```
When the executor sends a FAILED event for a QUEUED/SCHEDULED/DEFERRED TI,
`handle_failure()` is called but `ti.state != RUNNING`, so
`prepare_db_for_next_try()` — and therefore `TaskInstanceHistory.record_ti()` —
was never invoked. History lost, hostname gone.
Fixed by changing the guard to exclude only `RESTARTING` (which already had
`prepare_db_for_next_try` called during the clear operation):
```python
# After
if ti.state != TaskInstanceState.RESTARTING:
# Covers RUNNING (normal exception), QUEUED/SCHEDULED/DEFERRED (external
kill)
ti.prepare_db_for_next_try(session)
```
Added a parametrized regression test
`test_process_executor_events_queued_ti_retry_preserves_history` covering both
`QUEUED` and `SCHEDULED` states — verifies that after a FAILED executor event:
TI gets a new UUID (confirming `prepare_db_for_next_try` ran), a
`TaskInstanceHistory` row exists, and `hostname`/`start_date` are preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
