ChickenBenny opened a new issue, #76529: URL: https://github.com/apache/airflow/issues/76529
### Under which category would you file this issue? Airflow Core ### Apache Airflow version main; the same code path is present in 3.3.2 ### What happened and how to reproduce it? Follow-up from #76191 (fixing #76190). The scheduler calls `ti.handle_failure()` for each task instance inside a loop, with nothing that isolates an exception raised for one task instance: - `SchedulerJobRunner.process_executor_events`, when the executor reports a failure the TI did not record itself - the heartbeat-timeout purge (`_purge_task_instances_without_heartbeats`) If `handle_failure()` raises for any reason, the exception leaves `_run_scheduler_loop` and the scheduler process exits. After a restart the same task instance is still in the same state, so it is picked up again, raises again, and the scheduler keeps crashing. In #76190 this was a `KeyError` from fail_fast handling. A task with `retries=0` reached try 13 in about 30 minutes, because each restart reset the TI as orphaned and ran it again. #76191 removes that specific `KeyError`, but any other exception raised from `handle_failure()` (a failure callback lookup, fail_fast handling, a data problem in one Dag run) has the same effect. One bad task instance can stop scheduling for every Dag. To reproduce, make `handle_failure()` raise for one task instance. For example, on a build without #76191, follow the steps in #76190. ### What you think should happen instead? An exception from `handle_failure()` for one task instance should not stop the scheduler. It should log the error with the task instance key, and leave the TI in a state that does not cause it to be picked up again in the same way, for example `FAILED` without the extra handling. It should then continue with the remaining task instances. Open points: - If the exception comes from the database, the transaction may no longer be usable. Handling each TI inside a savepoint (`session.begin_nested()`) may be needed so that one TI's failure does not roll back the others. - Which fallback state to use when `handle_failure()` itself cannot complete. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
