1fanwang opened a new pull request, #66781: URL: https://github.com/apache/airflow/pull/66781
Closes #60172. `_run_task_state_change_callbacks` in `task-sdk/src/airflow/sdk/execution_time/task_runner.py` catches every exception from a callback and logs it. That's the right default for noisy cleanup work, but it also swallows the explicit `AirflowFailException` signal — a user raising it inside `on_retry_callback` to say "fail without retrying" had no way to actually fail the task. The state stayed `UP_FOR_RETRY` and another attempt was scheduled. The fix narrows the catch in the retry-callback path: - `AirflowFailException` re-raises so the caller can react. - Every other `Exception` is still logged + swallowed — Wei's concern on the prior attempt (PR #64198) about cleanup callbacks that may fail is preserved. `run()` now defers the supervisor `RetryTask` message until after the retry callback runs. `finalize()` gained an optional `msg` parameter: when the retry callback raises `AirflowFailException`, finalize promotes the state to `FAILED`, replaces the pending `RetryTask` with `TaskState(FAILED)`, and runs the failure-path finalizers (`on_failure_callback`, listener hook, `email_on_failure`). Two prior attempts (#60415 closed 2026-03-05, #64198 closed 2026-05-05) tried to solve this; this PR picks up the Wei-shaped design and ships it with a regression test. ## Tests `task-sdk/tests/task_sdk/execution_time/test_task_runner.py::TestTaskRunnerCallsCallbacks::test_airflow_fail_exception_in_on_retry_callback_fails_task` exercises the full path: retry callback raises `AirflowFailException`, the failure callback runs, the supervisor receives `TaskState(FAILED)` instead of `RetryTask`. The test fails on main and passes with this PR; existing callback tests in the file behave unchanged (generic `Exception` from a retry callback still swallowed + retry still scheduled). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
