amoghrajesh opened a new pull request, #73027:
URL: https://github.com/apache/airflow/pull/73027
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
### Motivation
Today, when a task goes into "FAILED" it alone tells you nothing about why
it stopped and that's something a retry policy in the first place should
surface best. A few examples:
1. Auth error, policy said don't bother retrying.
Your task hits an API and gets a 403 because the key expired. The policy is
smart enough to know retrying won't help (it'll just fail 3 more times the same
way), so it fails immediately after try 1. Without a reason, you just see
"FAILED after 1 try" and think something's broken with retries. With a reason,
you see: "auth error, policy chose not to retry" now you know exactly what to
go fix (the API key), not the code.
2. Rate limit, retries all used up.
Your task keeps hitting a rate limit. The policy says "retry" each time, so
it retries 3 times, but the budget (retries=3) runs out. Without a reason, you
just see 3 red X's and no explanation. With a reason: "rate limit; retries
exhausted (3 of 3)" and you immediately know it's not a bug; it's just that the
external service was too slow to respond, and maybe you should bump retries or
add a delay.
3. Different failures on different tries.
Try 1 failed for one reason, try 2 for a totally different reason. Right now
all you see is a row of identical red icons with no way to tell if its the same
recurring problem or three unrelated ones. Reasons attached to each try let
someone debugging a flaky task actually see the story, instead of guessing.
The underlying idea: today, all that classification work the policy does
(LLM or exception-based) happens, gets logged once in task logs. Only the "it
retried" case kept the reason. This change makes sure the reason survives for
the FAILED case too, so a future screen can show a plain sentence like "Stopped
at try 2: auth error, no retry" instead of just a bare failure with no story
behind it.
### What
If you've set a `retry_policy` on a task, it can decide things like "this
looks like an auth error, dont bother retrying" or "retries are exhausted, this
was a rate limit," but today that explanation is only ever logged to a log line
and thrown away otherwise. It never reaches the database, so no API response or
UI screen(I am proposing we build it to provide a better UX to users and its
more "in the face") can ever show it, no matter how much we build on top later.
This PR is the first, necessary step toward fixing that: make sure the
reason actually gets saved whenever a task fails, not just when it retries.
Once its reliably in the database, a future PR can expose it through the API
and the UI, so a Dag author looking at a failed task instance can see a plain
reason instead of just "FAILED" with no explanation.
### Current behaviour
`retry_reason` is already written to the database, but only when a retry
policy chooses to retry. The two failure outcomes people most want explained
never save anything: a policy deciding FAIL outright, and a policy deciding
RETRY but the retry budget being exhausted. Both currently build a bare
`TaskState(state=FAILED)` with no reason, so the classification text is logged
and discarded.
- Policy says FAIL: `TaskState(state=FAILED)`, reason dropped.
- Policy says RETRY but the budget is exhausted: same, reason dropped.
- Policy says RETRY with budget remaining: `RetryTask(retry_reason=...)`,
already persisted (unchanged).
- No retry policy at all: unaffected either way.
### Proposed change
Thread `retry_reason` through both FAILED paths, end to end:
- `_handle_current_task_failed`'s FAIL branch and `_finalize_task_failure`'s
exhausted budget branch now attach a reason to the `TaskState` they build. When
the budget is what stopped it, the reason is combined with an explicit note,
e.g. `"<reason>; retries exhausted (3 of 3)"`.
- `TaskState` (task-sdk message) gains a `retry_reason` field.
- FAILED does not go through the same "send immediately" path as RETRY; it's
deferred until the subprocess exits (the safety net that also covers a hard
crash with no message at all). So the supervisor now captures `retry_reason`
off the `TaskState` message and forwards it to the deferred `finish()` call.
- `TITerminalStatePayload` (execution API request schema) gains
`retry_reason`, gated behind a new Cadwyn version change added to the existing
unreleased `2026-10-30` version (tentative date)
- The FAILED branch of the state update route now truncates to 500 chars
(matching the existing RETRY branch) and persists the reason to
`task_instance.retry_reason`.
Only tasks with a configured retry policy are affected; a plain `retries=N`
task with no policy gets no reason and no behaviour change, since there's
nothing meaningful to attach.
### Testing
After running the `example_retry_policy` dag, earlier if I ran:
```sql
SELECT task_instance.state, task_instance.retry_reason from task_instance;
```
I would get:
```
failed,
failed,
up_for_retry,"rate_limit: The error message explicitly indicates a 429 HTTP
status code (""Too Many Requests"") with a rate limit exceeded message. The
error also provides guidance to retry after 60 seconds. Despite this being
attempt 3 of 3, rate limit errors are transient API throttling issues that
should be retried with appropriate backoff."
```
Now we see:
```
failed,auth: The error indicates an authentication failure due to an expired
API key for a service account. This is a credentials issue that requires manual
intervention to refresh or rotate the API key. Retrying will not resolve this
problem without updating the credentials.
up_for_retry,"rate_limit: The error message explicitly indicates ""429 Too
Many Requests"" and ""Rate limit exceeded"", which is a clear signal of API
throttling. The error also provides guidance to retry after 60 seconds."
failed,"data: This is a schema validation error where the input data does
not match the expected schema. The column 'user_id' was expected to be of type
INT but a STRING value was provided in row 42. This is a data quality issue
with the input data itself, not a transient infrastructure problem. Retrying
will not resolve this - the underlying data needs to be corrected or the schema
needs to be adjusted."
```
### What's next
`retry_reason` still isn't exposed anywhere outside the database. Adding it
to `TaskInstanceResponse` (API) and rendering it in the Task Instance UI panel
are the next two steps, so this data can actually reach a Dag author or surface
it on the UI.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
