danielito-orteguita opened a new issue #21562:
URL: https://github.com/apache/airflow/issues/21562
### Description
There should exist a mechanism to easily tell airflow whether a given failed
task should be retried or not.
My ideal solution would be for the Python operator to have a parameter to
specify a list of non-retriable exceptions. If the task fails with any of
those, it should be marked as failed and not be retried.
Eg:
`
class MyCustomException(Exception):
pass
task = PythonOperator(
task_id = "my_task",
python_callable = my_function,
retries = 3,
retry_delay = datetime.timedelta(hours=24),
non_retriable_exceptions = [ TypeError, MyCustomException ]
)
`
### Use case/motivation
I have a DAG whose tasks may fail due to mostly two kinds of errors.
One of them is retriable, because it is caused by a daily quota enforced by
an external API I make calls to, and which resets after 24h. It thus makes
sense to retry the task with a 24h delay when this error is encountered,
because the quota will have been refreshed by then.
The other error is non-retriable and requires human intervention, as it
occurs because of "dirty" external data out of my control. However, if I
configure the DAG retries according to the first error, the DAG won't continue
its work until [24h x retries] later, which is far too long. This is especially
inconvenient because it prevents further task and DAG executions from starting,
because I can only have one concurrent task at a time (because of the limited
quota in the aforementioned API). If the tasks simply failed and not be
retried, the task could fail, and airflow could move on.
A possible workaround would be adding branching operators and deciding
whether to go on or not depending on what exception is thrown. However, this
would only conceal the problem. I want the task to actually fail (and the DAG
at large).
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
