repl-chris opened a new pull request, #23690: URL: https://github.com/apache/airflow/pull/23690
<!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> Celery can lose tasks on worker shutdown, causing airflow to just wait on them indefinitely (may be related to https://github.com/celery/celery/issues/7266). This PR expands the "stalled tasks" functionality which is already in place for adopted tasks, and adds the ability to apply it to all tasks such that these lost/hung tasks can be automatically recovered and queued up again. Closes: https://github.com/apache/airflow/issues/19699 This is an alternate approach/implementation of https://github.com/apache/airflow/pull/23432. There are a couple advantages to this approach: * It is lighter weight - instead of db queries happening on a periodic schedule, it's a simple date comparison happening on each loop * The code is cleaner - it's not a whole new piece of functionality, rather it's a slight expansion of an existing capability * Disabled by default - unless the user explicitly sets the `stalled_task_timeout` config setting, there is minimal behavioural change. The only behavioural difference is: * in the existing code, stalled adopted tasks are not revoked from celery and they are marked as failed in airflow....which potentially results in: * the task being executed multiple times (if it wasn't actually stalled, just in a long celery queue) * failing with no error (if the task is set to `retries=0`) * in this implementation, stalled adopted tasks are revoked from celery and are re-set to `SCHEDULED` state so they'll get queued back up. cc: @ephraimbuddy @kristoffern @kaxil @XD-DENG @ashb --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
