[ https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501136#comment-17501136 ]
ASF GitHub Bot commented on AIRFLOW-5071: ----------------------------------------- ghostbody commented on issue #10790: URL: https://github.com/apache/airflow/issues/10790#issuecomment-1058801920 After STRUGLING, We found a method to 100% reproduce this issue !!! tl;dr https://github.com/apache/airflow/blob/9ac742885ffb83c15f7e3dc910b0cf9df073407a/airflow/models/taskinstance.py#L1253 Add a `raise` to simulate db error which will likely happen when the DB is under great pressure. Then you will get this issue `Was the task killed externally` in all the time. Conditions: - Airflow 2.2 - Celery Executor It's becasue the worker use a local task job which will spwan a child process to execute the job. The parent process set the task from `Queued` to `Running` State. However, when the prepare work for the parent process failed, it will lead to this error directly. related code is here: https://github.com/apache/airflow/blob/2.2.2/airflow/jobs/local_task_job.py#L89 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Thousand os Executor reports task instance X finished (success) although the > task says its queued. Was the task killed externally? > ---------------------------------------------------------------------------------------------------------------------------------- > > Key: AIRFLOW-5071 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5071 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler > Affects Versions: 1.10.3 > Reporter: msempere > Priority: Critical > Fix For: 1.10.12 > > Attachments: image-2020-01-27-18-10-29-124.png, > image-2020-07-08-07-58-42-972.png > > > I'm opening this issue because since I update to 1.10.3 I'm seeing thousands > of daily messages like the following in the logs: > > ``` > {{__init__.py:1580}} ERROR - Executor reports task instance <TaskInstance: X > 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says > its queued. Was the task killed externally? > {{jobs.py:1484}} ERROR - Executor reports task instance <TaskInstance: X > 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says > its queued. Was the task killed externally? > ``` > -And looks like this is triggering also thousand of daily emails because the > flag to send email in case of failure is set to True.- > I have Airflow setup to use Celery and Redis as a backend queue service. -- This message was sent by Atlassian Jira (v8.20.1#820001)