[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024541#comment-17024541
]
Ignacio Peluffo edited comment on AIRFLOW-5071 at 1/27/20 6:15 PM:
-------------------------------------------------------------------
Another one here, below is a capture of Flower showing that a task was
scheduled multiple times (the capture shows only three but there were 21
schedules):
!image-2020-01-27-18-10-29-124.png!
Then, in the database's table `task_instance`, the task row looks like:
{code:java}
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
| task_id | dag_id | execution_date |
start_date | end_date | duration | state | try_number |
hostname | unixname | job_id | pool | queue | priority_weight |
operator | queued_dttm | pid | max_tries | executor_config |
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
| clients | xxx | 2020-01-27 13:40:03.324162 | NULL | 2020-01-27
14:56:31.304658 | NULL | failed | 0 | | airflow | NULL
| default_pool | default | 1 | NULL | 2020-01-27
13:53:36.976145 | NULL | 0 | E'\\x80047D942E' |
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
{code}
I hope this help to find the cause and solution to the problem.
Also, any recommendation about which setting could be changed to fix this issue
would be appreciate it.
was (Author: ipeluffo):
Another one here, below is a capture of Flower showing that a task was
scheduled multiple times (the capture shows only three but there were 21
schedules):
!image-2020-01-27-18-10-29-124.png!
Then, in the database's table `task_instance`, the tasks row looks like:
{code:java}
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
| task_id | dag_id | execution_date |
start_date | end_date | duration | state | try_number |
hostname | unixname | job_id | pool | queue | priority_weight |
operator | queued_dttm | pid | max_tries | executor_config |
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
| clients | xxx | 2020-01-27 13:40:03.324162 | NULL | 2020-01-27
14:56:31.304658 | NULL | failed | 0 | | airflow | NULL
| default_pool | default | 1 | NULL | 2020-01-27
13:53:36.976145 | NULL | 0 | E'\\x80047D942E' |
+----------------------+-----------------+----------------------------+------------+----------------------------+----------+--------+------------+----------+----------+--------+--------------+---------+-----------------+----------+----------------------------+------+-----------+------------------+
{code}
I hope this help to find the cause and solution to the problem.
Also, any recommendation about which setting could be changed to fix this issue
would be appreciate it.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
> Affects Versions: 1.10.3
> Reporter: msempere
> Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance <TaskInstance: X
> 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance <TaskInstance: X
> 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
