stevenwoods commented on issue #9610:
URL: https://github.com/apache/airflow/issues/9610#issuecomment-810505695
I am not using kubernetes and I am using ` v2.0.1`, I get this message when
I have long running tasks (what seems like over 10 hours).
```
*** Reading remote log from s3://***/2021-03-18T19:00:00+00:00/1.log.
[2021-03-27 10:58:40,962] {taskinstance.py:845} INFO - Dependencies not met
for <TaskInstance: dump_restore_tables 2021-03-18T19:00:00+00:00 [running]>,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2021-03-27 10:58:40,962] {taskinstance.py:845} INFO - Dependencies not met
for <TaskInstance: dump_restore_tables 2021-03-18T19:00:00+00:00 [running]>,
dependency 'Task Instance Not Running' FAILED: Task is in the running state
[2021-03-27 10:58:40,963] {local_task_job.py:93} INFO - Task is not able to
be run
[2021-03-27 10:58:40,962] {taskinstance.py:845} INFO - Dependencies not met
for <TaskInstance: dump_restore_tables 2021-03-18T19:00:00+00:00 [running]>,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2021-03-27 10:58:40,962] {taskinstance.py:845} INFO - Dependencies not met
for <TaskInstance: dump_restore_tables 2021-03-18T19:00:00+00:00 [running]>,
dependency 'Task Instance Not Running' FAILED: Task is in the running state
[2021-03-27 10:58:40,963] {local_task_job.py:93} INFO - Task is not able to
be run
[2021-03-27 11:02:48,909] {bash.py:16} INFO - SET
```
Meanwhile I can tell on the instance and the database it is still running in
the background.
At 11:02 is when the task started logging.
Here is the end of the log file:
```
[2021-03-27 15:47:12,873] {bash.py:18} INFO - returncode: 0
[2021-03-27 15:47:12,953] {python.py:118} INFO - Done. Returned value was:
None
[2021-03-27 15:47:12,967] {taskinstance.py:1166} INFO - Marking task as
SUCCESS. dag_id=* task_id=dump_restore_tables, execution_date=20210318T190000,
start_date=20210327T085655, end_date=20210327T194712
[2021-03-27 15:47:13,486] {local_task_job.py:188} WARNING - State of this
instance has been externally set to success. Terminating instance.
[2021-03-27 15:47:13,500] {process_utils.py:100} INFO - Sending
Signals.SIGTERM to GPID 7599
[2021-03-27 15:47:13,500] {taskinstance.py:1239} ERROR - Received SIGTERM.
Terminating subprocesses.
[2021-03-27 15:47:13,712] {process_utils.py:66} INFO - Process
psutil.Process(pid=7599, status='terminated') (7599) terminated with exit code 1
```
This happens on every DAG run of tasks that are long running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
