Hi everyone, I'm using Airflow version 1.8.0, just upgraded from 1.7.1.3. The issue that I'm going to describe started already in 1.7.1.3, I upgraded hoping it might help resolve it.
I have several DAGs for which the *last* task is not moving from queued to running. These DAGs used to run fine some time ago, but then we had issues with rabbitmq cluster we use, and after resetting it up, the problem emerged. I'm pretty sure the queue is working fine, since all the tasks except the very last one are queued automatically and run fine. For the sake of testing, I added a copy of the last task to the DAG, and interestingly, the task that used to be the last and did not run, now started to run normally, but the new last task is stuck. I checked logs at the DEBUG level and I could see that scheduler queues the tasks, but those tasks don't show up in the Celery/Flower dashboard in the corresponding queue. When I run the task that is stuck from the webserver interface, they show up in the queue in Flower dashboard and run successfully. So, overall, it seems that the issue is present with the scheduler but not with webserver, and that this issue is only related to the very last task in the DAG. I'm really stuck now, I would welcome any suggestions / ideas on what can be done. Thank you in advance! BR, Dima -- Dmitry Smirnov (MSc.) Data Engineer @ Yousician mobile: +358 50 3015072
