Hi Dmitry, Please provide more information, such as logs and the DAG definition itself. This is very little to go on unfortunately.
Bolke > On 3 May 2017, at 10:22, Dmitry Smirnov <[email protected]> wrote: > > Hi everyone, > > I'm using Airflow version 1.8.0, just upgraded from 1.7.1.3. The issue that > I'm going to describe started already in 1.7.1.3, I upgraded hoping it > might help resolve it. > > I have several DAGs for which the *last* task is not moving from queued to > running. > These DAGs used to run fine some time ago, but then we had issues with > rabbitmq cluster we use, and after resetting it up, the problem emerged. > I'm pretty sure the queue is working fine, since all the tasks except the > very last one are queued automatically and run fine. > For the sake of testing, I added a copy of the last task to the DAG, and > interestingly, the task that used to be the last and did not run, now > started to run normally, but the new last task is stuck. > I checked logs at the DEBUG level and I could see that scheduler queues the > tasks, but those tasks don't show up in the Celery/Flower dashboard in the > corresponding queue. > When I run the task that is stuck from the webserver interface, they show > up in the queue in Flower dashboard and run successfully. > So, overall, it seems that the issue is present with the scheduler but not > with webserver, and that this issue is only related to the very last task > in the DAG. > I'm really stuck now, I would welcome any suggestions / ideas on what can > be done. > > Thank you in advance! > BR, Dima > > -- > > Dmitry Smirnov (MSc.) > Data Engineer @ Yousician > mobile: +358 50 3015072
