I had similar issues with Airflow running the Celery executor. The celery_result_backend should be a persistent database like Postgres or MySql. What broker are you using? I would recommend using Redis or RabbitMQ, depending on what you like the most.
Cheers, Fokko 2018-05-15 21:12 GMT+02:00 David Capwell <dcapw...@gmail.com>: > What I find is that when celery rejects we hit this. For us we don't do > work on the hosts so solve by over provisioning tasks in celery > > On Tue, May 15, 2018, 6:30 AM Andy Cooper <andy.coo...@astronomer.io> > wrote: > >> I have had very similar issues when there was a problem with the >> connection >> string pointing to the message broker. Triple check those connection >> strings and attempt to connect outside of airflow. >> >> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pratapagout...@gmail.com >> > >> wrote: >> >> > Hi all, >> > >> > I have been using airflow with Celery executor in the background >> > >> > https://hastebin.com/sipecovomi.ini --> airflow.cfg >> > >> > https://hastebin.com/urutokuvoq.py --> The dag I have been using >> > >> > >> > >> > This shows that the dag is always in running state. >> > >> > >> > >> > >> > Airflow flower shows nothing in the tasks or in the broker. >> > >> > >> > Did I miss anything can anyone help me in this regard. >> > >> > >> > -- >> > Cheers !!! >> > Goutham Pratapa >> > >> >
