We too face the same issue with scheduler (Celery executor). I opened a JIRA https://issues.apache.org/jira/browse/AIRFLOW-401 for this. You could try to verify it against master as suggested by Bolke in my JIRA.
The workaround that works for us is to restart the scheduler every X mins through an external script. This keeps the scheduler going hence launching tasks without fail. Thanks, Nadeem On Thu, Sep 1, 2016 at 11:45 AM, <[email protected]> wrote: > > > We face exactly the same issue... > I tried to describe it here this week, > But no one had a solution. > > ב-1 בספט׳ 2016, בשעה 17:54, Sergei Iakhnin <[email protected]> כתב/ה: > > > As far as I know even Airbnb themselves restart their schedulers every 30 > > minutes because of this issue. I ended up doing it as well with a cron > job > > after giving up hope that it would be fixed in the short term. > > > >> On Thu, 1 Sep 2016, 16:03 Charalampos Paravalos, <[email protected]> wrote: > >> > >> Hi, > >> > >> I am writting to ask for advise in an issue that I have with airflow and > >> til now I have not managed to resolve. Wondering if someone else had > >> something similar in the past. > >> > >> So, we use airflow to schedule DAGs that will run some jobs periodically > >> (every 30min/1hr). Jobs run as normal etc., but there are some times > that > >> suddenly after DAGs are finished, the next scheduled jobs do not start > at > >> all. It seems like the server does not kick off the scheduled jobs at > all, > >> for any of the DAGs defined (so no jobs are running on our server). When > >> that happens I have to restart the scheduler so jobs are kicked on > >> automatically after restart. And the jobs run until this issue appears > >> again (I noticed it happening every 1 or 2 days, it is quite often). > >> > >> This is very strange, tried to upgrade to 1.7.1.3 version but still that > >> issue is here. We use 32 concurrent jobs with celery workers, the > server is > >> able to manage the load well. > >> > >> I believe it has to do with the scheduler, but can't understand why. > >> Backfilled jobs maybe? Can this be? > >> > >> I am looking forward to hearing back from someone that has any ideas. > >> Please let me know what information you might need about my setup > anytime. > >> > >> Thanks for your help! > >> > >> Regards, > >> Babis > > -- > > > > Sergei >
