Can you please create a Jira and add (debug) logs at the moment this happens, including Celery logs please. I am trying to track this down. It seems only to happen with a CeleryExecutor.
Thanks Bolke > Op 1 sep. 2016, om 14:09 heeft Charalampos Paravalos <[email protected]> het > volgende geschreven: > > Hi, > > I am writting to ask for advise in an issue that I have with airflow and > til now I have not managed to resolve. Wondering if someone else had > something similar in the past. > > So, we use airflow to schedule DAGs that will run some jobs periodically > (every 30min/1hr). Jobs run as normal etc., but there are some times that > suddenly after DAGs are finished, the next scheduled jobs do not start at > all. It seems like the server does not kick off the scheduled jobs at all, > for any of the DAGs defined (so no jobs are running on our server). When > that happens I have to restart the scheduler so jobs are kicked on > automatically after restart. And the jobs run until this issue appears > again (I noticed it happening every 1 or 2 days, it is quite often). > > This is very strange, tried to upgrade to 1.7.1.3 version but still that > issue is here. We use 32 concurrent jobs with celery workers, the server is > able to manage the load well. > > I believe it has to do with the scheduler, but can't understand why. > Backfilled jobs maybe? Can this be? > > I am looking forward to hearing back from someone that has any ideas. > Please let me know what information you might need about my setup anytime. > > Thanks for your help! > > Regards, > Babis
