Hi, I am writting to ask for advise in an issue that I have with airflow and til now I have not managed to resolve. Wondering if someone else had something similar in the past.
So, we use airflow to schedule DAGs that will run some jobs periodically (every 30min/1hr). Jobs run as normal etc., but there are some times that suddenly after DAGs are finished, the next scheduled jobs do not start at all. It seems like the server does not kick off the scheduled jobs at all, for any of the DAGs defined (so no jobs are running on our server). When that happens I have to restart the scheduler so jobs are kicked on automatically after restart. And the jobs run until this issue appears again (I noticed it happening every 1 or 2 days, it is quite often). This is very strange, tried to upgrade to 1.7.1.3 version but still that issue is here. We use 32 concurrent jobs with celery workers, the server is able to manage the load well. I believe it has to do with the scheduler, but can't understand why. Backfilled jobs maybe? Can this be? I am looking forward to hearing back from someone that has any ideas. Please let me know what information you might need about my setup anytime. Thanks for your help! Regards, Babis
