Also, is this delay b/w adding of task in queue and beginning of task on the worker not included in SLA of the task? Or is the SLA period begins once the task actually starts on the worker? Also, if scheduler has to wait for a response from the worker for the final state of the task (success/failure), how can the scheduler loose track of the task?
FYI, I am comparing airflow with quartz, which has a mistrigger handling built in. Mistrigger in quartz means that the task was not started within a pre-configured interval beginning form the scheduled time of start. Isn't there something similar in airflow? Regards Shubham Gupta On Fri, Jul 20, 2018 at 1:42 PM Shubham Gupta <shubham180695...@gmail.com> wrote: > Hi Ruiqin Yang, > > Can you please elaborate on what is meant by "and the scheduler lost > track of it" in your second paragraph? When can this happen? Also, what > is the default state when the scheduler restarts? Is it not* None*? > > Thanks for your quick reply. > > Regards > Shubham Gupta > > > On Fri, Jul 20, 2018 at 1:04 AM Ruiqin Yang <yrql...@gmail.com> wrote: > >> Hi Shubham, >> >> Worker running actual airflow task will regularly heartbeat, which updates >> the task instance entry in the DB. Scheduler will kill task instance w/o >> heartbeat for a long time, called zombie tasks, and if the task has retry >> left it will try to reschedule it( given all trigger rules are satisfied). >> >> If workers have heavy load, the scheduler will still be able to schedule >> tasks( putting tasks into worker queue). And you will just wait for >> workers >> to pick up the tasks from the queue. If the tasks never get picked up and >> the scheduler lost track of it, their state will be reset to NONE when >> scheduler restarts, they are called orphan tasks. >> >> FYI, inside Airbnb, Alex Guziel( @saguziel <https://github.com/saguziel>) >> has a patch that will requeue tasks if they don't get picked up by workers >> for a long time and he has plan to open source it. >> >> Cheers, >> Kevin Y >> >> On Fri, Jul 20, 2018 at 12:40 AM Shubham Gupta < >> shubham180695...@gmail.com> >> wrote: >> >> > Hi, >> > >> > I would like to know what happens if a Celery worker running one of the >> > tasks crashes. Will the job be rescheduled? >> > >> > Also, if the scheduler is not able to schedule a task on time due to >> heavy >> > load on all workers, what will happen to the task? >> > >> > Regards >> > Shubham Gupta >> > >> >
