One drawback is the lack of proper cross DAG prioritization. Note that in >= 1.8:
* the scheduler is a multi process queue, in terms of fire power it should not be the bottleneck for anyone at this point. Effectively you could have 64+ scheduling threads on a single large machine. This should make the db the new bottleneck, but serve the needs of any large company doing something reasonable (from my guts I'd say firing thousands of tasks per minute should work smoothly) * each process in the queue evaluates a single DAG and reports runnable tasks to the main scheduler process which accumulates, prioritizes and puts messages in the queue on a specified interval. If you have DAGs that are slow to process for some reason, they'll miss the train get on board of the next one. * where double-triggers are normally an edge case, they become the norm when running multiple schedulers * in theory, double triggers should be handled properly, and fire once and only once. But we've seen occurrences of double-firing in 1.8.0, which we're fixing now, but minimizing double triggers helps lowering the risk of double-firing. * note that tasks may be indempotent from a sequential perspective but double-firing in parallel may have bad side-effects * the existing protections and light guarantees around double-firing, along with luck around "parallely idempotent" tasks is what mitigated double-triggers and made running multiple schedulers work historically For Airflow to have a "features distributed scheduling" stamp on it, we'd have to somehow prevent most double-triggering when running multiple schedulers. This could be achieved by the individual schedulers somehow taking locks, perhaps at the DAG level (impacts cross DAG prioritization in pools negatively), or by taking turns on running scheduling cycles somehow (would enable resiliency only, not distributing the workload). Ideally, I think we'd want to accumulate runnable tasks in a message queue, and schedulers would only take turns on the process that read from that queue, prioritizes and send messages to run tasks on workers. Max On Fri, Feb 24, 2017 at 11:51 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Can I assume your db is also not a SPOF? > > It seems a waste of processor time to me to have it every node. > > B. > > Sent from my iPhone > > > On 24 Feb 2017, at 20:39, Jason Chen <chingchien.c...@gmail.com> wrote: > > > > Thanks, Arthur. > > > > I think it avoids "single point of failure" (and maybe balance the > > scheduling load between nodes ?) > > Given a celery cluster, if the only node running scheduler is down, the > > whole cluster will fail to schedule jobs. > > Any downside why not having multiple schedulers ? > > > > -Jason > > > > > > On Fri, Feb 24, 2017 at 11:35 AM, Arthur Wiedmer < > arthur.wied...@gmail.com> > > wrote: > > > >> Jason, > >> > >> Why do you need a scheduler running on each node? > >> > >> We have a single scheduler powering work on many nodes each running a > >> celery worker via "airflow worker". We have one large metadata MySQL > >> instance. > >> > >> Best, > >> Arthur > >> > >> On Fri, Feb 24, 2017 at 11:04 AM, Jason Chen <chingchien.c...@gmail.com > > > >> wrote: > >> > >>> A side question related to this topic: > >>> I am running Airflow w/ celery executor in multiple nodes. Each node is > >>> running celery, worker, scheduler and webserver. > >>> These nodes are registered to a Redis for celery queue and these nodes > >> are > >>> sharing the same dags, logs folder (and MySQL) > >>> It seems running fine. > >>> Any concerns or suggestions ? > >>> I am thinking celery executor is designed for distributed env. > >>> > >>> Thanks. > >>> > >>> -Jason > >>> > >>> > >>> On Fri, Feb 24, 2017 at 10:58 AM, Jason Jho <jason....@blueapron.com. > >>> invalid > >>>> wrote: > >>> > >>>> Seems like this would inherently tied to the VM it's running on. > Either > >>>> way, would love to hear about any experiences as well! > >>>> On Fri, Feb 24, 2017 at 1:52 PM Wilson Lian <wwl...@google.com.invalid > >>> > >>>> wrote: > >>>> > >>>>> Out of curiosity, has anyone heard any war stories re: reaching the > >>>> limits > >>>>> of a single scheduler in terms of the number of > >> potentially-schedulable > >>>>> DAGs? > >>>>> > >>>>> On Fri, Feb 24, 2017 at 10:25 AM, Dan Davydov < > >>>>> dan.davy...@airbnb.com.invalid> wrote: > >>>>> > >>>>>> We just had two running by accident for some period of time. > >>>>>> > >>>>>> On Feb 24, 2017 5:52 AM, "Jason Jho" <jason....@blueapron.com. > >>> invalid> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Dan / Sid, > >>>>>>> > >>>>>>> Would you be able to elaborate on the multiple scheduler setup? > >>>> Curious > >>>>>> how > >>>>>>> that would have been deployed. Was the purpose to have some kind > >> of > >>>>>>> failover or to distribute execution of jobs? > >>>>>>> > >>>>>>> Thanks! > >>>>>>> On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov < > >>> dan.davy...@airbnb.com. > >>>>>>> invalid> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Fwiw Airbnb was running multiple schedulers for a short while > >> on > >>>>> 1.7.1 > >>>>>>> and > >>>>>>>> we didn't seem to have issues. > >>>>>>>> > >>>>>>>> On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbr...@gmail.com> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> While I agree with the assessment of Sid that a lot has > >> changed > >>>> and > >>>>>> we > >>>>>>> do > >>>>>>>>> not officially test on multiple schedulers, many changes were > >>> in > >>>>> the > >>>>>>> area > >>>>>>>>> of proper locking which benefit multiple schedulers. In > >>> addition > >>>>> the > >>>>>>>> tasks > >>>>>>>>> themselves have built in checks that they don’t run twice at > >>> the > >>>>> same > >>>>>>>> time. > >>>>>>>>> > >>>>>>>>> Yet YMMV. > >>>>>>>>> > >>>>>>>>> Bolke > >>>>>>>>> > >>>>>>>>>> On 24 Feb 2017, at 03:13, siddharth anand < > >> san...@apache.org > >>>> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> I did run 2 or more schedulers with Local Executors up > >> until > >>>> mid > >>>>>>> last > >>>>>>>>>> year. There have been enough changes to the code and > >> feature > >>>>>>> additions > >>>>>>>>> that > >>>>>>>>>> I don't think this is a recommended practice at this point. > >>>> Also, > >>>>>>> there > >>>>>>>>> is > >>>>>>>>>> not a lot of synchronization in the scheduler to ensure > >> this > >>>> will > >>>>>>> work. > >>>>>>>>>> > >>>>>>>>>> -s > >>>>>>>>>> > >>>>>>>>>> On Thu, Feb 9, 2017 at 6:47 AM, matus valo < > >>>> matusv...@gmail.com> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I am considering deployment of airflow as pipeline > >>> framework. > >>>> I > >>>>>> have > >>>>>>>>> found > >>>>>>>>>>> out multiple articles explaining deployment of airflow in > >>>>>>> distributed > >>>>>>>>>>> environment (e.g. [1]). Unfortunately, I was not able to > >>> find > >>>>> out > >>>>>>> any > >>>>>>>>> use > >>>>>>>>>>> case where scheduler is deployed distributed on multiple > >>>> nodes. > >>>>> Is > >>>>>>> it > >>>>>>>>>>> possible to have scheduler distributed on multiple nodes > >> to > >>>>>> prevent > >>>>>>>>> single > >>>>>>>>>>> point of failure? I haven’t found any mention about it in > >>>>>>>>> documentation. I > >>>>>>>>>>> have found out in [2] that it is not possible but on the > >>> other > >>>>>> hand > >>>>>>> in > >>>>>>>>> [3] > >>>>>>>>>>> is reference that this can be solved in new version of > >>>> airflow. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Matus > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> [1] http://site.clairvoyantsoft. > >> com/setting-apache-airflow- > >>>>>> cluster/ > >>>>>>>>>>> > >>>>>>>>>>> [2] > >>>>>>>> https://groups.google.com/forum/#!topic/airbnb_airflow/- > >>> 1wKa3OcwME > >>>>>>>>>>> > >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/AIRFLOW-678 > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >
