Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1 and we didn't seem to have issues.
On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbr...@gmail.com> wrote: > While I agree with the assessment of Sid that a lot has changed and we do > not officially test on multiple schedulers, many changes were in the area > of proper locking which benefit multiple schedulers. In addition the tasks > themselves have built in checks that they don’t run twice at the same time. > > Yet YMMV. > > Bolke > > > On 24 Feb 2017, at 03:13, siddharth anand <san...@apache.org> wrote: > > > > I did run 2 or more schedulers with Local Executors up until mid last > > year. There have been enough changes to the code and feature additions > that > > I don't think this is a recommended practice at this point. Also, there > is > > not a lot of synchronization in the scheduler to ensure this will work. > > > > -s > > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <matusv...@gmail.com> wrote: > > > >> Hi all, > >> > >> > >> > >> I am considering deployment of airflow as pipeline framework. I have > found > >> out multiple articles explaining deployment of airflow in distributed > >> environment (e.g. [1]). Unfortunately, I was not able to find out any > use > >> case where scheduler is deployed distributed on multiple nodes. Is it > >> possible to have scheduler distributed on multiple nodes to prevent > single > >> point of failure? I haven’t found any mention about it in > documentation. I > >> have found out in [2] that it is not possible but on the other hand in > [3] > >> is reference that this can be solved in new version of airflow. > >> > >> > >> > >> Thanks, > >> > >> > >> Matus > >> > >> > >> > >> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/ > >> > >> [2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME > >> > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678 > >> > >
