Re: Multiple Schedulers - "scheduler_lock"

2019-03-17 Thread Peter van t Hof
Hi, My proposal is focusing mainly on scalability and indeed not so much on HA. This mainly because that is also the main issue from the original author. Have a form of HA on this MainScheduler would still be nice to have. The problem with is that have a fixed number of scheduler does not scale

Re: Multiple Schedulers - "scheduler_lock"

2019-03-17 Thread Maxime Beauchemin
The proposal reads "Looking at the original AIP-15 the author proposes to use locking to enable the use of multiple schedulers, this might introduce unnecessary complexity" To me introducing multiple roles (master scheduler + scheduler minions), may be actually more complex than just having "share

Re: Mutli-layered official image for Airflow

2019-03-17 Thread Jarek Potiuk
Hello everyone, I believe I am ready now to involve more of the community people in the multi-layered Docker AIP-10 that I am working on for some time (with comments and encouragement from Ash and Fokko as explained in the AIP thread). Any comments, questions, critique, improvement proposals, or

Re: Suggestion for AIP improvement

2019-03-17 Thread Driesprong, Fokko
Thanks Bas for the feedback on the AIP process. I've would like to incorporate the suggested questions in the AIP template. In general, I think communication is key. As you already mentioned, a lot of communication is being done over a heterogeneous set of channels. For example, if you're reading

Re: Multiple Schedulers - "scheduler_lock"

2019-03-17 Thread Peter van t Hof
Hi all, I think that scheduling locking is maybe not the best way in solving this issue. Still I’m in support of taking a good look at the scheduler because it has some real scaling issues. I did wrote an alternative proposal to solve the scalability of the scheduler: https://cwiki.apache.org/c