I'm not convinced that this would add *that* much more load, we could
probably change this functionality now if we wanted to. Just my two cents.
On Thu, Mar 16, 2017 at 4:06 PM, Rui Wang wrote:
> Thanks all your comments!
>
> Then looks like we should focus on scalability of scheduler now rather
Thanks all your comments!
Then looks like we should focus on scalability of scheduler now rather than
adding more load on it. I will give up this centralized idea now.
On Tue, Mar 14, 2017 at 3:08 PM, Rui Wang wrote:
> Hi,
> The design doc below I created is trying to make airflow scheduler mor
A few related thoughts about the scheduler.
The scheduler is growing to take on much more than just scheduling, so much
so that "supervisor" would be a better name for it. It includes:
* parsing DAGs (eventually it may serialize their metadata to the database
to help make the web server stateless)
Hi Rui,
We have been discussing this during the hackathon at Airbnb as well. Besides
the reservations Gerard is documenting, I am also not enthusiastic about this
design. Currently, the scheduler is our main issue in scaling. Scheduler runs
will take longer and longer with more DAGs and more co
Hi Rui,
I worked a bit on the scheduler and added some of my comments below.
On Tue, Mar 14, 2017 at 11:08 PM, Rui Wang
wrote:
> Hi,
> The design doc below I created is trying to make airflow scheduler more
> centralized. Briefly speaking, I propose moving state change of
> TaskInstance to sch
Hi,
The design doc below I created is trying to make airflow scheduler more
centralized. Briefly speaking, I propose moving state change of
TaskInstance to scheduler. You can see the reasons for this change below.
Could you take a look and comment if you see anything does not make sense?
-Rui
-
