+1 We can always think about different ways of doing this later (fair share scheduling etc...)
Best, Arthur On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Dear All, > > I would like to drop "Schedule all pending DAG runs in a single scheduler > loop” from the 1.8.0 release (updated: https://github.com/apache/ > incubator-airflow/pull/1980 <https://github.com/apache/ > incubator-airflow/pull/1980>, original: https://github.com/apache/ > incubator-airflow/pull/1906 <https://github.com/apache/ > incubator-airflow/pull/1906>). The reason for this is that it, imho, > biases the scheduler towards a single DAG as it fills the queue with tasks > from one DAG and then goes to the next DAG. Starving DAGs that come after > the first for resources. As such it should be updated and that will take > time. > > Please let me know if I am incorrect. > > Thanks > Bolke > > > On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > > Dear All, > > > > I have made Airflow 1.8.0 alpha 4 available at > https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> . > Again no Apache release yet - this is for testing purposes. I consider this > Alpha to be a Beta if not for the pending features. If the pending features > are merged within a reasonable time frame (except for **, as no progress > currently) then I am planning to mark the tarball as Beta and only allow > bug fixes and (very) minor features. This week hopefully. > > > > Blockers: > > > > * None > > > > Fixed issues > > * Regression in email > > * LDAP case sensitivity > > * one_failed task not being run: now seems to pass suddenly (so fixed?) > -> need to investigate why > > * Email attachments > > * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) > > * Improve time units for task performance charts > > * XCom throws an duplicate / locking error > > * Add execution_date to trigger_dag > > > > Pending features: > > * DAG.catchup : minor changes needed, documentation still required, > integration tests seem to pass flawlessly > > * Cgroups + impersonation: clean up of patches on going, more tests and > more elaborate documentation required. Integration tests not executed yet > > * Schedule all pending DAG runs in a single scheduler loop: no progress > (**) > > > > Cheers! > > Bolke > >