Dear All, I would like to drop "Schedule all pending DAG runs in a single scheduler loop” from the 1.8.0 release (updated: https://github.com/apache/incubator-airflow/pull/1980 <https://github.com/apache/incubator-airflow/pull/1980>, original: https://github.com/apache/incubator-airflow/pull/1906 <https://github.com/apache/incubator-airflow/pull/1906>). The reason for this is that it, imho, biases the scheduler towards a single DAG as it fills the queue with tasks from one DAG and then goes to the next DAG. Starving DAGs that come after the first for resources. As such it should be updated and that will take time.
Please let me know if I am incorrect. Thanks Bolke > On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Dear All, > > I have made Airflow 1.8.0 alpha 4 available at > https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> . Again > no Apache release yet - this is for testing purposes. I consider this Alpha > to be a Beta if not for the pending features. If the pending features are > merged within a reasonable time frame (except for **, as no progress > currently) then I am planning to mark the tarball as Beta and only allow bug > fixes and (very) minor features. This week hopefully. > > Blockers: > > * None > > Fixed issues > * Regression in email > * LDAP case sensitivity > * one_failed task not being run: now seems to pass suddenly (so fixed?) -> > need to investigate why > * Email attachments > * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) > * Improve time units for task performance charts > * XCom throws an duplicate / locking error > * Add execution_date to trigger_dag > > Pending features: > * DAG.catchup : minor changes needed, documentation still required, > integration tests seem to pass flawlessly > * Cgroups + impersonation: clean up of patches on going, more tests and more > elaborate documentation required. Integration tests not executed yet > * Schedule all pending DAG runs in a single scheduler loop: no progress (**) > > Cheers! > Bolke
