Ok. For now to call it “beta” 4 items seems to be left:
Blocker: * retry_delay not respected * poison pill due to re-queue before process has finished (to be investigated) Features: * cgroups + impersonation * dag.catchup (Ben Tallman -> Only documentation is missing). PRs that contain documentation would really be appreciated. In my opinion we are lacking there. Think about docs covering: * new scheduler behaviour and options * task dependency engine * api / kerberized api * … Cheers Bolke > On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wied...@gmail.com> wrote: > > +1 > > We can always think about different ways of doing this later (fair share > scheduling etc...) > > Best, > Arthur > > On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Dear All, >> >> I would like to drop "Schedule all pending DAG runs in a single scheduler >> loop” from the 1.8.0 release (updated: https://github.com/apache/ >> incubator-airflow/pull/1980 <https://github.com/apache/ >> incubator-airflow/pull/1980>, original: https://github.com/apache/ >> incubator-airflow/pull/1906 <https://github.com/apache/ >> incubator-airflow/pull/1906>). The reason for this is that it, imho, >> biases the scheduler towards a single DAG as it fills the queue with tasks >> from one DAG and then goes to the next DAG. Starving DAGs that come after >> the first for resources. As such it should be updated and that will take >> time. >> >> Please let me know if I am incorrect. >> >> Thanks >> Bolke >> >>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> wrote: >>> >>> Dear All, >>> >>> I have made Airflow 1.8.0 alpha 4 available at >> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> . >> Again no Apache release yet - this is for testing purposes. I consider this >> Alpha to be a Beta if not for the pending features. If the pending features >> are merged within a reasonable time frame (except for **, as no progress >> currently) then I am planning to mark the tarball as Beta and only allow >> bug fixes and (very) minor features. This week hopefully. >>> >>> Blockers: >>> >>> * None >>> >>> Fixed issues >>> * Regression in email >>> * LDAP case sensitivity >>> * one_failed task not being run: now seems to pass suddenly (so fixed?) >> -> need to investigate why >>> * Email attachments >>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) >>> * Improve time units for task performance charts >>> * XCom throws an duplicate / locking error >>> * Add execution_date to trigger_dag >>> >>> Pending features: >>> * DAG.catchup : minor changes needed, documentation still required, >> integration tests seem to pass flawlessly >>> * Cgroups + impersonation: clean up of patches on going, more tests and >> more elaborate documentation required. Integration tests not executed yet >>> * Schedule all pending DAG runs in a single scheduler loop: no progress >> (**) >>> >>> Cheers! >>> Bolke >> >>
