Hey Dan, The engine is, but its settings are not. For example someone upgrading might be bitten by the location of the process_manager’s logs (I was!) - which are not particularly In a standard location by default. So an update to “UPDATING” would be appreciated and an update to the generic scheduler documentation as well.
This obviously doesn’t need to be you to write this up, my invitation was addressed to the general community. Cheers Bolke > On 11 Jan 2017, at 22:47, Dan Davydov <dan.davy...@airbnb.com.INVALID> wrote: > > The task dependency engine code is well commented, but I can provide a high > level overview specifically for developers if there is interest (note that > this would be the first documentation of it's kind in that it would be > developer-only documentation). The disadvantage is that it would create > duplication with the logic itself on quite a large scale. Let me know Bolke. > > On Wed, Jan 11, 2017 at 1:30 PM, Chris Riccomini <criccom...@apache.org> > wrote: > >> @bolke, this sounds like a good list. >> >> On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >> >>> Ok. >>> >>> For now to call it “beta” 4 items seems to be left: >>> >>> Blocker: >>> * retry_delay not respected >>> * poison pill due to re-queue before process has finished (to be >>> investigated) >>> >>> Features: >>> * cgroups + impersonation >>> * dag.catchup (Ben Tallman -> Only documentation is missing). >>> >>> PRs that contain documentation would really be appreciated. In my opinion >>> we are lacking there. Think about docs covering: >>> * new scheduler behaviour and options >>> * task dependency engine >>> * api / kerberized api >>> * … >>> >>> Cheers >>> Bolke >>> >>>> On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wied...@gmail.com> >>> wrote: >>>> >>>> +1 >>>> >>>> We can always think about different ways of doing this later (fair >> share >>>> scheduling etc...) >>>> >>>> Best, >>>> Arthur >>>> >>>> On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbr...@gmail.com> >>> wrote: >>>> >>>>> Dear All, >>>>> >>>>> I would like to drop "Schedule all pending DAG runs in a single >>> scheduler >>>>> loop” from the 1.8.0 release (updated: https://github.com/apache/ >>>>> incubator-airflow/pull/1980 <https://github.com/apache/ >>>>> incubator-airflow/pull/1980>, original: https://github.com/apache/ >>>>> incubator-airflow/pull/1906 <https://github.com/apache/ >>>>> incubator-airflow/pull/1906>). The reason for this is that it, imho, >>>>> biases the scheduler towards a single DAG as it fills the queue with >>> tasks >>>>> from one DAG and then goes to the next DAG. Starving DAGs that come >>> after >>>>> the first for resources. As such it should be updated and that will >> take >>>>> time. >>>>> >>>>> Please let me know if I am incorrect. >>>>> >>>>> Thanks >>>>> Bolke >>>>> >>>>>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> wrote: >>>>>> >>>>>> Dear All, >>>>>> >>>>>> I have made Airflow 1.8.0 alpha 4 available at >>>>> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> >> . >>>>> Again no Apache release yet - this is for testing purposes. I consider >>> this >>>>> Alpha to be a Beta if not for the pending features. If the pending >>> features >>>>> are merged within a reasonable time frame (except for **, as no >> progress >>>>> currently) then I am planning to mark the tarball as Beta and only >> allow >>>>> bug fixes and (very) minor features. This week hopefully. >>>>>> >>>>>> Blockers: >>>>>> >>>>>> * None >>>>>> >>>>>> Fixed issues >>>>>> * Regression in email >>>>>> * LDAP case sensitivity >>>>>> * one_failed task not being run: now seems to pass suddenly (so >> fixed?) >>>>> -> need to investigate why >>>>>> * Email attachments >>>>>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) >>>>>> * Improve time units for task performance charts >>>>>> * XCom throws an duplicate / locking error >>>>>> * Add execution_date to trigger_dag >>>>>> >>>>>> Pending features: >>>>>> * DAG.catchup : minor changes needed, documentation still required, >>>>> integration tests seem to pass flawlessly >>>>>> * Cgroups + impersonation: clean up of patches on going, more tests >> and >>>>> more elaborate documentation required. Integration tests not executed >>> yet >>>>>> * Schedule all pending DAG runs in a single scheduler loop: no >> progress >>>>> (**) >>>>>> >>>>>> Cheers! >>>>>> Bolke >>>>> >>>>> >>> >>> >>
