> For example someone upgrading might be bitten by the location of the process_manager’s logs (I was!)
Same. Started getting disk near-full alerts because I had no idea it was dumping gigs logs into /tmp. On Thu, Jan 12, 2017 at 6:17 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Hey Dan, > > The engine is, but its settings are not. For example someone upgrading > might be bitten by the location of the process_manager’s logs (I was!) - > which are not particularly > In a standard location by default. So an update to “UPDATING” would be > appreciated and an update to the generic scheduler documentation as well. > > This obviously doesn’t need to be you to write this up, my invitation was > addressed to the general community. > > Cheers > Bolke > > > On 11 Jan 2017, at 22:47, Dan Davydov <dan.davy...@airbnb.com.INVALID> > wrote: > > > > The task dependency engine code is well commented, but I can provide a > high > > level overview specifically for developers if there is interest (note > that > > this would be the first documentation of it's kind in that it would be > > developer-only documentation). The disadvantage is that it would create > > duplication with the logic itself on quite a large scale. Let me know > Bolke. > > > > On Wed, Jan 11, 2017 at 1:30 PM, Chris Riccomini <criccom...@apache.org> > > wrote: > > > >> @bolke, this sounds like a good list. > >> > >> On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin <bdbr...@gmail.com> > >> wrote: > >> > >>> Ok. > >>> > >>> For now to call it “beta” 4 items seems to be left: > >>> > >>> Blocker: > >>> * retry_delay not respected > >>> * poison pill due to re-queue before process has finished (to be > >>> investigated) > >>> > >>> Features: > >>> * cgroups + impersonation > >>> * dag.catchup (Ben Tallman -> Only documentation is missing). > >>> > >>> PRs that contain documentation would really be appreciated. In my > opinion > >>> we are lacking there. Think about docs covering: > >>> * new scheduler behaviour and options > >>> * task dependency engine > >>> * api / kerberized api > >>> * … > >>> > >>> Cheers > >>> Bolke > >>> > >>>> On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wied...@gmail.com> > >>> wrote: > >>>> > >>>> +1 > >>>> > >>>> We can always think about different ways of doing this later (fair > >> share > >>>> scheduling etc...) > >>>> > >>>> Best, > >>>> Arthur > >>>> > >>>> On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbr...@gmail.com> > >>> wrote: > >>>> > >>>>> Dear All, > >>>>> > >>>>> I would like to drop "Schedule all pending DAG runs in a single > >>> scheduler > >>>>> loop” from the 1.8.0 release (updated: https://github.com/apache/ > >>>>> incubator-airflow/pull/1980 <https://github.com/apache/ > >>>>> incubator-airflow/pull/1980>, original: https://github.com/apache/ > >>>>> incubator-airflow/pull/1906 <https://github.com/apache/ > >>>>> incubator-airflow/pull/1906>). The reason for this is that it, imho, > >>>>> biases the scheduler towards a single DAG as it fills the queue with > >>> tasks > >>>>> from one DAG and then goes to the next DAG. Starving DAGs that come > >>> after > >>>>> the first for resources. As such it should be updated and that will > >> take > >>>>> time. > >>>>> > >>>>> Please let me know if I am incorrect. > >>>>> > >>>>> Thanks > >>>>> Bolke > >>>>> > >>>>>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> wrote: > >>>>>> > >>>>>> Dear All, > >>>>>> > >>>>>> I have made Airflow 1.8.0 alpha 4 available at > >>>>> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/ > > > >> . > >>>>> Again no Apache release yet - this is for testing purposes. I > consider > >>> this > >>>>> Alpha to be a Beta if not for the pending features. If the pending > >>> features > >>>>> are merged within a reasonable time frame (except for **, as no > >> progress > >>>>> currently) then I am planning to mark the tarball as Beta and only > >> allow > >>>>> bug fixes and (very) minor features. This week hopefully. > >>>>>> > >>>>>> Blockers: > >>>>>> > >>>>>> * None > >>>>>> > >>>>>> Fixed issues > >>>>>> * Regression in email > >>>>>> * LDAP case sensitivity > >>>>>> * one_failed task not being run: now seems to pass suddenly (so > >> fixed?) > >>>>> -> need to investigate why > >>>>>> * Email attachments > >>>>>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) > >>>>>> * Improve time units for task performance charts > >>>>>> * XCom throws an duplicate / locking error > >>>>>> * Add execution_date to trigger_dag > >>>>>> > >>>>>> Pending features: > >>>>>> * DAG.catchup : minor changes needed, documentation still required, > >>>>> integration tests seem to pass flawlessly > >>>>>> * Cgroups + impersonation: clean up of patches on going, more tests > >> and > >>>>> more elaborate documentation required. Integration tests not executed > >>> yet > >>>>>> * Schedule all pending DAG runs in a single scheduler loop: no > >> progress > >>>>> (**) > >>>>>> > >>>>>> Cheers! > >>>>>> Bolke > >>>>> > >>>>> > >>> > >>> > >> > >
