Hello! I just tried to install the release candidate alpha 4. I was previously on airflow 1.7.0, and am not able to upgrade my metadata database on 1.8.0a4. I've detailed everything in a JIRA ticket at https://issues.apache.org/jira/browse/AIRFLOW-748. Let me know if I'm doing the install wrong, as I'd like to contribute to testing the new release and also get to use it soon :)
On Wed, Jan 11, 2017 at 4:47 PM, Dan Davydov <dan.davy...@airbnb.com.invalid > wrote: > The task dependency engine code is well commented, but I can provide a high > level overview specifically for developers if there is interest (note that > this would be the first documentation of it's kind in that it would be > developer-only documentation). The disadvantage is that it would create > duplication with the logic itself on quite a large scale. Let me know > Bolke. > > On Wed, Jan 11, 2017 at 1:30 PM, Chris Riccomini <criccom...@apache.org> > wrote: > > > @bolke, this sounds like a good list. > > > > On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin <bdbr...@gmail.com> > > wrote: > > > > > Ok. > > > > > > For now to call it “beta” 4 items seems to be left: > > > > > > Blocker: > > > * retry_delay not respected > > > * poison pill due to re-queue before process has finished (to be > > > investigated) > > > > > > Features: > > > * cgroups + impersonation > > > * dag.catchup (Ben Tallman -> Only documentation is missing). > > > > > > PRs that contain documentation would really be appreciated. In my > opinion > > > we are lacking there. Think about docs covering: > > > * new scheduler behaviour and options > > > * task dependency engine > > > * api / kerberized api > > > * … > > > > > > Cheers > > > Bolke > > > > > > > On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wied...@gmail.com> > > > wrote: > > > > > > > > +1 > > > > > > > > We can always think about different ways of doing this later (fair > > share > > > > scheduling etc...) > > > > > > > > Best, > > > > Arthur > > > > > > > > On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbr...@gmail.com> > > > wrote: > > > > > > > >> Dear All, > > > >> > > > >> I would like to drop "Schedule all pending DAG runs in a single > > > scheduler > > > >> loop” from the 1.8.0 release (updated: https://github.com/apache/ > > > >> incubator-airflow/pull/1980 <https://github.com/apache/ > > > >> incubator-airflow/pull/1980>, original: https://github.com/apache/ > > > >> incubator-airflow/pull/1906 <https://github.com/apache/ > > > >> incubator-airflow/pull/1906>). The reason for this is that it, imho, > > > >> biases the scheduler towards a single DAG as it fills the queue with > > > tasks > > > >> from one DAG and then goes to the next DAG. Starving DAGs that come > > > after > > > >> the first for resources. As such it should be updated and that will > > take > > > >> time. > > > >> > > > >> Please let me know if I am incorrect. > > > >> > > > >> Thanks > > > >> Bolke > > > >> > > > >>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > >>> > > > >>> Dear All, > > > >>> > > > >>> I have made Airflow 1.8.0 alpha 4 available at > > > >> https://people.apache.org/~bolke/ <https://people.apache.org/~ > bolke/> > > . > > > >> Again no Apache release yet - this is for testing purposes. I > consider > > > this > > > >> Alpha to be a Beta if not for the pending features. If the pending > > > features > > > >> are merged within a reasonable time frame (except for **, as no > > progress > > > >> currently) then I am planning to mark the tarball as Beta and only > > allow > > > >> bug fixes and (very) minor features. This week hopefully. > > > >>> > > > >>> Blockers: > > > >>> > > > >>> * None > > > >>> > > > >>> Fixed issues > > > >>> * Regression in email > > > >>> * LDAP case sensitivity > > > >>> * one_failed task not being run: now seems to pass suddenly (so > > fixed?) > > > >> -> need to investigate why > > > >>> * Email attachments > > > >>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) > > > >>> * Improve time units for task performance charts > > > >>> * XCom throws an duplicate / locking error > > > >>> * Add execution_date to trigger_dag > > > >>> > > > >>> Pending features: > > > >>> * DAG.catchup : minor changes needed, documentation still required, > > > >> integration tests seem to pass flawlessly > > > >>> * Cgroups + impersonation: clean up of patches on going, more tests > > and > > > >> more elaborate documentation required. Integration tests not > executed > > > yet > > > >>> * Schedule all pending DAG runs in a single scheduler loop: no > > progress > > > >> (**) > > > >>> > > > >>> Cheers! > > > >>> Bolke > > > >> > > > >> > > > > > > > > >
