Hey Bolke, Thanks for taking this on. I'm definitely up for running stuff in our environments to verify everything is working.
Can I ask that you create a 1.8 alpha 1 branch in the git repo? This will make it easier for us to track what changes are getting cherry picked into the branch, and will also make it easier for users to pip install, if they want to do so via github. Also, yea, when we switch to beta, we need to stop merging anything other than bug fixes into the release branch. Cheers, Chris On Tue, Jan 3, 2017 at 10:31 AM, Dan Davydov <dan.davy...@airbnb.com.invalid > wrote: > All very reasonable to me, one reason we may not have hit the bugs in our > production is because we are running off a different merge base and our > cherries aren't 1-1 with what we are running in production (we still test > them but we can't run them in production), that being said I don't think I > authored the commits you are referring to so I don't have full context. > > On Tue, Jan 3, 2017 at 1:27 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > Hi Dan et al, > > > > That sounds good to me, however I will be pretty critical of the changes > > in the scheduler and the cleanliness of the patches. This is due to the > > fact I have been chasing quite some bugs in master that were pretty hard > to > > track down even with a debugger at hand. I’m surprised that those didn’t > > pop up in your production or maybe I am concerned ;-). Anyways, I hope > you > > understand I might be a bit picky in understanding and needing (design) > > documentation for some of the changes. > > > > What I would like to suggest is that for the Alpha versions we still > > accept “new” features so these PRs can get in, but from Beta we will not > > accept new features anymore. For new features in the area of the > scheduler > > an integration DummyDag should be supplied, so others can test the > > behaviour. Does this sound ok? > > > > My list of open code items for a release looks now like this: > > > > Blockers > > * one_failed not honoured > > * Alex’s sensor issue > > > > New features: > > * Schedule all pending DAGs in a single loop > > * Add support for backfill true/false > > * Impersonation > > * CGroups > > * Add Cloud Storage updated sensor > > > > Alpha2 I will package tomorrow. Packages are signed now by my apache.org > < > > http://apache.org/> key. Please verify and let me know if something is > > off. I’m still waiting for access to the incubating dist repository. > > > > Bolke > > > > > > > On 3 Jan 2017, at 14:38, Dan Davydov <dan.davy...@airbnb.com.INVALID> > > wrote: > > > > > > I have also started on this effort, recently Alex Guziel and I have > been > > > pushing Airbnb's custom cherries onto master to get Airbnb back onto > > master > > > in order for us to do a release. > > > > > > I think it might make sense to wait for these two commits to get merged > > in > > > since they would be quite nice to have for all Airflow users and seem > > like > > > they will be merged soon: > > > Schedule all pending DAG runs in a single scheduler loop - > > > https://github.com/apache/incubator-airflow/pull/1906 < > > https://github.com/apache/incubator-airflow/pull/1906> > > > Add Support for dag.backfill=(True|False) Option - > > > https://github.com/apache/incubator-airflow/pull/1830 < > > https://github.com/apache/incubator-airflow/pull/1830> > > > Impersonation Support + Cgroups - https://github.com/apache/ < > > https://github.com/apache/> > > > incubator-airflow/pull/1934 (this is kind of important from the Airbnb > > side > > > so that we can help test the new master without having to cherrypick > this > > > PR on top of it which would make the testing unreliable for others). > > > > > > If there are PRs that affect the core of Airflow that other committers > > > think are important to merge we could include these too. I can commit > to > > > pushing out the Impersonation/Cgroups PR this week pending PR comments. > > > What do you think Bolke? > > > > > > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin <bdbr...@gmail.com > > <mailto:bdbr...@gmail.com>> wrote: > > > > > >> Hey Alex, > > >> > > >> I have noticed the same, and it is also the reason why we have Alpha > > >> versions. For now I have noticed the following: > > >> > > >> * Tasks can get in limbo between scheduler and executor: > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948> < > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948>> > > >> * Try_number not increased due to reset in LocalTaskJob: > > >> https://github.com/apache/incubator-airflow/pull/1969 < > > https://github.com/apache/incubator-airflow/pull/1969> < > > >> https://github.com/apache/incubator-airflow/pull/1969 < > > https://github.com/apache/incubator-airflow/pull/1969>> > > >> * one_failed trigger not executed > > >> > > >> My idea is to move to a Samba style of releases eventually, but for > now > > I > > >> would like to get master into a state that we understand and therefore > > not > > >> accept any patches that do not address any bugs. > > >> > > >> If you (or anyone else) can review the above PRs and add your own as > > well > > >> then I can create another Alpha version. I’ll be on gitter as much as > I > > can > > >> so we can speed up if needed. > > >> > > >> - Bolke > > >> > > >>> On 3 Jan 2017, at 08:51, Alex Van Boxel <a...@vanboxel.be> wrote: > > >>> > > >>> Hey Bolke, > > >>> > > >>> thanks for getting this moving. But I already have some blockers, > > since I > > >>> moved up master to this release (moved from end November to now) > > >> stability > > >>> has gone down (certainly on Celary). I'm trying to identify the core > > >>> problems and see if I can fix them. > > >>> > > >>> On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin <bdbr...@gmail.com > > >> <mailto:bdbr...@gmail.com <mailto:bdbr...@gmail.com>>> wrote: > > >>> > > >>> Dear All, > > >>> > > >>> On the verge of the New Year, I decided to be a little bit cheeky and > > to > > >>> make available an Airflow 1.8.0 Alpha 1. We have been talking about > it > > >> for > > >>> a long time now and by doing this I wanted bootstrap the process. It > > >> should > > >>> by no means be considered an Apache release yet. This is for testing > > >>> purposes in the dev community around Airflow, nothing else. > > >>> > > >>> The build is exactly the same as the state of master (git 410736d) > plus > > >> the > > >>> change to version “1.8.0.alpha1” in version.py. > > >>> > > >>> I am dedicating quite some time next week and beyond to get a release > > >> out. > > >>> Hopefully we can get some help with testing, changelog etc. To make > > this > > >>> possible I would like to propose a freeze to adding new features for > at > > >>> least two weeks - say until Jan 15. > > >>> > > >>> You can find the tar here: http://people.apache.org/~bolke/ < > > http://people.apache.org/~bolke/> < > > >>> http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/> > < > > http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/>>> . > > >> It isn’t signed. Following versions > > >>> will be. SHA is available. > > >>> > > >>> Lastly, Alpha 1 does not have the fix for retries yet. So we will get > > an > > >>> Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in > > >>> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948> < > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948>> < > > >>> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948> < > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > https://github.com/apache/incubator-airflow/pull/1948>>> , but your > > >> feedback > > >>> is required as it is entrenched in new processing code that you are > > >> running > > >>> in production afaik - so I wonder what happens in your fork. > > >>> > > >>> Happy New Year! > > >>> > > >>> Bolke > > >>> > > >>> > > >>> > > >>> -- > > >>> _/ > > >>> _/ Alex Van Boxel > > > > >
