> To address point 1: I would favour individual, non-cal ver releases of the > providers, and that way we don't have to release the providers that don't > change. >
Agree with releasing only what's needed. I still think CalVer is good as we can release (for example) GCP packages several times if we add new operators (we plan to add at least a few in the coming months). Do you have an alternative proposal to that Ash if we want to release providers-google-cloud several times? What versioning scheme you propose for that ? > We _could_ split out the providers in to separate repos if we wanted, much > like https://github.com/terraform-providers repos (though we wouldn't do > it as a separate github org). Again, all optional. But splitting things in > to separate repos should make tests for providers much much quicker, as > they'd only have to run a tiny number of tests in comparison to now. > That said it's not without it's organisational challenges. I agree - AIP-8 and future releases of Airflow 2.0 are a whole different subject. That will include not only splitting but also how we approach installation for AIrflow 2.0, whether we use semantic versioning and whether we do bugfix releases. This is far more complex discussion and it will take a lot of time to decide. And I do not think we need to decide now nor we are blocked by lack of decisions there. I see backporting packages as related and necessary prerequisite to AIP-8 but I treat it as a tactical 1.10 backporting solution now (it does provide some prerequisites for AIP-8 but it does not block any approach for the 2.0). And this is why I would like us to decide how we want to handle AIP-8 :) > My goal is to improve the lives of current users ASAP and I think the plan to do it is quite viable. The future plan needs to be decided but I think waiting for it is totally not needed. I don't think we need to decide now - we can proceed with backporting first and start another discussion on 2.0 versioning and packaging later. I would really love to turn it into AIP-8 eventually. I just think we can give big value to current 1.10 users and we are not at all blocked or locked by this decision with AIP-8 discussion. Those two are linked but not depend on each other (similarly as AIP-21 - it's linked via system tests but automation is not fully needed to be hashed out before we release backporting). I really want to treat "airflow-1.10" and "airflow-2.0" separately. I really don't think there is any real need to wait with that nor it puts any big issue on us. People will upgrade to 2.0 regardless if they will have providers backport packages installed or not and no matter what we decide for 2.0 it won't be impacted by how we treat backport installations. I proposed (from the very beginning) that the packages we produce are strictly limited to be "1.10-backport" only (and that's why I put it in the name of the packages). > -ash > On Feb 19 2020, at 12:19 pm, Driesprong, Fokko <[email protected]> > wrote: > > I'm sorry, I've been a bit busy lately. Keeping track of these > discussions > > in the evening doesn't always for out, as it seems. > > > > 1. So instead of releasing one package, as stated in the wiki, we're > going > > to release over 50 provider packages > > (apache-airflow-providers-*-*-1.10-YYYY.MM.DD)? If I understand > > it correctly. That sounds to me like a lot of work to release. > > 2. CalVer is fine with me. > > 3. Why is this compatibility matrix needed? Providers package still has a > > version in them, right? If we keep backward compatibility between major > > versions, this should go fine. > > 4. Personally I like the idea, but I'm not sure if this is going to > happen. > > > > What's the future plan here? Are we going to remove the operators/sensors > > completely from the Airflow codebase, as suggested in > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303 > > > > Cheers, Fokko > > Op wo 19 feb. 2020 om 13:00 schreef Jarek Potiuk < > [email protected]>: > > > +Fokko Driesprong <[email protected]> -> in case you missed it > this is > > > the thread where I proposed and we had some discussion about > backporting > > > release process (you asked for it in the cancelled [VOTE] thread). The > > > links to PR are earlier in the discussion but for the sake of > restarting > > > the discussion I will summarise it here again. Please let me know what > you > > > think, > > > > > > There were doubts raised by Ash in > > > > https://lists.apache.org/thread.html/re9f71fa838671e484ef0e7a1a4011b7c7946838008fce8eb05ad2fa0%40%3Cdev.airflow.apache.org%3E > > > so I cancelled the vote and I want to return to the proposal. > > > > > > I have a modified proposal that is - I think addressing those concerns. > > > It also relates very much to parallel discussion we have about AIP-4 - > > > automated system tests, which in my view are the best way to "stamp" > the > > > released backport packages as tested. The AIP-4 discussion is here: > > > > https://lists.apache.org/thread.html/r8c37e2289e8c8b438acc1415e9f2a1b74cd6feac831e02a17b01d88d%40%3Cdev.airflow.apache.org%3E > and > > > while it is related - it's not a prerequisite for backport packages. > > > > > > *Summary of the discussion:* > > > Just to summarize the discussion above. We have two PRs now which are > > > related: > > > > > > * [AIRFLOW-6663] Prepare backporting packages - > > > https://github.com/apache/airflow/pull/7391 > > > * [AIRFLOW-6763] Make systems tests ready for backport tests - > > > https://github.com/apache/airflow/pull/7389 > > > > > > The AIRFLOW-6663 introduce automated scripts that can prepare backport > > > packages, the AIRFLOW-6763 introduces changes that allow to use breeze > to > > > install older releases of Airflow and run system tests automatically > using > > > backport packages produced in AIRFLOW-6663. > > > > > > The AIRFLOW-6663 for now produces .whl packages (so ready to be > uploaded > > > to pypi). We have both one-big "providers" package as well as smaller > > > "per-provider" packages ("google.cloud", "apache.cassandra" etc.). So > this > > > discussion is mainly about how are we going to release them > "officially" - > > > for anyone to be able to install it via PyPI. > > > > > > The AIRFLOW-6763 modifies current system tests execution approach that > we > > > have in GCP to make it more generic and reusable across all operators. > > > The current state is that we have nearly 100% coverage of "system > tests" > > > in GCP. We were adding system tests in GCP for 1.5 year now and they > allow > > > to test automatically master/backported packages against the old > airflow in > > > fully automated way (providing that we have external systems we can > talk to > > > and can authorise in). AIP-4 is a related discussion - on how we can > bring > > > those system tests to other services and automate execution of those > tests > > > in regular way in CI. > > > > > > It's not necessary to implement AIP-4 fully to test backport packages > - we > > > do not have to fully automate it. We can trigger the tests manually - > > > separately for each provider (as soon as those tests will be > implemented by > > > particular provider). Once we merge AIRFLOW-6763 other providers - not > only > > > GCP will be able to use similar approach as GCP system tests (it boils > down > > > to turning example_dags into actually runnable DAGs that will run > > > backported operators in a DAG run on Airflow 1.10.x). With AIRFLOW-6763 > > > it's super simple and it opens up the possibility for anyone to write > and > > > test "backport" providers against any 1.10. version of airflow very > easily. > > > > > > *Modified proposal:* > > > 1) We should release separate packages per provider. After a bit of > > > thought I agree with Ash one big blob is not a good idea for multiple > > > reasons (stability, dependencies, etc). I already have a change in my > mind > > > which will make those packages fully independent to install even if > they > > > are using cross-provider transfer operators. With that change, transfer > > > operators will not work if the "other" package is not installed but the > > > "provider" package will be importable at least. +Ash Berlin-Taylor > > > <[email protected]> - I think it addresses your "blob" concern. Until > each > > > provider is tested, this provider's package should be "devel" packages > that > > > are only available via our CI or manual preparation. > > > > > > 2) I think still CalVer is best idea for versioning of the provider > > > package apache-airflow-providers-google-cloud-1.10-YYYY.MM.DD is an > > > example package name/version. Those packages have automatically proper > > > dependencies set and we can automatically check if they are clearly > > > installing on top of any airflow version. This can be very easily > automated > > > in CI. > > > > > > 3) I can introduce and lead a compatibility/test matrix (I already > planned > > > to keep it in the > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series > document) > > > where we will mark version of released packages and whether it was > tested > > > automatically with system tests (and by whom !). I am super-happy to > assume > > > the role of release manager for the backport packages and make the > process > > > happen, update the release guidelines in wiki etc. and finally > physically > > > make the releases when voting completes on each package and testing is > done. > > > > > > 4) Initially this will be only GCP-related packages (as they already > have > > > full system test coverage as well as we are dedicated to test them) but > > > this should be a learning experience. We will provide clear > instructions > > > and examples for others how to add and run system tests and maybe later > > > with AIP-4 we fully automate it. This way we will be able to mark those > > > tested packages as "tested" and release them officially via usual PyPi > > > channel. Note that the providers operators are already unit-tested in > > > master, so running system tests with old airflow IMHO is the only > thing we > > > need to see if there are no backporting problems. I will also start > > > campaign and encourage our community members to add system tests and > run > > > the tests for the providers they are interested in. The idea will be > to add > > > the system tests gradually so that with every future release of > backport > > > packages we can release more and more provider packages because system > > > tests will be automated. I think that should address your ("_we_ should > > > test it" concern Ash). > > > > > > Let me know what you think - Fokko, Ash (does it address your > concerns?) > > > and others who are interested. > > > > > > J. > > > > > > > > > > > > > > > On Mon, Feb 17, 2020 at 9:54 AM Kamil Breguła < > [email protected]> > > > wrote: > > > > > > > +1 on voting. > > > > I read the discussions very carefully and I agree that it is worth > > > > creating a interm solution. It is a very good idea if it will not be > a > > > > temporary and experimental solution indefinitely. We have a clear > > > > deadline, so we can go further with this. > > > > > > > > On Mon, Feb 17, 2020 at 8:35 AM Jarek Potiuk < > [email protected]> > > > > wrote: > > > > > > > > > > Others? WDYT? Shall we start voting on it ? Any more comments? > > > > > I think I would like to propose an interim solution where all the > > > > > backported packages for 1.10 will be released as a single big > package > > > > > > > > with > > > > > Calver Versioning and with some compatibility matrix where we will > mark > > > > > which of the providers were tested (semi-automatically ?) possibly > over > > > > > time automatically using system tests (following the AIP-4 > proposal). > > > > > > > > > > Eventually - maybe even for 2.0 - we will be able to split the > packages > > > > on > > > > > per-provider basis and release them independently - but that is > > > > > > > > something > > > > > that we can test and agree later - when we will be discussing > overall > > > > > release approach (including possibly semantic or calendar > versioning for > > > > > 2.* releases). > > > > > > > > > > Let me know if you have any objections, if not, I will call a vote > on > > > > that > > > > > in a day or so. > > > > > > > > > > J. > > > > > > > > > > On Fri, Feb 14, 2020 at 9:46 PM Jarek Potiuk < > [email protected]> > > > > > wrote: > > > > > > > > > > > How about going both routes ? > > > > > > 1) Provide one big "backport" package for 1.10 > > > > > > 2) Once we release 2.0 split providers to micro-packages > > > > > > > > > > > > J. > > > > > > On Fri, Feb 14, 2020 at 9:30 PM Ash Berlin-Taylor < > [email protected]> > > > > wrote: > > > > > > > > > > > > > I think before we take this discussion any further we should > work out > > > > > > > what our plan is for AIP-8 > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303&src=contextnavpagetreemode > > > > > > > ( though likely needs updating as it still talks about contrib > which > > > > > > > > > > > > > > > > > > > isn't > > > > > > > relevant anymore) > > > > > > > > > > > > > > AIP-8 talks about "One hook or operator per package, following > the > > > > "micro > > > > > > > package" philosophy." as it's long term goal, and I think I > broadly > > > > > > > > > > > > > > > > > > > agree > > > > > > > with that. > > > > > > > Given we have almost all the things in place to have this, I > would > > > > > > > > > > > > > > > > > > > rather > > > > > > > we didn't release a single large "backport" package, only to > have to > > > > > > > > > > > > > > > > > > > have > > > > > > > users to then switch over to using new packages. > > > > > > > > We can follow the same process/keys etc as for releasing the > main > > > > > > > > > > > > > > airflow > > > > > > > > package, but I think it can be a bit more relaxed in terms of > > > > > > > > > > > > > > > > > > > > > > > > > > testing - > > > > > > > and > > > > > > > > we can release it more often (as long as there will be new > changes > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > providers). Those packages might be released on "as-is" > basis - > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > guarantee that they work for all operators/hooks/sensors - > and > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > guarantee that they will work for all 1.10.* versions. > > > > > > > > > > > > > > > > > > > > > I'm in favour of this as a general idea. > > > > > > > My preferred way is to have each "provider" be it's own > package. > > > > > > > > > > > > > > > > > > > This is > > > > > > > a slightly fuzzy concept, as for instance > airflow.providers.goolgle > > > > > > > probably makes sense as a single package rather than a > .google.cloud > > > > > > > > > > > > > > > > > > > and > > > > > > > .google.marketing etc packages (as per Kamil's comment on > Github), > > > > > > > > > > > > > > > > > > > but > > > > > > > apache.airflow.providers.apache should _not_ be one package. So > > > > > > > > > > > > > > > > > > > there's no > > > > > > > easily expressible rule here, but (to me) there is an obvious > way > > > > > > > > > > > > > > > > > > > for each > > > > > > > case. > > > > > > > Anyway, to provide smalle releases of providers as per > terraform, or > > > > > > > > > > > > > > > > > > > to > > > > > > > backport to make 2.0 adoption easier? > > > > > > > -a > > > > > > > On Feb 11 2020, at 3:43 pm, Jarek Potiuk < > [email protected]> > > > > > > > wrote: > > > > > > > > Any more opinions? > > > > > > > > > > > > > > > > I gave some thoughts to that and I think we should : > > > > > > > > 1) release one big providers* package with Calver versioning > - > > > > > > > > apache-airflow-providers-backport-2020.02.11 if were to > release it > > > > > > > > > > > > > > > > > > > > > > > > > > today > > > > > > > > (we can always break them into smaller packages when we > decide in > > > > > > > > > > > > > > > > > > > > > > > > > > 2.0). > > > > > > > And > > > > > > > > then we could change the package names. > > > > > > > > 2) scheduled or regular releases. We should release them as > needed > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > i.e. > > > > > > > > if we have large change at one or few of the providers or > serious > > > > > > > > > > > > > > bugfix, > > > > > > > > we can release it again. > > > > > > > > 3) it should be manual effort involving voting and PMC > approvals. > > > > > > > > > > > > > > > > What do you think? > > > > > > > > J. > > > > > > > > > > > > > > > > On Mon, Feb 10, 2020 at 2:43 PM Tomasz Urbaszek < > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I am ok with users building their own packages. > > > > > > > > > T. > > > > > > > > > On Mon, Feb 10, 2020 at 1:47 PM Jarek Potiuk < > > > > > > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I think it should be a deliberate effort for releasing - > with > > > > > > > voting. We > > > > > > > > > > are releasing the source code and IMHO it should follow > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > same > > > > > > > rules as > > > > > > > > > > releasing airflow itself. > > > > > > > > > > With this change - anyone will be able to build and > prepare > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > their > > > > > > > own > > > > > > > > > > > > > > > > > > .whl > > > > > > > > > > packages and install them locally, so I do not think > there is a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > need to > > > > > > > > > > automatically release those packages? > > > > > > > > > > > > > > > > > > > > However releasing them in PyPi should be quite an > important > > > > event > > > > > > > as pypi > > > > > > > > > > releases are supposed to be used by users not developers. > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > On Mon, Feb 10, 2020 at 11:16 AM Tomasz Urbaszek < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > I think as long as we follow: > > > > > > > > > > > > The only people who are supposed to know about such > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > developer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > resources > > > > > > > > > > > are individuals actively participating in development > or > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > following the > > > > > > > > > > > > > > > > > > > > dev > > > > > > > > > > > list and thus aware of the conditions placed on > unreleased > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > materials. > > > > > > > > > > > > > > > > > > > > > > we should be ok. My impression is that people are > usually > > > > aware of > > > > > > > > > > > what "nightly build" means and what are the risks. But > it's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > just a > > > > > > > > > > > suggestion that I made thinking about all those people > who > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > integration and can't use it "officialy" for let say > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > following 2 > > > > > > > > > > > months. I was also thinking about this result > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.digitalocean.com/currents/december-2019/#generational-expectations-for-open-source-maintenance > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > T. > > > > > > > > > > > On Mon, Feb 10, 2020 at 10:52 AM Ash Berlin-Taylor < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > That might be a grey area according to my reading of > the > > > > Apache > > > > > > > > > release > > > > > > > > > > > policies: > > > > > > > > > > > > > > > > > > > > > > > > > https://apache.org/legal/release-policy.html#publication > > > > > > > > > > > > > During the process of developing software and > preparing a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > release, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > various packages are made available to the development > > > > community > > > > > > > for > > > > > > > > > > > testing purposes. Projects MUST direct outsiders > towards > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > official > > > > > > > > > > > > > > > > > > > > releases > > > > > > > > > > > rather than raw source repositories, nightly builds, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > snapshots, > > > > > > > release > > > > > > > > > > > candidates, or any other similar packages. The only > people > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > who are > > > > > > > > > > > > > > > > > > > > supposed > > > > > > > > > > > to know about such developer resources are individuals > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively > > > > > > > > > > > participating in development or following the dev list > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thus > > > > > > > aware > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > the conditions placed on unreleased materials. > > > > > > > > > > > > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > As per the frequency of releases maybe we can > consider > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "nightly > > > > > > > > > > > > > builds" for providers? In this way any contributed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > hook/operator > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > be pip-installable in 24h, so users can start to > use it = > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > test it. > > > > > > > > > > > > > This can help us reduce the number of releases with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > unworking > > > > > > > > > > > > > integrations. > > > > > > > > > > > > > > > > > > > > > > > > > > Tomek > > > > > > > > > > > > > On Mon, Feb 10, 2020 at 12:11 AM Jarek Potiuk < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > TL;DR; I wanted to discuss the approach we are > going to > > > > > > > take for > > > > > > > > > > > backported > > > > > > > > > > > > > > providers packages. This is important for PMCs to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > decide > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > how > > > > > > > > > > > we are > > > > > > > > > > > > > > going to make release process for it, but I > wanted to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > make > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public > > > > > > > > > > > > > > discussion so that anyone else can chime-in and > we can > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > discuss it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > a > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Context* > > > > > > > > > > > > > > As explained in the other thread - we are close > to have > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > releasable/tested > > > > > > > > > > > > > > backport packages for Airflow 1.10.* series for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "providers" > > > > > > > > > > > > > > operators/hooks/packages. The main purpose of > those > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backport > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > packages is to > > > > > > > > > > > > > > let users migrate to the new operators before > they > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > migrate > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2.0.* > > > > > > > > > > > version > > > > > > > > > > > > > > of Airflow. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The 2.0 version is still some time in the > future, and > > > > we > > > > > > > have a > > > > > > > > > > > number of > > > > > > > > > > > > > > operators/hooks/sensors implemented that are not > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > used/tests > > > > > > > > > > > > > > because they are in master version. There are a > number > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > changes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and fixes > > > > > > > > > > > > > > only implemented in master/2.0 so it would be > great to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > use > > > > > > > them > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > 1.10 - > > > > > > > > > > > > > > to use the new features but also to test the > master > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > versions as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > early as > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another great property of the backport packages > is that > > > > > > > they can > > > > > > > > > be > > > > > > > > > > > used to > > > > > > > > > > > > > > ease migration process - users can install the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "apache-airflow-providers" > > > > > > > > > > > > > > package and start using the new operators without > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > migrating > > > > > > > to a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > Airflow. They can incrementally move all their > DAGs to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > use > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > "providers" package and only after all that is > migrated > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > they can > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > migrate > > > > > > > > > > > > > > Airflow to 2.0 when they are ready. That allows > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have a > > > > > > > smooth > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > migration > > > > > > > > > > > > > > path for those users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Testing* > > > > > > > > > > > > > > The issue we have with those packages is that we > are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not > > > > > > > 100% > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > sure > > > > > > > > > > > if the > > > > > > > > > > > > > > "providers" operators will work with any 1.10.* > airflow > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were > > > > > > > > > > > > > > no fundamental changes and they SHOULD work - > but we > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > never > > > > > > > know > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > until we > > > > > > > > > > > > > > test. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some preliminary tests with subset of GCP > operators > > > > show > > > > > > > that the > > > > > > > > > > > operators > > > > > > > > > > > > > > work out-of-the box. We have a big set of > "system" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tests for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "GCP" > > > > > > > > > > > > > > operators that we will run semi-automatically > and make > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > sure > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > GCP > > > > > > > > > > > > > > operators are working fine. This is already a > great > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > test (GCP > > > > > > > > > > > > > > operators are about 1/3 of all operators for > Airflow). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > approach used in GCP system tests can be applied > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > operators. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I plan to have a matrix of "compatibilities" in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series > > > > > > > > > > > > > > and > > > > > > > > > > > > > > ask community to add/run tests with other > packages as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well. > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > should be > > > > > > > > > > > > > > rather easy to add system tests for other > systems - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > following the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > way it is > > > > > > > > > > > > > > implemented for GCP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Releases* > > > > > > > > > > > > > > I think the most important decision is how we are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > going to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > release > > > > > > > > > > > the > > > > > > > > > > > > > > packages. This is where PMCs have to decide I > think as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > legal > > > > > > > > > > > > > > responsibility for releasing Apache Airflow > official > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > software. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What we have now (after the PRs get merged) - > wheel and > > > > > > > source > > > > > > > > > > > packages > > > > > > > > > > > > > > build automatically in Travis CI and uploaded to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > file.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ephemeral > > > > > > > > > > > storage. > > > > > > > > > > > > > > The builds upload all the packages there - one > big > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "providers" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > package and > > > > > > > > > > > > > > separate packages for each "provider". > > > > > > > > > > > > > > > > > > > > > > > > > > > > It would be great if we can officially publish > > > > packages for > > > > > > > > > > > backporting in > > > > > > > > > > > > > > pypi however and here where we have to agree on > the > > > > > > > > > > > > > > process/versioning/cadence. > > > > > > > > > > > > > > > > > > > > > > > > > > > > We can follow the same process/keys etc as for > > > > releasing > > > > > > > the main > > > > > > > > > > > airflow > > > > > > > > > > > > > > package, but I think it can be a bit more > relaxed in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > terms > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > testing - and > > > > > > > > > > > > > > we can release it more often (as long as there > will be > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > in > > > > > > > > > > > > > > providers). Those packages might be released on > "as-is" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > basis - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > > > > > > > guarantee that they work for all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > operators/hooks/sensors - > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > > > > > > > guarantee that they will work for all 1.10.* > versions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > the > > > > > > > > > > > > > > "compatibility" statement/matrix in our wiki > where > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > people > > > > > > > who > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tested > > > > > > > > > > > some > > > > > > > > > > > > > > package might simply state that it works for > them. At > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Polidea we > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > assume > > > > > > > > > > > > > > stewardship on the GCP packages and test them > using our > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > automated > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > tests for every release for example - maybe > others can > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > assume > > > > > > > > > > > > > > stewardship for other providers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > For that - we will need some versioning/release > > > > policy. I > > > > > > > would > > > > > > > > > say > > > > > > > > > > > a CalVer > > > > > > > > > > > > > > <https://calver.org/> approach might work best > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (YYYY.MM.DD). And > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > make it > > > > > > > > > > > > > > simple we should release one "big" providers > package > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > providers in. > > > > > > > > > > > > > > We can have roughly monthly cadence for it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I am also open to any suggestions here. > > > > > > > > > > > > > > Please let me know what you think. > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Software > > > > > > > Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Tomasz Urbaszek > > > > > > > > > > > > > Polidea | Software Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > M: +48 505 628 493 > > > > > > > > > > > > > E: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > Unique Tech > > > > > > > > > > > > > Check out our projects! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Tomasz Urbaszek > > > > > > > > > > > Polidea | Software Engineer > > > > > > > > > > > > > > > > > > > > > > M: +48 505 628 493 > > > > > > > > > > > E: [email protected] > > > > > > > > > > > > > > > > > > > > > > Unique Tech > > > > > > > > > > > Check out our projects! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Engineer > > > > > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Tomasz Urbaszek > > > > > > > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > > > > > > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > > > > > > E: [email protected] < > [email protected]> > > > > > > > > > > > > > > > > > > Unique Tech > > > > > > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > -- > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
