Any more comments :) ?
On Wed, Feb 19, 2020 at 2:13 PM Jarek Potiuk <[email protected]> wrote: > And just to clarify - I do not think we should make "massive" releases of > all providers. Ash - you are completely right we should only release what's > changed AND when it is tested. And it can all be automated so that the > overhead will be rather small. > > And in case it is your concern - together with my team we are really happy > to coordinate the whole process once we agree to it and make sure it is > manageable and support any provider release as long as someone can > contribute system tests for that provider. > > Just to be clear and transparent - Google Cloud is (obviously) supporting > a lot of this work. We are in close contact with the Composer team and they > want to make sure they can release internally the "2.0" versions of GCP > operators in Composer. In fact they already can do it and they know how to > prepare their own packages as discussed previously and there is nothing to > prevent them doing so, but my personal goal is to make it available to all > providers and to all users (regardless if they are using Composer or not > and which cloud services they use). > > Releasing backport packages to pypi officially and managing release > process for everyone seems like a good idea for the community I think :). > > J. > > > On Wed, Feb 19, 2020 at 2:04 PM Jarek Potiuk <[email protected]> > wrote: > >> >> To address point 1: I would favour individual, non-cal ver releases of >>> the providers, and that way we don't have to release the providers that >>> don't change. >>> >> >> Agree with releasing only what's needed. I still think CalVer is good as >> we can release (for example) GCP packages several times if we add new >> operators (we plan to add at least a few in the coming months). Do you have >> an alternative proposal to that Ash if we want to release >> providers-google-cloud several times? What versioning scheme you propose >> for that ? >> >> >>> We _could_ split out the providers in to separate repos if we wanted, >>> much like https://github.com/terraform-providers repos (though we >>> wouldn't do it as a separate github org). Again, all optional. But >>> splitting things in to separate repos should make tests for providers much >>> much quicker, as they'd only have to run a tiny number of tests in >>> comparison to now. >>> That said it's not without it's organisational challenges. >> >> >> I agree - AIP-8 and future releases of Airflow 2.0 are a whole different >> subject. That will include not only splitting but also how we >> approach installation for AIrflow 2.0, whether we use semantic versioning >> and whether we do bugfix releases. This is far more complex discussion and >> it will take a lot of time to decide. And I do not think we need to decide >> now nor we are blocked by lack of decisions there. I see backporting >> packages as related and necessary prerequisite to AIP-8 but I treat it as a >> tactical 1.10 backporting solution now (it does provide some prerequisites >> for AIP-8 but it does not block any approach for the 2.0). >> >> And this is why I would like us to decide how we want to handle AIP-8 :) >>> >> >> My goal is to improve the lives of current users ASAP and I think the >> plan to do it is quite viable. The future plan needs to be decided but I >> think waiting for it is totally not needed. I don't think we need to decide >> now - we can proceed with backporting first and start another discussion on >> 2.0 versioning and packaging later. I would really love to turn it into >> AIP-8 eventually. I just think we can give big value to current 1.10 users >> and we are not at all blocked or locked by this decision with AIP-8 >> discussion. Those two are linked but not depend on each other (similarly as >> AIP-21 - it's linked via system tests but automation is not fully needed to >> be hashed out before we release backporting). I really want to treat >> "airflow-1.10" and "airflow-2.0" separately. >> >> I really don't think there is any real need to wait with that nor it puts >> any big issue on us. People will upgrade to 2.0 regardless if they will >> have providers backport packages installed or not and no matter what we >> decide for 2.0 it won't be impacted by how we treat backport installations. >> I proposed (from the very beginning) that the packages we produce are >> strictly limited to be "1.10-backport" only (and that's why I put it in the >> name of the packages). >> >> >>> -ash >>> On Feb 19 2020, at 12:19 pm, Driesprong, Fokko <[email protected]> >>> wrote: >>> > I'm sorry, I've been a bit busy lately. Keeping track of these >>> discussions >>> > in the evening doesn't always for out, as it seems. >>> > >>> > 1. So instead of releasing one package, as stated in the wiki, we're >>> going >>> > to release over 50 provider packages >>> > (apache-airflow-providers-*-*-1.10-YYYY.MM.DD)? If I understand >>> > it correctly. That sounds to me like a lot of work to release. >>> > 2. CalVer is fine with me. >>> > 3. Why is this compatibility matrix needed? Providers package still >>> has a >>> > version in them, right? If we keep backward compatibility between major >>> > versions, this should go fine. >>> > 4. Personally I like the idea, but I'm not sure if this is going to >>> happen. >>> > >>> > What's the future plan here? Are we going to remove the >>> operators/sensors >>> > completely from the Airflow codebase, as suggested in >>> > >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303 >>> > >>> > Cheers, Fokko >>> > Op wo 19 feb. 2020 om 13:00 schreef Jarek Potiuk < >>> [email protected]>: >>> > > +Fokko Driesprong <[email protected]> -> in case you missed it >>> this is >>> > > the thread where I proposed and we had some discussion about >>> backporting >>> > > release process (you asked for it in the cancelled [VOTE] thread). >>> The >>> > > links to PR are earlier in the discussion but for the sake of >>> restarting >>> > > the discussion I will summarise it here again. Please let me know >>> what you >>> > > think, >>> > > >>> > > There were doubts raised by Ash in >>> > > >>> https://lists.apache.org/thread.html/re9f71fa838671e484ef0e7a1a4011b7c7946838008fce8eb05ad2fa0%40%3Cdev.airflow.apache.org%3E >>> > > so I cancelled the vote and I want to return to the proposal. >>> > > >>> > > I have a modified proposal that is - I think addressing those >>> concerns. >>> > > It also relates very much to parallel discussion we have about AIP-4 >>> - >>> > > automated system tests, which in my view are the best way to "stamp" >>> the >>> > > released backport packages as tested. The AIP-4 discussion is here: >>> > > >>> https://lists.apache.org/thread.html/r8c37e2289e8c8b438acc1415e9f2a1b74cd6feac831e02a17b01d88d%40%3Cdev.airflow.apache.org%3E >>> and >>> > > while it is related - it's not a prerequisite for backport packages. >>> > > >>> > > *Summary of the discussion:* >>> > > Just to summarize the discussion above. We have two PRs now which are >>> > > related: >>> > > >>> > > * [AIRFLOW-6663] Prepare backporting packages - >>> > > https://github.com/apache/airflow/pull/7391 >>> > > * [AIRFLOW-6763] Make systems tests ready for backport tests - >>> > > https://github.com/apache/airflow/pull/7389 >>> > > >>> > > The AIRFLOW-6663 introduce automated scripts that can prepare >>> backport >>> > > packages, the AIRFLOW-6763 introduces changes that allow to use >>> breeze to >>> > > install older releases of Airflow and run system tests automatically >>> using >>> > > backport packages produced in AIRFLOW-6663. >>> > > >>> > > The AIRFLOW-6663 for now produces .whl packages (so ready to be >>> uploaded >>> > > to pypi). We have both one-big "providers" package as well as smaller >>> > > "per-provider" packages ("google.cloud", "apache.cassandra" etc.). >>> So this >>> > > discussion is mainly about how are we going to release them >>> "officially" - >>> > > for anyone to be able to install it via PyPI. >>> > > >>> > > The AIRFLOW-6763 modifies current system tests execution approach >>> that we >>> > > have in GCP to make it more generic and reusable across all >>> operators. >>> > > The current state is that we have nearly 100% coverage of "system >>> tests" >>> > > in GCP. We were adding system tests in GCP for 1.5 year now and they >>> allow >>> > > to test automatically master/backported packages against the old >>> airflow in >>> > > fully automated way (providing that we have external systems we can >>> talk to >>> > > and can authorise in). AIP-4 is a related discussion - on how we can >>> bring >>> > > those system tests to other services and automate execution of those >>> tests >>> > > in regular way in CI. >>> > > >>> > > It's not necessary to implement AIP-4 fully to test backport >>> packages - we >>> > > do not have to fully automate it. We can trigger the tests manually - >>> > > separately for each provider (as soon as those tests will be >>> implemented by >>> > > particular provider). Once we merge AIRFLOW-6763 other providers - >>> not only >>> > > GCP will be able to use similar approach as GCP system tests (it >>> boils down >>> > > to turning example_dags into actually runnable DAGs that will run >>> > > backported operators in a DAG run on Airflow 1.10.x). With >>> AIRFLOW-6763 >>> > > it's super simple and it opens up the possibility for anyone to >>> write and >>> > > test "backport" providers against any 1.10. version of airflow very >>> easily. >>> > > >>> > > *Modified proposal:* >>> > > 1) We should release separate packages per provider. After a bit of >>> > > thought I agree with Ash one big blob is not a good idea for multiple >>> > > reasons (stability, dependencies, etc). I already have a change in >>> my mind >>> > > which will make those packages fully independent to install even if >>> they >>> > > are using cross-provider transfer operators. With that change, >>> transfer >>> > > operators will not work if the "other" package is not installed but >>> the >>> > > "provider" package will be importable at least. +Ash Berlin-Taylor >>> > > <[email protected]> - I think it addresses your "blob" concern. Until >>> each >>> > > provider is tested, this provider's package should be "devel" >>> packages that >>> > > are only available via our CI or manual preparation. >>> > > >>> > > 2) I think still CalVer is best idea for versioning of the provider >>> > > package apache-airflow-providers-google-cloud-1.10-YYYY.MM.DD is an >>> > > example package name/version. Those packages have automatically >>> proper >>> > > dependencies set and we can automatically check if they are clearly >>> > > installing on top of any airflow version. This can be very easily >>> automated >>> > > in CI. >>> > > >>> > > 3) I can introduce and lead a compatibility/test matrix (I already >>> planned >>> > > to keep it in the >>> > > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series >>> document) >>> > > where we will mark version of released packages and whether it was >>> tested >>> > > automatically with system tests (and by whom !). I am super-happy to >>> assume >>> > > the role of release manager for the backport packages and make the >>> process >>> > > happen, update the release guidelines in wiki etc. and finally >>> physically >>> > > make the releases when voting completes on each package and testing >>> is done. >>> > > >>> > > 4) Initially this will be only GCP-related packages (as they already >>> have >>> > > full system test coverage as well as we are dedicated to test them) >>> but >>> > > this should be a learning experience. We will provide clear >>> instructions >>> > > and examples for others how to add and run system tests and maybe >>> later >>> > > with AIP-4 we fully automate it. This way we will be able to mark >>> those >>> > > tested packages as "tested" and release them officially via usual >>> PyPi >>> > > channel. Note that the providers operators are already unit-tested in >>> > > master, so running system tests with old airflow IMHO is the only >>> thing we >>> > > need to see if there are no backporting problems. I will also start >>> > > campaign and encourage our community members to add system tests and >>> run >>> > > the tests for the providers they are interested in. The idea will be >>> to add >>> > > the system tests gradually so that with every future release of >>> backport >>> > > packages we can release more and more provider packages because >>> system >>> > > tests will be automated. I think that should address your ("_we_ >>> should >>> > > test it" concern Ash). >>> > > >>> > > Let me know what you think - Fokko, Ash (does it address your >>> concerns?) >>> > > and others who are interested. >>> > > >>> > > J. >>> > > >>> > > >>> > > >>> > > >>> > > On Mon, Feb 17, 2020 at 9:54 AM Kamil Breguła < >>> [email protected]> >>> > > wrote: >>> > > >>> > > > +1 on voting. >>> > > > I read the discussions very carefully and I agree that it is worth >>> > > > creating a interm solution. It is a very good idea if it will not >>> be a >>> > > > temporary and experimental solution indefinitely. We have a clear >>> > > > deadline, so we can go further with this. >>> > > > >>> > > > On Mon, Feb 17, 2020 at 8:35 AM Jarek Potiuk < >>> [email protected]> >>> > > > wrote: >>> > > > > >>> > > > > Others? WDYT? Shall we start voting on it ? Any more comments? >>> > > > > I think I would like to propose an interim solution where all the >>> > > > > backported packages for 1.10 will be released as a single big >>> package >>> > > > >>> > > > with >>> > > > > Calver Versioning and with some compatibility matrix where we >>> will mark >>> > > > > which of the providers were tested (semi-automatically ?) >>> possibly over >>> > > > > time automatically using system tests (following the AIP-4 >>> proposal). >>> > > > > >>> > > > > Eventually - maybe even for 2.0 - we will be able to split the >>> packages >>> > > > on >>> > > > > per-provider basis and release them independently - but that is >>> > > > >>> > > > something >>> > > > > that we can test and agree later - when we will be discussing >>> overall >>> > > > > release approach (including possibly semantic or calendar >>> versioning for >>> > > > > 2.* releases). >>> > > > > >>> > > > > Let me know if you have any objections, if not, I will call a >>> vote on >>> > > > that >>> > > > > in a day or so. >>> > > > > >>> > > > > J. >>> > > > > >>> > > > > On Fri, Feb 14, 2020 at 9:46 PM Jarek Potiuk < >>> [email protected]> >>> > > > > wrote: >>> > > > > >>> > > > > > How about going both routes ? >>> > > > > > 1) Provide one big "backport" package for 1.10 >>> > > > > > 2) Once we release 2.0 split providers to micro-packages >>> > > > > > >>> > > > > > J. >>> > > > > > On Fri, Feb 14, 2020 at 9:30 PM Ash Berlin-Taylor < >>> [email protected]> >>> > > > wrote: >>> > > > > > >>> > > > > > > I think before we take this discussion any further we should >>> work out >>> > > > > > > what our plan is for AIP-8 >>> > > > > > > >>> > > > >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303&src=contextnavpagetreemode >>> > > > > > > ( though likely needs updating as it still talks about >>> contrib which >>> > > > > > >>> > > > > >>> > > > >>> > > > isn't >>> > > > > > > relevant anymore) >>> > > > > > > >>> > > > > > > AIP-8 talks about "One hook or operator per package, >>> following the >>> > > > "micro >>> > > > > > > package" philosophy." as it's long term goal, and I think I >>> broadly >>> > > > > > >>> > > > > >>> > > > >>> > > > agree >>> > > > > > > with that. >>> > > > > > > Given we have almost all the things in place to have this, I >>> would >>> > > > > > >>> > > > > >>> > > > >>> > > > rather >>> > > > > > > we didn't release a single large "backport" package, only to >>> have to >>> > > > > > >>> > > > > >>> > > > >>> > > > have >>> > > > > > > users to then switch over to using new packages. >>> > > > > > > > We can follow the same process/keys etc as for releasing >>> the main >>> > > > > > > >>> > > > > > > airflow >>> > > > > > > > package, but I think it can be a bit more relaxed in terms >>> of >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > testing - >>> > > > > > > and >>> > > > > > > > we can release it more often (as long as there will be new >>> changes >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > in >>> > > > > > > > providers). Those packages might be released on "as-is" >>> basis - >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > without >>> > > > > > > > guarantee that they work for all operators/hooks/sensors - >>> and >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > without >>> > > > > > > > guarantee that they will work for all 1.10.* versions. >>> > > > > > > >>> > > > > > > >>> > > > > > > I'm in favour of this as a general idea. >>> > > > > > > My preferred way is to have each "provider" be it's own >>> package. >>> > > > > > >>> > > > > >>> > > > >>> > > > This is >>> > > > > > > a slightly fuzzy concept, as for instance >>> airflow.providers.goolgle >>> > > > > > > probably makes sense as a single package rather than a >>> .google.cloud >>> > > > > > >>> > > > > >>> > > > >>> > > > and >>> > > > > > > .google.marketing etc packages (as per Kamil's comment on >>> Github), >>> > > > > > >>> > > > > >>> > > > >>> > > > but >>> > > > > > > apache.airflow.providers.apache should _not_ be one package. >>> So >>> > > > > > >>> > > > > >>> > > > >>> > > > there's no >>> > > > > > > easily expressible rule here, but (to me) there is an >>> obvious way >>> > > > > > >>> > > > > >>> > > > >>> > > > for each >>> > > > > > > case. >>> > > > > > > Anyway, to provide smalle releases of providers as per >>> terraform, or >>> > > > > > >>> > > > > >>> > > > >>> > > > to >>> > > > > > > backport to make 2.0 adoption easier? >>> > > > > > > -a >>> > > > > > > On Feb 11 2020, at 3:43 pm, Jarek Potiuk < >>> [email protected]> >>> > > > > > > wrote: >>> > > > > > > > Any more opinions? >>> > > > > > > > >>> > > > > > > > I gave some thoughts to that and I think we should : >>> > > > > > > > 1) release one big providers* package with Calver >>> versioning - >>> > > > > > > > apache-airflow-providers-backport-2020.02.11 if were to >>> release it >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > today >>> > > > > > > > (we can always break them into smaller packages when we >>> decide in >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > 2.0). >>> > > > > > > And >>> > > > > > > > then we could change the package names. >>> > > > > > > > 2) scheduled or regular releases. We should release them >>> as needed >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > - >>> > > > > > > i.e. >>> > > > > > > > if we have large change at one or few of the providers or >>> serious >>> > > > > > > >>> > > > > > > bugfix, >>> > > > > > > > we can release it again. >>> > > > > > > > 3) it should be manual effort involving voting and PMC >>> approvals. >>> > > > > > > > >>> > > > > > > > What do you think? >>> > > > > > > > J. >>> > > > > > > > >>> > > > > > > > On Mon, Feb 10, 2020 at 2:43 PM Tomasz Urbaszek < >>> > > > > > > [email protected]> >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > I am ok with users building their own packages. >>> > > > > > > > > T. >>> > > > > > > > > On Mon, Feb 10, 2020 at 1:47 PM Jarek Potiuk < >>> > > > > > > > >>> > > > > > > >>> > > > > > > [email protected]> >>> > > > > > > > > wrote: >>> > > > > > > > > >>> > > > > > > > > > I think it should be a deliberate effort for releasing >>> - with >>> > > > > > > voting. We >>> > > > > > > > > > are releasing the source code and IMHO it should >>> follow the >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > same >>> > > > > > > rules as >>> > > > > > > > > > releasing airflow itself. >>> > > > > > > > > > With this change - anyone will be able to build and >>> prepare >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > their >>> > > > > > > own >>> > > > > > > > > >>> > > > > > > > > .whl >>> > > > > > > > > > packages and install them locally, so I do not think >>> there is a >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > need to >>> > > > > > > > > > automatically release those packages? >>> > > > > > > > > > >>> > > > > > > > > > However releasing them in PyPi should be quite an >>> important >>> > > > event >>> > > > > > > as pypi >>> > > > > > > > > > releases are supposed to be used by users not >>> developers. >>> > > > > > > > > > >>> > > > > > > > > > J. >>> > > > > > > > > > On Mon, Feb 10, 2020 at 11:16 AM Tomasz Urbaszek < >>> > > > > > > > > > [email protected]> wrote: >>> > > > > > > > > > >>> > > > > > > > > > > I think as long as we follow: >>> > > > > > > > > > > > The only people who are supposed to know about such >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > developer >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > resources >>> > > > > > > > > > > are individuals actively participating in >>> development or >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > following the >>> > > > > > > > > > >>> > > > > > > > > > dev >>> > > > > > > > > > > list and thus aware of the conditions placed on >>> unreleased >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > materials. >>> > > > > > > > > > > >>> > > > > > > > > > > we should be ok. My impression is that people are >>> usually >>> > > > aware of >>> > > > > > > > > > > what "nightly build" means and what are the risks. >>> But it's >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > just a >>> > > > > > > > > > > suggestion that I made thinking about all those >>> people who >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > contribute >>> > > > > > > > > > > integration and can't use it "officialy" for let say >>> the >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > following 2 >>> > > > > > > > > > > months. I was also thinking about this result >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > >>> https://www.digitalocean.com/currents/december-2019/#generational-expectations-for-open-source-maintenance >>> > > > > > > > > > > :) >>> > > > > > > > > > > >>> > > > > > > > > > > T. >>> > > > > > > > > > > On Mon, Feb 10, 2020 at 10:52 AM Ash Berlin-Taylor < >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > [email protected]> >>> > > > > > > > > > wrote: >>> > > > > > > > > > > > >>> > > > > > > > > > > > That might be a grey area according to my reading >>> of the >>> > > > Apache >>> > > > > > > > > release >>> > > > > > > > > > > policies: >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> https://apache.org/legal/release-policy.html#publication >>> > > > > > > > > > > > > During the process of developing software and >>> preparing a >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > release, >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > various packages are made available to the >>> development >>> > > > community >>> > > > > > > for >>> > > > > > > > > > > testing purposes. Projects MUST direct outsiders >>> towards >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > official >>> > > > > > > > > > >>> > > > > > > > > > releases >>> > > > > > > > > > > rather than raw source repositories, nightly builds, >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > snapshots, >>> > > > > > > release >>> > > > > > > > > > > candidates, or any other similar packages. The only >>> people >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > who are >>> > > > > > > > > > >>> > > > > > > > > > supposed >>> > > > > > > > > > > to know about such developer resources are >>> individuals >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > actively >>> > > > > > > > > > > participating in development or following the dev >>> list and >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > thus >>> > > > > > > aware >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > of >>> > > > > > > > > > > the conditions placed on unreleased materials. >>> > > > > > > > > > > > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek < >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > [email protected]> >>> > > > > > > > > > > wrote: >>> > > > > > > > > > > > > As per the frequency of releases maybe we can >>> consider >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > "nightly >>> > > > > > > > > > > > > builds" for providers? In this way any >>> contributed >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > hook/operator >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > will >>> > > > > > > > > > > > > be pip-installable in 24h, so users can start to >>> use it = >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > test it. >>> > > > > > > > > > > > > This can help us reduce the number of releases >>> with >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > unworking >>> > > > > > > > > > > > > integrations. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Tomek >>> > > > > > > > > > > > > On Mon, Feb 10, 2020 at 12:11 AM Jarek Potiuk < >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > [email protected]> wrote: >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > TL;DR; I wanted to discuss the approach we are >>> going to >>> > > > > > > take for >>> > > > > > > > > > > backported >>> > > > > > > > > > > > > > providers packages. This is important for PMCs >>> to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > decide >>> > > > > > > about >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > how >>> > > > > > > > > > > we are >>> > > > > > > > > > > > > > going to make release process for it, but I >>> wanted to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > make >>> > > > > > > it >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > public >>> > > > > > > > > > > > > > discussion so that anyone else can chime-in >>> and we can >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > discuss it >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > as >>> > > > > > > > > > > a >>> > > > > > > > > > > > > > community. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > *Context* >>> > > > > > > > > > > > > > As explained in the other thread - we are >>> close to have >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > releasable/tested >>> > > > > > > > > > > > > > backport packages for Airflow 1.10.* series for >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > "providers" >>> > > > > > > > > > > > > > operators/hooks/packages. The main purpose of >>> those >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > backport >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > packages is to >>> > > > > > > > > > > > > > let users migrate to the new operators before >>> they >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > migrate >>> > > > > > > to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > 2.0.* >>> > > > > > > > > > > version >>> > > > > > > > > > > > > > of Airflow. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > The 2.0 version is still some time in the >>> future, and >>> > > > we >>> > > > > > > have a >>> > > > > > > > > > > number of >>> > > > > > > > > > > > > > operators/hooks/sensors implemented that are >>> not >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > actively >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > used/tests >>> > > > > > > > > > > > > > because they are in master version. There are >>> a number >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > of >>> > > > > > > changes >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > and fixes >>> > > > > > > > > > > > > > only implemented in master/2.0 so it would be >>> great to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > use >>> > > > > > > them >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > in >>> > > > > > > > > > > 1.10 - >>> > > > > > > > > > > > > > to use the new features but also to test the >>> master >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > versions as >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > early as >>> > > > > > > > > > > > > > possible. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > Another great property of the backport >>> packages is that >>> > > > > > > they can >>> > > > > > > > > be >>> > > > > > > > > > > used to >>> > > > > > > > > > > > > > ease migration process - users can install the >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > "apache-airflow-providers" >>> > > > > > > > > > > > > > package and start using the new operators >>> without >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > migrating >>> > > > > > > to a >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > new >>> > > > > > > > > > > > > > Airflow. They can incrementally move all their >>> DAGs to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > use >>> > > > > > > the >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > new >>> > > > > > > > > > > > > > "providers" package and only after all that is >>> migrated >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > they can >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > migrate >>> > > > > > > > > > > > > > Airflow to 2.0 when they are ready. That >>> allows to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > have a >>> > > > > > > smooth >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > migration >>> > > > > > > > > > > > > > path for those users. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > *Testing* >>> > > > > > > > > > > > > > The issue we have with those packages is that >>> we are >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > not >>> > > > > > > 100% >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > sure >>> > > > > > > > > > > if the >>> > > > > > > > > > > > > > "providers" operators will work with any >>> 1.10.* airflow >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > version. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > There were >>> > > > > > > > > > > > > > no fundamental changes and they SHOULD work - >>> but we >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > never >>> > > > > > > know >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > until we >>> > > > > > > > > > > > > > test. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > Some preliminary tests with subset of GCP >>> operators >>> > > > show >>> > > > > > > that the >>> > > > > > > > > > > operators >>> > > > > > > > > > > > > > work out-of-the box. We have a big set of >>> "system" >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > tests for >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > "GCP" >>> > > > > > > > > > > > > > operators that we will run semi-automatically >>> and make >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > sure >>> > > > > > > that >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > all >>> > > > > > > > > > > GCP >>> > > > > > > > > > > > > > operators are working fine. This is already a >>> great >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > compatibility >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > test (GCP >>> > > > > > > > > > > > > > operators are about 1/3 of all operators for >>> Airflow). >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > But >>> > > > > > > also >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > the >>> > > > > > > > > > > > > > approach used in GCP system tests can be >>> applied to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > other >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > operators. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > I plan to have a matrix of "compatibilities" in >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series >>> > > > > > > > > > > > > > and >>> > > > > > > > > > > > > > ask community to add/run tests with other >>> packages as >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > well. >>> > > > > > > It >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > should be >>> > > > > > > > > > > > > > rather easy to add system tests for other >>> systems - >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > following the >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > way it is >>> > > > > > > > > > > > > > implemented for GCP. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > *Releases* >>> > > > > > > > > > > > > > I think the most important decision is how we >>> are >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > going to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > release >>> > > > > > > > > > > the >>> > > > > > > > > > > > > > packages. This is where PMCs have to decide I >>> think as >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > we >>> > > > > > > have >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > legal >>> > > > > > > > > > > > > > responsibility for releasing Apache Airflow >>> official >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > software. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > What we have now (after the PRs get merged) - >>> wheel and >>> > > > > > > source >>> > > > > > > > > > > packages >>> > > > > > > > > > > > > > build automatically in Travis CI and uploaded >>> to >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > file.io >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > ephemeral >>> > > > > > > > > > > storage. >>> > > > > > > > > > > > > > The builds upload all the packages there - one >>> big >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > "providers" >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > package and >>> > > > > > > > > > > > > > separate packages for each "provider". >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > It would be great if we can officially publish >>> > > > packages for >>> > > > > > > > > > > backporting in >>> > > > > > > > > > > > > > pypi however and here where we have to agree >>> on the >>> > > > > > > > > > > > > > process/versioning/cadence. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > We can follow the same process/keys etc as for >>> > > > releasing >>> > > > > > > the main >>> > > > > > > > > > > airflow >>> > > > > > > > > > > > > > package, but I think it can be a bit more >>> relaxed in >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > terms >>> > > > > > > of >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > testing - and >>> > > > > > > > > > > > > > we can release it more often (as long as there >>> will be >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > new >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > changes >>> > > > > > > > > > in >>> > > > > > > > > > > > > > providers). Those packages might be released >>> on "as-is" >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > basis - >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > without >>> > > > > > > > > > > > > > guarantee that they work for all >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > operators/hooks/sensors - >>> > > > > > > and >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > without >>> > > > > > > > > > > > > > guarantee that they will work for all 1.10.* >>> versions. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > We >>> > > > > > > can >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > have >>> > > > > > > > > > > the >>> > > > > > > > > > > > > > "compatibility" statement/matrix in our wiki >>> where >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > people >>> > > > > > > who >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > tested >>> > > > > > > > > > > some >>> > > > > > > > > > > > > > package might simply state that it works for >>> them. At >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > Polidea we >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > can >>> > > > > > > > > > > assume >>> > > > > > > > > > > > > > stewardship on the GCP packages and test them >>> using our >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > automated >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > system >>> > > > > > > > > > > > > > tests for every release for example - maybe >>> others can >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > assume >>> > > > > > > > > > > > > > stewardship for other providers. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > For that - we will need some versioning/release >>> > > > policy. I >>> > > > > > > would >>> > > > > > > > > say >>> > > > > > > > > > > a CalVer >>> > > > > > > > > > > > > > <https://calver.org/> approach might work best >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > (YYYY.MM.DD). And >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > to >>> > > > > > > > > > > make it >>> > > > > > > > > > > > > > simple we should release one "big" providers >>> package >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > with >>> > > > > > > all >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > providers in. >>> > > > > > > > > > > > > > We can have roughly monthly cadence for it. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > But I am also open to any suggestions here. >>> > > > > > > > > > > > > > Please let me know what you think. >>> > > > > > > > > > > > > > J. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > -- >>> > > > > > > > > > > > > > Jarek Potiuk >>> > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > Software >>> > > > > > > Engineer >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> >>> > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > -- >>> > > > > > > > > > > > > Tomasz Urbaszek >>> > > > > > > > > > > > > Polidea | Software Engineer >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > M: +48 505 628 493 >>> > > > > > > > > > > > > E: [email protected] >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Unique Tech >>> > > > > > > > > > > > > Check out our projects! >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > -- >>> > > > > > > > > > > Tomasz Urbaszek >>> > > > > > > > > > > Polidea | Software Engineer >>> > > > > > > > > > > >>> > > > > > > > > > > M: +48 505 628 493 >>> > > > > > > > > > > E: [email protected] >>> > > > > > > > > > > >>> > > > > > > > > > > Unique Tech >>> > > > > > > > > > > Check out our projects! >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > -- >>> > > > > > > > > > Jarek Potiuk >>> > > > > > > > > > Polidea <https://www.polidea.com/> | Principal >>> Software >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > Engineer >>> > > > > > > > > > >>> > > > > > > > > > M: +48 660 796 129 <+48660796129> >>> > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > -- >>> > > > > > > > > Tomasz Urbaszek >>> > > > > > > > > Polidea <https://www.polidea.com/> | Software Engineer >>> > > > > > > > > >>> > > > > > > > > M: +48 505 628 493 <+48505628493> >>> > > > > > > > > E: [email protected] < >>> [email protected]> >>> > > > > > > > > >>> > > > > > > > > Unique Tech >>> > > > > > > > > Check out our projects! < >>> https://www.polidea.com/our-work> >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > -- >>> > > > > > > > Jarek Potiuk >>> > > > > > > > Polidea <https://www.polidea.com/> | Principal Software >>> Engineer >>> > > > > > > > >>> > > > > > > > M: +48 660 796 129 <+48660796129> >>> > > > > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Jarek Potiuk >>> > > > > > Polidea <https://www.polidea.com/> | Principal Software >>> Engineer >>> > > > > > >>> > > > > > M: +48 660 796 129 <+48660796129> >>> > > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > > > >>> > > > > > >>> > > > > >>> > > > > -- >>> > > > > Jarek Potiuk >>> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer >>> > > > > >>> > > > > M: +48 660 796 129 <+48660796129> >>> > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > >>> > > > >>> > > >>> > > >>> > > -- >>> > > Jarek Potiuk >>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer >>> > > >>> > > M: +48 660 796 129 <+48660796129> >>> > > [image: Polidea] <https://www.polidea.com/> >>> > >>> > >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
