BTW. It will also be possible for anyone in the community to cherry-pick changes from main and make a PR (which also a committer will have to approve and merge). This is really no different that we have already done with cherry-picked commits to "v1-10-stable" and "v-2-3-stable" branches by non-committers. Random example here: https://github.com/apache/airflow/pull/14090
We do not give any privileges to the organisations. Quite the opposite - we make them responsible for preparing the PRs to be reviewed by committers. J. On Mon, Jun 20, 2022 at 9:07 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > I think we should continue to be strictly vendor-neutral. No > organization should be able to gain special privileges or control a > project’s direction. > > This is strictly vendor-neutral - Kamil - we are going to release the same > changes that we are releasing already in main providers, just selectively > cherry-picked (and then reviewed and merged by committer to the branch in > airflow repo) - why do you think it is non-vendor neutral? > > On Mon, Jun 20, 2022 at 9:04 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > We can keep these branches in forks managed by stakeholders teams, but >> I am afraid of the benefit that it will be then copied by us to our >> repository and then released by us. If the release was prepared by an >> external team, I think we should make it clear that it was prepared by >> another team, including by publishing on the Pypi account of the team that >> dealt with it. >> >> > Yes this is exactly what I proposed. >> >> Correction. I misread it. >> >> We are going to merge - only the cherry-picked changes that have been >> reviewed and merged by the committer. Same way as today. Just the process >> of cherry-picks is going to be done by the stakeholders, selecting the >> things to cherry-pick (all the changes to cherry-pick should be already >> merged in main). What we are going to do is to release subset of the >> changes we already approved (and released - because we are going to release >> those changes in the latest provider). So there will be no "new" changes in >> those forks - those will be just cherry-picked changes reviewed by the >> committer. There is no reason to mark it as "other" code - it will be the >> same changes that are going to be released anyway (just a subset of those). >> >> J. >> >> On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> > Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>> committers, because only they have write permission to the apache/airflow >>> repository. As far as I know, Github does not allow us to grant write-only >>> permissions to the selected branch. >>> >>> Kamil - you misunderstood it. The branch will be in the FORK of those >>> users's choice - not in airflow repo. committer will merge that branch in >>> the same way we do as today - but with fast-forwarding rather than >>> squashing. >>> >>> > We can keep these branches in forks managed by stakeholders teams, but >>> I am afraid of the benefit that it will be then copied by us to our >>> repository and then released by us. If the release was prepared by an >>> external team, I think we should make it clear that it was prepared by >>> another team, including by publishing on the Pypi account of the team that >>> dealt with it. >>> >>> Yes this is exactly what I proposed. >>> >>> > I think that everything Apache PMC releases should be prepared and >>> created fully within the apache / airflow repositories. If stakeholders >>> team do not have such a possibility, we should figure out that these teams >>> become part of the community, and therefore work together with the entire >>> community, not in isolation. Only then will we be able to act in accordance >>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>> particular each individual person will be able to contribute to the >>> community as an individual, and not as a company or stakeholders team >>> (Community of Peers) and no person will get special privileges just on the >>> basis of their employment status (Earned Authority) >>> >>> Very true. And exactly follows my proposal :) >>> >>> J. >>> >>> >>> On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >>> >>>> +1 -- We have discussed this during the Airflow Summit in-person with >>>> Ash, Rafal (and his team), Jarek and I about this for a long time, and I >>>> think this is a good step forward. >>>> >>>> Regards, >>>> Kaxil >>>> >>>> On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <dzaku...@gmail.com> wrote: >>>> >>>>> I think we should continue to be strictly vendor-neutral. No >>>>> organization should be able to gain special privileges or control a >>>>> project’s direction. >>>>> >>>>> pon., 20 cze 2022 o 18:14 Kamil Breguła <dzaku...@gmail.com> >>>>> napisał(a): >>>>> >>>>>> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>>>> committers, because only they have write permission to the apache/airflow >>>>>> repository. As far as I know, Github does not allow us to grant >>>>>> write-only >>>>>> permissions to the selected branch. >>>>>> >>>>>> We can keep these branches in forks managed by stakeholders teams, >>>>>> but I am afraid of the benefit that it will be then copied by us to our >>>>>> repository and then released by us. If the release was prepared by an >>>>>> external team, I think we should make it clear that it was prepared by >>>>>> another team, including by publishing on the Pypi account of the team >>>>>> that >>>>>> dealt with it. >>>>>> >>>>>> I think that everything Apache PMC releases should be prepared and >>>>>> created fully within the apache / airflow repositories. If stakeholders >>>>>> team do not have such a possibility, we should figure out that these >>>>>> teams >>>>>> become part of the community, and therefore work together with the entire >>>>>> community, not in isolation. Only then will we be able to act in >>>>>> accordance >>>>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>>>> particular each individual person will be able to contribute to the >>>>>> community as an individual, and not as a company or stakeholders team >>>>>> (Community of Peers) and no person will get special privileges just on >>>>>> the >>>>>> basis of their employment status (Earned Authority) >>>>>> >>>>>> >>>>>> >>>>>> pon., 20 cze 2022 o 15:54 Jarek Potiuk <ja...@potiuk.com> napisał(a): >>>>>> >>>>>>> > Will the people who maintain the providers' packages have the >>>>>>> commiter >>>>>>> >>>>>>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>>>>>> cherry-picking can be done in separate branches (and in this case in >>>>>>> forks). >>>>>>> Then the branch can be fast-forwarded by the committer in the >>>>>>> "airflow" repo. No problem with that. >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <dzaku...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Will the people who maintain the providers' packages have the >>>>>>>> commiter >>>>>>>> status? I guess it is necessary for people to have write access to >>>>>>>> the >>>>>>>> repository and therefore to be able to make cherry-pick changes to >>>>>>>> the >>>>>>>> branch. >>>>>>>> >>>>>>>> pon., 20 cze 2022 o 09:13 Elad Kalif <elad...@apache.org> >>>>>>>> napisał(a): >>>>>>>> > >>>>>>>> > +1 >>>>>>>> > From my side the proposal handles all concerns I raised in >>>>>>>> previous threads. >>>>>>>> > I think mixed-governance is a step in the right direction. >>>>>>>> > >>>>>>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <ja...@potiuk.com> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Hello everyone, >>>>>>>> >> >>>>>>>> >> This is a follow-up after a few discussions started about >>>>>>>> providers that were put on hold around the summit. I held a number of >>>>>>>> discussions during theSummit and after, and as result I think I have a >>>>>>>> proposal that can move forward some of the "stalled" decisions we need >>>>>>>> to >>>>>>>> make. >>>>>>>> >> >>>>>>>> >> TL;DR; >>>>>>>> >> >>>>>>>> >> My proposal is a "mixed-governance" model where "stakeholders" >>>>>>>> are more responsible for cherry-picking and testing their providers >>>>>>>> (including system testing) while Airflow PMC members will continue to >>>>>>>> be >>>>>>>> responsible for releasing them. >>>>>>>> >> >>>>>>>> >> Why do we need that? >>>>>>>> >> >>>>>>>> >> Google, Amazon and possibly others teams who are interested in >>>>>>>> maintaining more backwards compatible versions of their providers will >>>>>>>> commit to make PRs of the cherry-picks for older release branches of >>>>>>>> their >>>>>>>> providers. Those providers we release in parallel with the latest >>>>>>>> versions >>>>>>>> during the normal provider cycle. We can deprecate changes more >>>>>>>> aggressively in the "latest" release if we do that. >>>>>>>> >> >>>>>>>> >> Those cherry-picked PRs will be driven, tested and performed by >>>>>>>> the stakeholder teams (Google/Amazon, Databricks, others) and will only >>>>>>>> contain cherry-picks, while we - as PMC - will release them following >>>>>>>> the >>>>>>>> ASF rules (this is very important for the ASF to follow strict release >>>>>>>> policies regarding who and how performs releases). >>>>>>>> >> >>>>>>>> >> This also allows us to introduce similar rules for new >>>>>>>> provider's acceptance for new providers for "main releases". It also >>>>>>>> allows >>>>>>>> running the "system tests" for the provider under control of the >>>>>>>> stakeholder (after applying AIP-47 changes). >>>>>>>> >> >>>>>>>> >> Example 1: Google team can cherry-pick changes to a >>>>>>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>>>>>> provider with some of the bug-fixes and features (together with - say >>>>>>>> latest 8.1.0). >>>>>>>> >> >>>>>>>> >> Example 2: DataLake provider from Databricks - can get accepted >>>>>>>> if Databricks commits to maintaining it. We will release the provider >>>>>>>> as >>>>>>>> long as Databricks maintains it. >>>>>>>> >> >>>>>>>> >> Longer context: >>>>>>>> >> >>>>>>>> >> I have - in my mind so far - a longer roadmap for providers that >>>>>>>> will lead them to be separated from the core and I want to write an AIP >>>>>>>> about that soon. This AIP will detail all the steps needed - I will >>>>>>>> work >>>>>>>> with multiple interested parties on it and it will take some time to >>>>>>>> agree >>>>>>>> and complete. But I want to start with something tangible that will >>>>>>>> solve >>>>>>>> quite a few problems that were raised recently and something that >>>>>>>> seems to >>>>>>>> be possible to be solved in the current provider release cycle (till >>>>>>>> the >>>>>>>> end of June) and test some of the governance approach. >>>>>>>> >> >>>>>>>> >> This proposal simply builds on our semver approach - we do not >>>>>>>> change it, we just start releasing some providers (those that have some >>>>>>>> backing from stakeholders) in more than one version - including >>>>>>>> "latest" >>>>>>>> and earlier, more backwards-compatible branches. Not all providers - >>>>>>>> just >>>>>>>> some. Not all branches - just those that the stakeholders will commit >>>>>>>> to >>>>>>>> maintain. >>>>>>>> >> >>>>>>>> >> We need such a commitment from stakeholders, because we - as >>>>>>>> the Airflow community and maintainers, want to only actively maintain >>>>>>>> the >>>>>>>> latest releases, where it is in the interest of the stakeholders to >>>>>>>> cherry-pick and test also earlier, more backwards compatible releases >>>>>>>> of >>>>>>>> their choice. >>>>>>>> >> >>>>>>>> >> What problems this proposal solves: >>>>>>>> >> >>>>>>>> >> * Problem 1: DEPRECATION REMOVAL >>>>>>>> >> >>>>>>>> >> we can remove deprecations faster in "main" versions of the >>>>>>>> providers - no need to introduce a deprecation policy - the >>>>>>>> stakeholders >>>>>>>> for the providers will take care about cherry-picking and maintaining >>>>>>>> more >>>>>>>> "backwards-compatible" versions. We are free to remove deprecations in >>>>>>>> major releases (in a cherry-pickable way of course). >>>>>>>> >> >>>>>>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>>>>>> >> >>>>>>>> >> We avoid the problem (already happened with the composer >>>>>>>> release) that the stakeholders in a given provider had to release >>>>>>>> their own >>>>>>>> version which was not available in their community - with some >>>>>>>> cherry-picks. We want to avoid "diverging" there - by releasing the >>>>>>>> cherry-picked providers by the community, we also give other users an >>>>>>>> opportunity to follow "slower" deprecation policies for as long as it >>>>>>>> is >>>>>>>> maintained. >>>>>>>> >> >>>>>>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>>>>>> >> >>>>>>>> >> We are going to test a governance model that we might apply when >>>>>>>> we split providers. We are talking about it for quite some time - but >>>>>>>> this >>>>>>>> is what helps us to test the model where stakeholders provide more >>>>>>>> "maintenance" while the community still takes care about releases. We >>>>>>>> (as >>>>>>>> community) can commit to releasing such a version of a provider as >>>>>>>> long as >>>>>>>> the stakeholder will actively maintain it. We can stop at any moment >>>>>>>> if we >>>>>>>> do not have support from the stakeholder. If it works - we can keep it >>>>>>>> as a >>>>>>>> long-term solution. In the future we can think of other scenarios >>>>>>>> (passing >>>>>>>> ownership of a provider to stakeholders who want it - providing we >>>>>>>> want it >>>>>>>> too) but we can decide about it when we learn from the mixed-governance >>>>>>>> model and see if it works. >>>>>>>> >> >>>>>>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>>>>>> >> >>>>>>>> >> If this is an acceptable approach - we can also apply a very >>>>>>>> similar governance model to adding new providers and that should >>>>>>>> unblock >>>>>>>> some of the PRs that are waiting for our decision. Knowing that we are >>>>>>>> going to split and that we can expect "commitment" from a stakeholder, >>>>>>>> we >>>>>>>> should be able to accept new providers. This might be possible assuming >>>>>>>> that the stakeholder will make a similar commitment - but for new >>>>>>>> providers, that commitment might also have to cover reviewing and >>>>>>>> testing >>>>>>>> new changes. We might also decide as a community to stop releasing new >>>>>>>> providers there if such support is missing. This way we can set the >>>>>>>> expectations we have as a community for new providers - we will release >>>>>>>> them as long as the stakeholder will actively make sure it is >>>>>>>> maintained. >>>>>>>> >> >>>>>>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>>>>>> >> >>>>>>>> >> We all know we want to split providers from core. By introducing >>>>>>>> mixed-governance we can test if it will work for the providers before >>>>>>>> we >>>>>>>> split them. It will take some time (and detailed AIP) to split, but in >>>>>>>> the >>>>>>>> meantime we can see if we will be able to apply the mixed-governance >>>>>>>> after >>>>>>>> the split. We will see if we can agree when it comes to expectations >>>>>>>> and >>>>>>>> find solutions before we actually split. >>>>>>>> >> >>>>>>>> >> J. >>>>>>>> >>>>>>>
