Thanks everyone. I proposed a PR that completes the lifecycle description of providers : https://github.com/apache/airflow/pull/35277 - once there are some comments and initial approvals/consensus, I will start a separate `[VOTE]` thread on it. In the meantime I will start `[LAZY CONSENSUS]` on removing Qubole, since there seem to be a general consensus on it already.
On Mon, Oct 30, 2023 at 2:49 PM Josh Fell <josh.d.f...@astronomer.io.invalid> wrote: > Seems reasonable to me! > > On Mon, Oct 30, 2023 at 7:28 AM Wei Lee <weilee...@gmail.com> wrote: > > > +1 > > > > Best, > > Wei > > > > > On Oct 30, 2023, at 5:36 AM, Utkarsh Sharma < > > utkarsh.sha...@astronomer.io.INVALID> wrote: > > > > > > +1 for the implementing the process, have little clue about the > > provider. :) > > > > > > Thanks, > > > Utkarsh Sharma > > > > > > On Sun, Oct 29, 2023 at 10:59 PM Pierre Jeambrun < > pierrejb...@gmail.com> > > wrote: > > >> > > >> +1 > > >> > > >> Le ven. 27 oct. 2023 à 19:16, Aritra Basu <aritrabasu1...@gmail.com> > a > > >> écrit : > > >> > > >>> Sounds like a good time to set the process up. +1 from me as well. > > >>> > > >>> -- > > >>> Regards, > > >>> Aritra Basu > > >>> > > >>> On Fri, Oct 27, 2023, 6:42 PM Vincent Beck <vincb...@apache.org> > > wrote: > > >>> > > >>>> I like that. I also think it is important to have a process to > remove > > >>>> provider if needed. +1 > > >>>> > > >>>> On 2023/10/27 09:00:25 Jarek Potiuk wrote: > > >>>>>> I think in the case of Qubole it is pretty easy to remove it from > > the > > >>>>> provider codebase. I'm pretty sure that almost no one even noticed > > this > > >>>>> removal. > > >>>>> > > >>>>> Yeah. Agree. This one is pretty "obvious" that's why I would like > to > > >>>> create > > >>>>> a process for doing it along the way so that in the future if we > have > > >>>> other > > >>>>> non-obvious cases we can just "follow the process". > > >>>>> > > >>>>> BTW.That would be really great to have as with it, we will have a > > >>>> complete > > >>>>> "life-cycle" of the providers. > > >>>>> > > >>>>> * we know how we approve the new ones > > >>>>> * we know how we release new versions (Elad is amazing to release > > them > > >>>>> every few weeks) > > >>>>> * we know how we maintain back-compatibility (bumping min-version > of > > >>>>> Airflow that we regularly do ) > > >>>>> * we know how to involve stakeholders to system-test them and to > make > > >>>> them > > >>>>> work in a stable way when they connect to external services (Amazon > > >>>> works, > > >>>>> Google in Progres, Open API and others promised by Astronomer) > > >>>>> * we know how to involve stakeholders with mixed-governance in case > > >>> they > > >>>>> want older releases (never happened yet but we know how to do it) > > >>>>> * we know how to suspend and resume them when they prove to be > > >>>> problematic > > >>>>> and pass resolution of that to external stakeholders (happened with > > >>>> Yandex > > >>>>> - both suspend and resume) > > >>>>> * we (will finally) know how to retire them when we decide we do > not > > >>> want > > >>>>> to maintain them - except security fixes - any more > > >>>>> > > >>>>> That will pretty much complete our process of "life-cycle" > management > > >>> for > > >>>>> providers. > > >>>>> > > >>>>> J. > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Thu, Oct 26, 2023 at 10:00 PM Jarek Potiuk <ja...@potiuk.com> > > >>> wrote: > > >>>>> > > >>>>>> > > >>>>>>> I suggest also removing it from pypi for security reasons. If > there > > >>>> is a > > >>>>>>> security issue with it then the issue will remain with us. > > >>>>>>> > > >>>>>>> > > >>>>>> I am quite sure we still have to handle security issues if someone > > >>>> finds > > >>>>>> them. releasing such a provider will still be possible using the > > >>>> tag/branch > > >>>>>> and we will be obliged to release a new version IF it is still > used > > >>>> and a > > >>>>>> security issue is found. > > >>>>>> Removal of packages from PyPI does not remove our obligation to > fix > > a > > >>>>>> security problem. We have also source packages released via > > >>>>>> downloads.apache.org and archives.apache.org - and those we can't > > >>>> remove > > >>>>>> either. > > >>>>>> > > >>>>>> I think this is one of the obligations of the Foundation being the > > >>>>>> "steward" of software it releases - that's why there is also the > > >>>> attic PMC > > >>>>>> to manage projects that PMC is unable to support (projects are > moved > > >>> to > > >>>>>> attic when they fail a roll call from the board with less than 2 > PMC > > >>>>>> members confirming that they are still there and ready to handle > > >>>> releases > > >>>>>> if needed. It's also being discussed to be more formal for the CRA > > >>>>>> regulations right now - "stewards" of the software put on the > market > > >>>> should > > >>>>>> be responsible for handling security issues in a timely manner). > The > > >>>> act of > > >>>>>> release with 3 +1s of PMC is a legal act of the Foundation placing > > >>>> software > > >>>>>> on the market and we can't make it "unhappen". > > >>>>>> > > >>>>>> > > >>>>>>> B. > > >>>>>>> > > >>>>>>> Sent from my iPhone > > >>>>>>> > > >>>>>>>> On 26 Oct 2023, at 20:20, Jarek Potiuk <ja...@potiuk.com> > wrote: > > >>>>>>>> > > >>>>>>>> Hello Airflow community, > > >>>>>>>> > > >>>>>>>> How do we feel about removing the Qubole provider completely > > >>>> (leaving > > >>>>>>> only > > >>>>>>>> old releases in PyPI? > > >>>>>>>> > > >>>>>>>> On September 1 2023 ( > > >>>>>>>> > https://lists.apache.org/thread/p394d7w7gc7lz61g7qdthl96bc9kprxh) > > >>>> the > > >>>>>>>> Qubole operator ws suspended. > > >>>>>>>> > > >>>>>>>> Due to the reasons described in the thread (Qubole got acquired > > >>> and > > >>>> the > > >>>>>>>> service is generally abandoned) there is pretty much no chance > for > > >>>> it > > >>>>>>> to be > > >>>>>>>> resumed. > > >>>>>>>> > > >>>>>>>> I'd love to remove it completely and introduce a process where > we > > >>>> can do > > >>>>>>>> similar things in the future for other providers if we decide to > > >>> do > > >>>> so. > > >>>>>>>> > > >>>>>>>> I checked in the Attic project in the ASF (this is where > abandoned > > >>>>>>> project > > >>>>>>>> of the ASF get moved to) and it seems that just removing part of > > >>> the > > >>>>>>>> project that has an active PMC is not going through attic > > >>>>>>>> https://issues.apache.org/jira/browse/ATTIC-218 . We are free > to > > >>>>>>> define our > > >>>>>>>> rules for that and I would like to use the opportunity to hash > it > > >>>> out > > >>>>>>> and > > >>>>>>>> propose a process (similarly to suspension) and criteria to > remove > > >>>>>>>> providers from being maintained by us. > > >>>>>>>> > > >>>>>>>> It's more than suspension. We will completely stop updating the > > >>>> related > > >>>>>>>> code (right now some automated changes can still be applied and > > >>>>>>> suspended > > >>>>>>>> providers can be resumed with simple PR). I would like to have > the > > >>>>>>> "next" > > >>>>>>>> step after "suspending" - removal. > > >>>>>>>> > > >>>>>>>> Roughly - we send PROPOSAL followed by VOTE (or immediately > VOTE > > >>> in > > >>>>>>>> obvious cases) with justification, PMC members only have the > > >>> binding > > >>>>>>> votes > > >>>>>>>> (similar as for releases). > > >>>>>>>> > > >>>>>>>> Only git history will remain - all the rest will be removed > > >>>> (including > > >>>>>>>> extra) - no traces of the provider remain in the next MINOR > > >>> release > > >>>>>>> (2.8.0 > > >>>>>>>> in the case of Quibole). The provider will still be in PyPI and > > >>>>>>> historical > > >>>>>>>> releases will be in https://archive.apache.org . If someone > would > > >>>> like > > >>>>>>> to > > >>>>>>>> bring back such a provider, It should go through the same > process > > >>>> as a > > >>>>>>> new > > >>>>>>>> provider (voting/consensus). And we might reject it. > > >>>>>>>> > > >>>>>>>> WDYT? Any comments for such an approach / process ? > > >>>>>>>> > > >>>>>>>> J. > > >>>>>>> > > >>>>>>> > > >>> --------------------------------------------------------------------- > > >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >>>> > --------------------------------------------------------------------- > > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>>> > > >>>> > > >>> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
