Oh I hadn't checked that PR, I will take a look some point tonight/tomorrow
On Thu, 30 Mar 2023 at 14:33, Beck, Vincent <[email protected]> wrote: > Big yes to "WDYT about documenting those policies in this way?". That > would finally set the rules and process to add a new provider. It will make > it clearer for anyone who's thinking of creating a new provider and > potentially avoid question we already had in the past in this email list > (even though every question is welcome). > > On 2023-03-30, 7:42 AM, "Jarek Potiuk" <[email protected] <mailto: > [email protected]>> wrote: > > > > Yep. I deliberately do not want to discuss (yet) a policy about > "removing" providers. As discussed in this comment on > https://github.com/apache/airflow/pull/30359#discussion_r1152808160 < > https://github.com/apache/airflow/pull/30359#discussion_r1152808160> > - attic is the way to go, and when we get there to remove some of the > providers, we should formalize it :) > > > Note "suspended" is not "removed" - it is quite far from one to the > other I think. > > > J. > > > On Thu, Mar 30, 2023 at 2:23 PM Kaxil Naik <[email protected] <mailto: > [email protected]>> wrote: > > > > We need some version of attic <https://attic.apache.org/> < > https://attic.apache.org/>> when retiring > > providers that aren't maintained. > > > > On Thu, 30 Mar 2023 at 12:52, Kaxil Naik <[email protected] <mailto: > [email protected]>> wrote: > > > > > I think this is a good starting point, we can iterate over it, over > time. > > > > > > * WDYT about documenting those policies in this way? > > > > > > Yes, absolutely > > > > > > * should we expect Integration tests from kafka provider? > > > > > > I am leaning towards YES. It would have been a strong YES if we had > > > created separate provider repo but my only fear here is the explosion > of > > > provider versions (Kafka, Trino versions) that we would have to run on > > > main, adding to our already long CI jobs. Selective checks should help > it > > > keep lightweight on the PRs though. > > > > > > Regards, > > > Kaxil > > > > > > On Thu, 30 Mar 2023 at 09:31, Jarek Potiuk <[email protected] <mailto: > [email protected]>> wrote: > > > > > >> Hello everyone, > > >> > > >> While discussing the "suspension" I realised that we don't have a > > >> formally described process of what is needed to approve new providers. > > >> TL;DR; I would like to describe it more formally in our policies > > >> > > >> We have discussed it with the mixed governance model for "service" > > >> providers and in numerous emails, but we do not have a clear > > >> description about "how to get a new provider in". > > >> > > >> I believe the general consensus is that: > > >> > > >> * for new providers that have a strong "backing" - in the form of a > > >> 3rd-party (service usually) that is mostly interested in managing and > > >> supporting such a provider and has the capabilities to take the > > >> maintenance of the provider, the strong preference is that provider is > > >> released outside of the community - by the service provider. and then > > >> they are free to mention their provider in various registries and in > > >> our https://airflow.apache.org/ecosystem/ < > https://airflow.apache.org/ecosystem/> page. I think we should > > >> strongly state that this is the default mode and very strong > > >> preference. > > >> > > >> * if the service provider has really good reasons and can convince us > > >> to take on the maintenance (think the recent "Huawei Cloud" discussion > > >> https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc < > https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc> ), > > >> the minimum requirement (on top of the regular code review rules/unit > > >> testing/documentation etc.) is that the provider adds system tests > > >> covering the service and provide the dashboard similar to the AWS one > > >> > > >> > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > < > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > > > > >> . > > >> This will allow the release manager to make a decision on release > > >> viability, additionally (after the lazy consensus is reached) we > > >> reserve the right of suspending such service provider (assuming lazy > > >> consensus/vote of the community) if such provider is holding us back. > > >> > > >> * for providers that are referring to non-services (mostly open source > > >> integration) providers, I think we should agree on being much more > > >> open here. A good example is Apache Kafka provider > > >> https://github.com/apache/airflow/pull/30175/files < > https://github.com/apache/airflow/pull/30175/files> that Dylan works > > >> on. I am generally very receptive to having Kafka providers in > > >> (especially if we agree on suspension rules - we will have a way to > > >> manage problematic providers this way). One question I am not so sure > > >> (but I am inclined to say "yes") - do we require integration tests > > >> for such providers - we already have mongo, pinot. trino, cassandra, > > >> celery and kerberos and they are now nicely separated out > > >> > https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089 > <https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089 > > > > >> . > > >> They are a bit more involved, but we have good mechanisms that make it > > >> relatively easy to plug in new integration tests. > > >> > > >> > > >> In either case - I think any time a new provider is added (regardless > > >> if it is a service one or not), LAZY CONSENSUS/VOTE should be cast on > > >> the devlist. It is not enough to just approve a PR IMHO. > > >> > > >> * WDYT about documenting those policies in this way? > > >> * should we expect Integration tests from kafka provider? > > >> > > >> I think if we explicitly spell it out it would make it easier for > > >> anyone (like Huawei Cloud case) to understand the options they have > > >> and how much investment they need to make upfront, rather than finding > > >> out after making some investment, that they are expected to do much > > >> more. > > >> > > >> J. > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] <mailto: > [email protected]> > > >> For additional commands, e-mail: [email protected] <mailto: > [email protected]> > > >> > > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] <mailto: > [email protected]> > For additional commands, e-mail: [email protected] <mailto: > [email protected]> > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
