We need some version of attic <https://attic.apache.org/> when retiring providers that aren't maintained.
On Thu, 30 Mar 2023 at 12:52, Kaxil Naik <[email protected]> wrote: > I think this is a good starting point, we can iterate over it, over time. > > * WDYT about documenting those policies in this way? > > Yes, absolutely > > * should we expect Integration tests from kafka provider? > > I am leaning towards YES. It would have been a strong YES if we had > created separate provider repo but my only fear here is the explosion of > provider versions (Kafka, Trino versions) that we would have to run on > main, adding to our already long CI jobs. Selective checks should help it > keep lightweight on the PRs though. > > Regards, > Kaxil > > On Thu, 30 Mar 2023 at 09:31, Jarek Potiuk <[email protected]> wrote: > >> Hello everyone, >> >> While discussing the "suspension" I realised that we don't have a >> formally described process of what is needed to approve new providers. >> TL;DR; I would like to describe it more formally in our policies >> >> We have discussed it with the mixed governance model for "service" >> providers and in numerous emails, but we do not have a clear >> description about "how to get a new provider in". >> >> I believe the general consensus is that: >> >> * for new providers that have a strong "backing" - in the form of a >> 3rd-party (service usually) that is mostly interested in managing and >> supporting such a provider and has the capabilities to take the >> maintenance of the provider, the strong preference is that provider is >> released outside of the community - by the service provider. and then >> they are free to mention their provider in various registries and in >> our https://airflow.apache.org/ecosystem/ page. I think we should >> strongly state that this is the default mode and very strong >> preference. >> >> * if the service provider has really good reasons and can convince us >> to take on the maintenance (think the recent "Huawei Cloud" discussion >> https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc ), >> the minimum requirement (on top of the regular code review rules/unit >> testing/documentation etc.) is that the provider adds system tests >> covering the service and provide the dashboard similar to the AWS one >> >> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards >> . >> This will allow the release manager to make a decision on release >> viability, additionally (after the lazy consensus is reached) we >> reserve the right of suspending such service provider (assuming lazy >> consensus/vote of the community) if such provider is holding us back. >> >> * for providers that are referring to non-services (mostly open source >> integration) providers, I think we should agree on being much more >> open here. A good example is Apache Kafka provider >> https://github.com/apache/airflow/pull/30175/files that Dylan works >> on. I am generally very receptive to having Kafka providers in >> (especially if we agree on suspension rules - we will have a way to >> manage problematic providers this way). One question I am not so sure >> (but I am inclined to say "yes") - do we require integration tests >> for such providers - we already have mongo, pinot. trino, cassandra, >> celery and kerberos and they are now nicely separated out >> https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089 >> . >> They are a bit more involved, but we have good mechanisms that make it >> relatively easy to plug in new integration tests. >> >> >> In either case - I think any time a new provider is added (regardless >> if it is a service one or not), LAZY CONSENSUS/VOTE should be cast on >> the devlist. It is not enough to just approve a PR IMHO. >> >> * WDYT about documenting those policies in this way? >> * should we expect Integration tests from kafka provider? >> >> I think if we explicitly spell it out it would make it easier for >> anyone (like Huawei Cloud case) to understand the options they have >> and how much investment they need to make upfront, rather than finding >> out after making some investment, that they are expected to do much >> more. >> >> J. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
