Sure you can improve visibility there, but also you have to be mindful that we (the Apache Airflow PMC) are not allowed to endorse or promote any non-community managed stuff - 3rd-party code etc. on our website. The website, release process, management, decision making and responsibility (and the last one is the main reason why we often hesitate to accept code - because code = responsibility = potential liability) .
This is very important limitation - we have to avoid the confusion with our users and either endorse 3rd-parties or confuse the users that the software that is released by someone else belongs and is under 100% control of the Project Management Commitee - our official website should never mix the "Apache Software Foundation released software" with "Other / 3rd-party released software" - our users should trust that the software we release is managed in a vendor-neutral way, and released according to the standards and processes that the ASF mandates and encourages - and by definition, we cannot do it for a 3rd-party software managed by others (and we have absolutely no desire to even want to do it - we are all volunteers and we want to spend our time on releasing the ASF software in the "Apache Way" - and we do not forbid others to do whatever they want with their software, we even encourage them to do so, but since we have no control over it, we cannot and should not promote or encourage it. You can learn more how the ASF and PMCs work here https://www.apache.org/foundation/how-it-works/#management - and there is a good deal there about individual members of PMC who are the only ones who can make decision, vendor neutrality, decision making process, branding, promoting others etc. (with a lot of information linked from that page). This is why the "Ecosystem" page is a separate one https://airflow.apache.org/ecosystem/ from all other pages in our site, And it has this preamble: > These resources and services are not maintained, nor endorsed by the Apache Airflow® Community and Apache Airflow project (maintained by the Committers and the Airflow PMC). Use them at your sole discretion. The community does not verify the licences nor validity of those tools, so it’s your responsibility to verify them. > If you would you like to be included on this page, please reach out to the Apache Airflow dev or user mailing list and let us know or simply open a Pull Request to that page. This is a very deliberate decision to structure it in this way and this is the way how the ASF recommends their PMC to link to "ecosystem" of 3rd-party software connected to the project that they have no control over - anyone (including you) can make a change to that page, and provide proposals how to change and improve it, you do not need our permission to change the content - as long as you do it in the way that you keep those properties: * it's a separate page (prefferably only one) that clearly states that this is "3rd-party/ecosystem" and that PMC neither endorses, nor verifies what is linked there * there should be no assesment, evaluation, promotion of any kind and especially any kinds of comparisons assessments, evaluations etc. should not be there * anyone in the community can propose a change and barring changing those properties we accept any submissions there without any real evaluation of whether it is "good", "recommended", "verified" - generally you can assume we have not visited the page (other than then checking if it is not illegal content to link to) - and we do not keep it updated, check if they still point to valid sites etc. - however anyone from the community is free to make PR fixing any of such issues - basically that page is "outsourced" for the general community to manage (within the boundaries of not being confused with anything "recommended" or "official") So absolutely - if you - as anyone else in the community - wish to improve the structure of that page (within the boundaries above) - you are absolutely free to propose it. J. On Thu, May 15, 2025 at 3:22 AM Shyam Rajamannar <shyam.ven...@couchbase.com.invalid> wrote: > Hi Jarek, Vikram, and the Airflow community, > > Thank you for the thoughtful feedback and valuable discussion. > > We understand and respect the perspective shared regarding maintaining the > Couchbase provider outside the main Airflow repository—it makes sense, and > we appreciate the clarity. > > However, one of the key motivations behind our original proposal was to > ensure easier discoverability of the Couchbase provider—especially for new > users evaluating Airflow. Even existing users sometimes miss the ecosystem > page, whereas the official providers listing page is often the first place > users consult for integrations. Being listed there can significantly > improve adoption and awareness. > > As a constructive next step, we’d like to propose contributing a change to > the Airflow documentation or website to highlight external providers more > prominently—ideally by including them in the official providers page or > linking to them from a dedicated section. > > This approach keeps the Couchbase provider decoupled, while still ensuring > the visibility it—and other external providers—need within the broader > Airflow ecosystem. > > If the community is open to this idea, we’d be happy to prepare and submit > a PR for review. > > Looking forward to your thoughts. > > > Regards, > Shyam Venkat > > From: Jarek Potiuk <ja...@potiuk.com> > Date: Friday, 18 April 2025 at 6:02 PM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache > Airflow > > 1. Visibility is one of the motivations, but not the only one. Our > main goal is to ensure a better experience for users who are already using > Couchbase with Airflow, and to support future users with a provider that is > more easily discoverable, better integrated with Airflow’s ecosystem, and > aligned with its standards and lifecycle. > > Having providers outside of our codebase is perfectly fine with our > standards and alignments. That's a valid way to release your provider. Many > other parties are doing it. > > 1. We fully understand that becoming an official provider comes with > > reduced flexibility, such as adhering to Airflow’s version > requirements—but > > we believe this is a reasonable trade-off, as it simplifies compatibility > > and aligns us more closely with the Airflow ecosystem. The provider > > currently only exposes cluster, scope, and collection methods, along with > > basic CRUD operations. Changes to these interfaces are very rare, so we > > expect the provider to remain stable with minimal maintenance overhead. > > > > > Then you should not need to upgrade it as well. Just released provider > from your repo should work in this case and the only time change is needed, > you could release it then. I do not see what being part of regular releases > would bring other than more effort on the community side and less effort on > your side. > > > > > > 1. Being part of the official Airflow repo would ensure the Couchbase > > provider is automatically tested with new Airflow releases, helping us > > catch issues early and stay compatible. > > > > Not really. We are not testing it with the real "couchbase" system when we > release it. We only run unit tests in Airflow and you can do the same > every time we release a new airflow version . The tests with couchbase > service (which to be honest is far more likely to change than internals of > airflow providers) is something only you can do having access to couchbase > account and service. We have none and we do not really want to bother with > testing the provider when anything changes on the couchbase side - such as > when new library is released for couchbase, when Couchbase changes the APIs > or modifies anything, this is is only done for some providers by system > tests > > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > run by some of our stakeholders who agreed to build and run such dashboards > and tests. And we are not running those tests. This will not change if the > code is moved to the Airflow codebase. > > I think it would really be great if you answer the specific questions from > Vikram - about metrics and examples of your visibility. > > > > > > Regards, > > Shyam Venkat > > > > From: Vikram Koka <vik...@astronomer.io.INVALID> > > Date: Friday, 11 April 2025 at 11:49 PM > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > Subject: Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache > > Airflow > > Shyam, > > > > First off, I'm really glad to hear that Couchbase is interested in a > deeper > > integration with Airflow and that Couchbase users are already finding > value > > in it. > > > > Just to follow up with a couple of specific questions around your > request: > > Could you share more about what's missing today with the integration > being > > maintained in the Couchbase repo, versus having it packaged within the > > Airflow repo? > > > > You mentioned benefits like "better visibility, easier adoption, and > > tighter integration with Airflow's architecture and release processes." > > If you could expand on those - especially in terms of what you see > changing > > before vs. after, and ideally any metrics or examples - that context > would > > be very helpful. > > > > Best regards, > > Vikram > > > > > > On Fri, Apr 11, 2025 at 3:02 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > I am not convinced. > > > > > > First of all what do you mean by visibility ? I see no problem with the > > > visibility of your current provider - couchbase seems to have well > > > established ecosystem with multiple plugins and extensions - > > > https://github.com/Couchbase-Ecosystem and I would say it's quite > likely > > > Couchbase users will be looking for Airflow integration among those. > Also > > > you will have (as you have now) freedom to release whatever changes and > > > improvements needed as Couchbase evolves and if you already commit to > > > maintain and keep the couchbase integration by Couchbase team (you > wrote > > it > > > above) - you can equally well maintain it in and release it in your > repo. > > > > > > You already seem to be a good steward of that integration and by > keeping > > > your ecosystem healthy you do a good job, I do not see a particular > > reason > > > why you'd want to contribute it to Airflow. We have no knowledge of > > > Couchbase itself here, you are the experts and you can decide what is > the > > > best integration level you provide, the set of operators and hooks are > > best > > > for you - and I think it's better if you listen to the Couchbase > > community > > > feedback and get your community to contribute there - Airflow provides > a > > > nice set of interfaces for you to adhere, and that's basically all that > > > (from Airflow Community point of view) you need to know and follow - > and > > > you already do it apparently. > > > > > > Couchbase is "source-available" database > > > https://www.couchbase.com/blog/couchbase-adopts-bsl-license/ - which > is > > > not > > > an OSI compliant open-source produict, which makes it even more > suitable > > > for your company to keep maintaining the integration. Of course > > eventually > > > it's a community decision whether to accept such a provider or not, > but I > > > would be rather against it - just for that reason. > > > > > > J. > > > > > > > > > > > > On Fri, Apr 11, 2025 at 11:17 AM Shyam Rajamannar > > > <shyam.ven...@couchbase.com.invalid> wrote: > > > > > > > Couchbase has customers already using Apache Airflow, and having an > > > > officially supported provider within the ecosystem would simplify > > > > integration, improve reliability, and encourage adoption. > > > > > > > > We previously released airflow-providers-couchbase< > > > > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase > ><< > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase%3e%3c> > > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase%3e>< > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase%3e%3e> > > > > through an external repository also listed under airflow ecosystem. > > > > However, it has been less visible to our customers and the broader > > > Airflow > > > > community. Contributing this provider directly to the official > Airflow > > > > ecosystem will ensure better visibility, easier adoption, and tighter > > > > integration with Airflow’s architecture and release processes. > > > > > > > > As part of the initial implementation, we have created a Couchbase > hook > > > > that exposes the Couchbase client, scope, and collection, allowing > > users > > > to > > > > easily interact with Couchbase within Airflow tasks. The provider > also > > > > includes functionality to customize connection field behaviour, > > adhering > > > to > > > > Airflow’s connection management standards. > > > > > > > > Our initial scope is deliberately minimal—focused on foundational > > > > operations. While we may introduce CRUD functionality later, we > expect > > > the > > > > provider to remain stable with minimal and infrequent changes. > > > > > > > > We are committed to maintaining the provider, ensuring compatibility > > with > > > > future Couchbase API changes, and responding to community > > > feedback—similar > > > > to the stewardship shown by other officially supported database > > > providers. > > > > > > > > If maintenance is a concern, we are fully willing to act as active > > > > maintainers, ensuring the provider remains well-supported, reliable, > > and > > > > aligned with Airflow’s evolution. > > > > > > > > Regards, > > > > Shyam Venkat > > > > > > > > > > > > > > > > > >