This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new eeee433eed6 Split PROVIDERS.rst into focused documents under
providers/ (#63305)
eeee433eed6 is described below
commit eeee433eed6305e47af395d3398fdddf00ceb0c7
Author: Jarek Potiuk <[email protected]>
AuthorDate: Tue Mar 10 23:41:45 2026 +0100
Split PROVIDERS.rst into focused documents under providers/ (#63305)
The monolithic PROVIDERS.rst was difficult to navigate. Split it into
focused documents while keeping PROVIDERS.rst as a hub with stub
sections that preserve all existing anchor links from changelogs.
New documents:
- providers/PROVIDER_GOVERNANCE.rst - governance framework, stewardship,
lifecycle stages, health metrics, periodic reviews
- providers/ACCEPTING_PROVIDERS.rst - prerequisites, approval process,
historical examples
- providers/PROVIDER_RELEASES.rst - release process, versioning, states,
min Airflow version policy
- providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst - suspension and
removal criteria and process
- providers/THIRD_PARTY_PROVIDERS.rst - 3rd-party providers, system test
dashboards, mixed governance model
Also restructured MANAGING_PROVIDERS_LIFECYCLE.rst with consistent
heading hierarchy and logical section grouping.
Co-authored-by: Claude Opus 4.6 <[email protected]>
---
PROVIDERS.rst | 517 +++-------------------
providers/ACCEPTING_PROVIDERS.rst | 81 ++++
providers/MANAGING_PROVIDERS_LIFECYCLE.rst | 543 ++++++++++++------------
providers/PROVIDER_GOVERNANCE.rst | 215 ++++++++++
providers/PROVIDER_RELEASES.rst | 103 +++++
providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst | 110 +++++
providers/THIRD_PARTY_PROVIDERS.rst | 54 +++
7 files changed, 898 insertions(+), 725 deletions(-)
diff --git a/PROVIDERS.rst b/PROVIDERS.rst
index a48233eb3e8..2caa00273ba 100644
--- a/PROVIDERS.rst
+++ b/PROVIDERS.rst
@@ -48,6 +48,9 @@ You can read more about it in the
`Installation and upgrade scenarios
<https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios>`_
chapter of our user documentation.
+List of all available community providers is available at the `Providers index
<https://airflow.apache.org/docs/>`_.
+
+
Community managed providers
===========================
@@ -70,10 +73,34 @@ Because of the constraint and potential conflicting
dependencies, the community
updated and the community might decide to suspend releases of a provider if we
find out that we have trouble
with updating the dependencies, or if we find out that the provider is not
compatible with other more
popular providers and when the popular providers are limited by the
constraints of the less popular ones.
-See the section below for more details on suspending releases of the community
providers.
+See `Suspending releases for providers`_ below for more details.
-List of all available community providers is available at the `Providers index
<https://airflow.apache.org/docs/>`_.
+Detailed documents
+==================
+
+The following documents provide detailed information about specific aspects of
provider management:
+
+.. list-table::
+ :widths: 30 70
+
+ * - `Provider Governance <providers/PROVIDER_GOVERNANCE.rst>`_
+ - Governance framework, stewardship model, lifecycle stages (incubation,
production,
+ attic/deprecation), health metrics, and periodic reviews
+ * - `3rd-Party Providers <providers/THIRD_PARTY_PROVIDERS.rst>`_
+ - Relation to community providers, system test dashboards, and mixed
governance model
+ * - `Accepting New Providers <providers/ACCEPTING_PROVIDERS.rst>`_
+ - Prerequisites, approval process, and historical examples for proposing
new community providers
+ * - `Provider Releases and Versioning <providers/PROVIDER_RELEASES.rst>`_
+ - Release process, SEMVER versioning, provider distribution states, and
minimum Airflow
+ version policy
+ * - `Suspending and Removing Providers
<providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst>`_
+ - Criteria and process for suspending or removing community providers
+ * - `Managing Provider's Lifecycle
<providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
+ - Technical how-to for creating, suspending, resuming, and removing
providers
+
+
+.. _provider-governance-framework:
Community providers lifecycle
=============================
@@ -81,482 +108,58 @@ Community providers lifecycle
This document describes the complete life-cycle of community providers - from
inception and approval to
Airflow main branch to being decommissioned and removed from the main branch
in Airflow repository.
-.. note::
-
- Technical details on how to manage lifecycle of providers are described in
the document:
-
- `Managing provider's lifecycle
<https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
-
-
-Provider Governance Framework
------------------------------
-
-This section describes the governance framework for community providers,
including lifecycle stages,
-stewardship model, and quantitative health metrics.
-
-Airflow's success is built on its extensive ecosystem of community-supported
integrations—over 1,600
-hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
-for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
-number of integrations while ensuring quality. Actual code acceptance and
release governance remains with the Airflow PMC.
-
-Provider Stewardship Model
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Each provider or integration component must have designated **stewards**
responsible for ensuring
-the health criteria described below are met:
+For full details, see `Provider Governance
<providers/PROVIDER_GOVERNANCE.rst>`_.
-* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
-* **Role definition**: Stewards are subject matter experts for the
integration. This could be expertise in the
- service being integrated, the language being supported by the provider, or
the framework being integrated.
- Stewardship is a responsibility, not an additional authority or privilege
-* **Committer sponsorship**: Each steward must be sponsored by at least one
Airflow Committer. The
- sponsor ensures that stewards fulfill their responsibilities and provides
guidance on maintaining the
- provider according to best practices. This includes regular dependency
updates, issue resolution, and
- monitoring that the provider meets the health metrics required for its
current lifecycle stage (that is, incubation
- or production). The sponsor is responsible for PR reviews and merges
(including ensuring coding standards are met), but
- is NOT responsible for active maintenance of the provider's codebase, which
remains the responsibility of the stewards.
- While sponsors should be accountable when it comes to reviews and merges,
it's also OK and welcome that other committers merge code providing it fulfill
the criteria.
-* **Accountability**: Ultimate accountability remains with the Airflow PMC and
Committers
-* **Transitions**: Neither sponsorship nor stewardship are roles in
perpetuity; they can be
- transitioned to others based on mutual agreement and PMC approval
+Governance framework
+--------------------
-.. note::
+See `Provider Governance: Governance framework
<providers/PROVIDER_GOVERNANCE.rst#governance-framework>`_.
- The quantitative criteria described below are aspirational. The PMC will
revisit these metrics
- based on actual experience 6 months from the date of the first quarterly
review of the provider metrics being published.
-Provider Lifecycle Stages
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Providers generally follow a three-stage lifecycle: **Incubation**,
**Production**, and **Attic/Deprecation**. However,
-not all providers will move through all stages. Additionally, providers may be
designated as **Mature** under specific circumstances.
-
-**Incubation Stage**
-
-All new providers or integration modules (such as Notifiers, Message
components, etc.) should start
-in incubation unless specifically accelerated by the PMC. Incubation ensures
that code, licenses, and
-processes align with ASF guidelines and community principles such as
community, collaborative
-development, and openness.
-
-Requirements for incubation:
-
-* Working codebase to bootstrap credibility and attract contributions
-* Visibility in the provider health dashboard (The provider health dashboard
is to be added)
-* Designated stewards with committer sponsorship
+Accepting new community providers
+----------------------------------
-Quantitative graduation criteria:
+See `Accepting New Community Providers <providers/ACCEPTING_PROVIDERS.rst>`_.
-.. list-table::
- :header-rows: 1
- :widths: 40 60
-
- * - Metric
- - Threshold
- * - PRs submitted
- - Minimum of 10 PRs in the last 6 months
- * - PR review time
- - PRs reviewed within 14 days
- * - Issues reported
- - Minimum of 15 unique issues filed in the last 6 months
- * - Contributors
- - At least 3 unique individuals making contributions (code or
documentation) in the last 6 months
- * - Issue resolution rate
- - At least 50% of reported issues closed within 90 days
- * - Security/release issues
- - All release and security related issues closed within 60 days
- * - Governance participation
- - Demonstrated participation in project governance channels including
quarterly updates
- * - Quality standards
- - Meet quality criteria for code, documentation, and tests as listed in
the Contributing Guide
-
-**Production Stage**
-
-All modules in production are expected to be well managed with prompt
resolution of issues and
-support for a consistent release cadence (at least monthly, but typically
every 2 weeks when changes
-exist). Production providers must:
-
-* Stay consistent with the main branch
-* Pass tests for main + all supported Airflow versions
-* Follow Airflow support guidelines, staying current with main Airflow releases
-
-Exceptions can be granted based on a PMC/devlist vote (PMC members only having
binding votes) for
-valid and one-off criteria.
-
-Quantitative criteria to maintain production status:
-.. list-table::
- :header-rows: 1
- :widths: 40 60
-
- * - Metric
- - Threshold
- * - PRs submitted
- - Minimum of 10 PRs in the last 6 months
- * - PR review time
- - PRs reviewed within 14 days
- * - Issues reported
- - Minimum of 20 unique issues filed in the last 6 months
- * - Contributors
- - At least 5 unique individuals making contributions (code or
documentation) in the last 6 months
- * - Issue resolution rate
- - At least 60% of reported issues closed within 90 days
- * - Security/release issues
- - All release and security related issues closed within 30 days
- * - Feature releases
- - At least 1 feature release every 6 months
- * - User engagement
- - Maintain support activity with response to questions within 2 weeks on
average
- * - Governance participation
- - Demonstrated participation in project governance channels including
quarterly updates
-
-**Attic / Deprecation Stage**
-
-Modules should be moved into the Attic when relevance wanes, typically
measured by declining
-activity. This commonly occurs when the integrated solution has faded in
popularity and is replaced
-by more modern alternatives.
-
-Movement to the Attic must be communicated on the dev list and voted on by the
PMC. Exceptions can
-be granted based on the vote.
-
-Quantitative criteria triggering attic consideration:
+Releases and versioning
+=======================
-.. list-table::
- :header-rows: 1
- :widths: 40 60
-
- * - Metric
- - Threshold
- * - PRs submitted
- - Fewer than 5 PRs in the last 6 months
- * - PR review time
- - PRs not being reviewed in more than a month
- * - Issues reported
- - Fewer than 10 unique issues filed in the last 6 months
- * - Contributors
- - Fewer than 3 unique individuals making contributions (code or
documentation) in the last 6 months
- * - Issue resolution rate
- - Less than 30% of reported issues closed within 90 days
- * - Security/release issues
- - Release and security related issues not getting closed within 30 days
- * - Feature releases
- - No feature releases in the last 6 months
-
-Consequences of attic status:
-
-* Modules remain readable but do not receive active maintenance
-* Module is not actively tested in "main" in Airflow CI, its dependencies are
not checked for conflicts
- with other main providers, and common refactorings are not applied to it.
-* After a period of at least 6 months in the attic, modules can be chosen for
removal with
- appropriate communication (see `Removing community providers`_ below)
-* It is possible for the provider to be resurrected from the attic as long as
there is confidence that there is a
- clear need for the provider and the (possibly new) stewards are able to
maintain it actively on this go around.
- It should be noted that significant effort may be required to resurrect a
provider from the attic.
-
-**Mature Providers**
-
-Some providers may have very stable interfaces requiring minimal changes on a
regular basis (e.g.,
-HTTP provider integration). At the discretion of the Airflow PMC, certain
providers can be tagged
-as **"mature providers"**, which will not automatically be deprecated and
moved into the attic due
-to lack of activity alone.
-
-Periodic Reviews
-^^^^^^^^^^^^^^^^
-
-The Airflow PMC is responsible for reviewing the health status of integrations
on a **quarterly
-basis** and making decisions such as:
-
-* Graduating providers from incubation to production
-* Moving providers from production to the attic
-* Granting exceptions for specific providers
-
-These discussions will be held in public and subsequently summarized and
shared on the Airflow devlist.
+See `Provider Releases and Versioning <providers/PROVIDER_RELEASES.rst>`_.
+Release process
+---------------
-Accepting new community providers
----------------------------------
+See `Provider Releases: Release process
<providers/PROVIDER_RELEASES.rst#release-process>`_.
-The Airflow community welcomes new provider contributions. All new providers
enter through the
-**Incubation** stage (unless specifically accelerated by the PMC) to ensure
alignment with ASF
-guidelines and community principles.
+Versioning scheme
+-----------------
-**Prerequisites for proposing a new provider:**
+See `Provider Releases: Versioning scheme
<providers/PROVIDER_RELEASES.rst#versioning-scheme>`_.
-1. **Working codebase**: A functional implementation demonstrating the
integration
-2. **Stewardship commitment**: At least two individuals willing to serve as
stewards
-3. **Committer sponsorship**: At least one existing Airflow Committer willing
to sponsor the stewards
-4. **Quality standards**: Code, tests, and documentation meeting the
Contributing Guide standards
+Provider distribution states
+-----------------------------
-**Approval process:**
+See `Provider Releases: Provider distribution states
<providers/PROVIDER_RELEASES.rst#provider-distribution-states>`_.
-Accepting new community providers requires a ``[DISCUSSION]`` followed by
``[VOTE]`` thread at the
-Airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_.
+Upgrading minimum supported version of Airflow
+-----------------------------------------------
-For integrations with well-established open-source software (Apache Software
Foundation, Linux
-Foundation, or similar organizations with established governance), a ``[LAZY
CONSENSUS]`` process
-may be sufficient, provided the PR includes comprehensive test coverage and
documentation.
+See `Provider Releases: Upgrading minimum supported version of Airflow
<providers/PROVIDER_RELEASES.rst#upgrading-minimum-supported-version-of-airflow>`_.
-The ``[DISCUSSION]`` thread should include:
-* Description of the integration and its value to the Airflow ecosystem
-* Identification of the proposed stewards and their sponsoring committer(s)
-* Commitment to meet incubation health metrics within 6 months
-* Plan for participating in quarterly governance updates
+Suspending releases for providers
+----------------------------------
-The ``[VOTE]`` follows the usual Apache Software Foundation voting rules
concerning
-`Votes on Code Modification
<https://www.apache.org/foundation/voting.html#votes-on-code-modification>`_
+See `Suspending and Removing Providers: Suspending releases
<providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst#suspending-releases-for-providers>`_.
-**Alternative: 3rd-party managed providers**
-For service providers or systems integrators with dedicated teams to manage
their provider and who wish to not participate
-in the Airflow community, we encourage considering 3rd-party management of
providers. The
-`Ecosystem page
<https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers>`_
-provides visibility for 3rd-party providers, and this approach allows service
providers or systems integrators to:
+Removing community providers
+------------------------------
-* Synchronize releases with their service updates
-* Maintain direct control over the integration
-* Support older Airflow versions if needed
-
-There is no difference in technical capabilities between community and
3rd-party providers.
-
-**Historical examples:**
-
-* Huawei Cloud provider - `Discussion
<https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc>`_
-* Cloudera provider - `Discussion
<https://lists.apache.org/thread/2z0lvgj466ksxxrbvofx41qvn03jrwwb>`_, `Vote
<https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm>`_
-
-
-Community providers release process
------------------------------------
-
-The community providers are released regularly (usually every 2 weeks) in
batches consisting of any providers
-that need to be released because they changed since last release. The release
manager decides which providers
-to include and whether some or all providers should be released (see the next
chapter about upgrading the
-minimum version of Airflow for example the case where we release all active
meaning non-suspended providers,
-together in a single batch). Also Release Manager decides on the version bump
of the provider (depending on
-classification, whether there are breaking changes, new features or just bugs
comparing to previous version).
-
-Upgrading Minimum supported version of Airflow
-----------------------------------------------
-
-.. note::
-
- The following policy applies for Airflow 2. It has not yet been finalized
for Airflow 3 and is subject to changes.
-
-One of the important limitations of the Providers released by the community is
that we introduce the limit
-of a minimum supported version of Airflow. The minimum version of Airflow is
the ``MINOR`` version (2.4, 2.5 etc.)
-indicating that the providers might use features that appeared in this
release. The default support timespan
-for the minimum version of Airflow (there could be justified exceptions) is
that we increase the minimum
-Airflow version to the next MINOR release, when 12 months passed since the
first release for the
-MINOR version of Airflow.
-
-For example this means that by default we upgrade the minimum version of
Airflow supported by providers
-to 3.1.0 in the first Provider's release after 20th of May 2026. 20th of May
2025 was the date when the
-first ``PATCHLEVEL`` version of 2.11 (2.11.0) was released and since Airflow
3.0 was released in April 2025,
-we go straight to Airflow 3.1 as minimum supported version of Airflow for
providers in May 2026.
-
-When we increase the minimum Airflow version, this is not a reason to bump
``MAJOR`` version of the providers
-(unless there are other breaking changes in the provider). The reason for that
is that people who use
-older version of Airflow will not be able to use that provider (so it is not a
breaking change for them)
-and for people who are using supported version of Airflow this is not a
breaking change on its own - they
-will be able to use the new version without breaking their workflows. When we
upgraded min-version to
-2.2+, our approach was different but as of 2.3+ upgrade (November 2022) we
only bump ``MINOR`` version of the
-provider when we increase minimum Airflow version.
-
-Increasing the minimum version of the Providers is one of the reasons why
3rd-party provider maintainers
-might want to maintain their own providers - as they can decide to support
older versions of Airflow.
-
-3rd-parties relation to community providers
--------------------------------------------
-
-Providers, can also be maintained and released by 3rd parties (service
providers or systems integrators).
-There is no difference between the community and 3rd party providers - they
have all the same capabilities
-and limitations.
-
-This is especially in case the provider concerns 3rd-party service that has a
team that can manage provider
-on their own, has a rapidly evolving live service, and believe they need a
faster release cycle than the community
-can provide.
-
-Information about such 3rd-party providers are usually published at the
-`Ecosystem: plugins and providers
<https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers>`_
-page of the Airflow website and we encourage the service providers to publish
their providers there. You can also
-find a 3rd-party registries of such providers, that you can use if you search
for existing providers (they
-are also listed at the "Ecosystem" page in the same chapter)
-
-While we already have - historically - a number of 3rd-party service providers
managed by the community,
-most of those services have dedicated teams that keep an eye on the community
providers and not only take
-active part in managing them (see mixed-governance model below), but also
provide a way that we can
-verify whether the provider works with the latest version of the service via
dashboards that show
-status of System Tests for the provider. This allows us to have a high level
of confidence that when we
-release the provider it works with the latest version of the service. System
Tests are part of the Airflow
-code, but they are executed and verified by those 3rd party service teams. We
are working with the 3rd
-party service teams (who are often important stakeholders of the Apache
Airflow project) to add dashboards
-for the historical providers that are managed by the community, and current
set of Dashboards can be also
-found at the
-`Ecosystem: system test dashboards
<https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards>`_
-
-Mixed governance model for 3rd-party related community providers
-----------------------------------------------------------------
-
-Providers are often connected with some stakeholders that are vitally
interested in maintaining backwards
-compatibilities in their integrations (for example cloud providers, or
specific service providers). But,
-we are also bound with the `Apache Software Foundation release policy
<https://www.apache.org/legal/release-policy.html>`_
-which describes who releases, and how to release the ASF software. The
provider's governance model is something we name
-``mixed governance`` - where we follow the release policies, while the burden
of maintaining and testing
-the cherry-picked versions is on those who commit to perform the cherry-picks
and make PRs to older
-branches.
-
-The "mixed governance" (optional, per-provider) means that:
-
-* The Airflow Community and release manager decide when to release those
providers.
- This is fully managed by the community and the usual release-management
process following the
- `Apache Software Foundation release policy
<https://www.apache.org/legal/release-policy.html>`_
-* The contributors (who might or might not be direct stakeholders in the
provider) will carry the burden
- of cherry-picking and testing the older versions of providers.
-* There is no "selection" and acceptance process to determine which version of
the provider is released.
- It is determined by the actions of contributors raising the PR with
cherry-picked changes and it follows
- the usual PR review process where maintainer approves (or not) and merges
(or not) such PR. Simply
- speaking - the completed action of cherry-picking and testing the older
version of the provider make
- it eligible to be released. Unless there is someone who volunteers and
perform the cherry-picking and
- testing, the provider is not released.
-* Branches to raise PR against are created when a contributor commits to
perform the cherry-picking
- (as a comment in PR to cherry-pick for example)
-
-Usually, community effort is focused on the most recent version of each
provider. The community approach is
-that we should rather aggressively remove deprecations in "major" versions of
the providers - whenever
-there is an opportunity to increase major version of a provider, we attempt to
remove all deprecations.
-However, sometimes there is a contributor (who might or might not represent
stakeholder),
-willing to make their effort on cherry-picking and testing the non-breaking
changes to a selected,
-previous major branch of the provider. This results in releasing at most two
versions of a
-provider at a time:
-
-* potentially breaking "latest" major version
-* selected past major version with non-breaking changes applied by the
contributor
-
-Cherry-picking such changes follows the same process for releasing Airflow
-patch-level releases for a previous minor Airflow version. Usually such
cherry-picking is done when
-there is an important bugfix and the latest version contains breaking changes
that are not
-coupled with the bugfix. Releasing them together in the latest version of the
provider effectively couples
-them, and therefore they're released separately. The cherry-picked changes
have to be merged by the committer following the usual rules of the
-community.
-
-There is no obligation to cherry-pick and release older versions of the
providers.
-The community continues to release such older versions of the providers for as
long as there is an effort
-of the contributors to perform the cherry-picks and carry-on testing of the
older provider version.
-
-The availability of stakeholder that can manage "service-oriented" maintenance
and agrees to such a
-responsibility, will also drive our willingness to accept future, new
providers to become community managed.
+See `Suspending and Removing Providers: Removing providers
<providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst#removing-community-providers>`_.
-Suspending releases for providers
----------------------------------
-
-In case a provider is found to require old dependencies that are not
compatible with upcoming versions of
-the Apache Airflow or with newer dependencies required by other providers, the
provider's release
-process can be suspended.
-
-This means:
-
-* The provider's state in ``provider.yaml`` is set to "suspended"
-* No new releases of the provider will be made until the problem with
dependencies is solved
-* Sources of the provider remain in the repository for now (in the future we
might add process to remove them)
-* No new changes will be accepted for the provider (other than the ones that
fix the dependencies)
-* The provider will be removed from the list of Apache Airflow extras in the
next Airflow release
- (including patch-level release if it is possible/easy to cherry-pick the
suspension change)
-* Tests of the provider will not be run on our CI (in main branch)
-* Dependencies of the provider will not be installed in our main branch CI
image nor included in constraints
-* We can still decide to apply security fixes to released providers - by
adding fixes to the main branch
- but cherry-picking, testing and releasing them in the patch-level branch of
the provider similar to the
- mixed governance model described above.
-
-The suspension may be triggered by any committer after the following criteria
are met:
-
-* The maintainers of dependencies of the provider are notified about the issue
and are given a reasonable
- time to resolve it (at least 1 week)
-* Other options to resolve the issue have been exhausted and there are good
reasons for upgrading
- the old dependencies in question
-* Explanation why we need to suspend the provider is stated in a public
discussion in the devlist. Followed
- by ``[LAZY CONSENSUS]`` or ``[VOTE]`` discussion at the devlist (with the
majority of the binding votes
- agreeing that we should suspend the provider)
-
-The suspension will be lifted when the dependencies of the provider are made
compatible with the Apache
-Airflow and with other providers - by merging a PR that removes the suspension
and succeeds.
+3rd-party providers
+===================
-Removing community providers
-----------------------------
-
-After a Provider has been deprecated, as described above with a ``[VOTE]``
thread, it can
-be removed from main branch of Airflow when the community agrees that there
should be no
-more updates to the providers done by the community - except maybe potentially
security fixes found. There
-might be various reasons for the providers to be removed:
-
-* the service they connect to is no longer available
-* the dependencies for the provider are not maintained anymore and there is no
viable alternative
-* there is another, more popular provider that supersedes community provider
-* etc. etc.
-
-Generally speaking a discussion thread ``[DISCUSS]`` is advised before such
removal and
-sufficient time should pass (at least a week) to give a chance for community
members to express their
-opinion on the removal.
-
-There are the following consequences (or lack of them) of removing the
provider:
-
-* One last release of the provider is done with documentation updated
informing that the provider is no
- longer maintained by the Apache Airflow community - linking to this page.
This information should also
- find its way to the package documentation and consequently - to the
description of the package in PyPI.
-* An ``[ANNOUNCE]`` thread is sent to the devlist and user list announcing
removal of the provider
-* The released providers remain available on PyPI and in the
- `Archives <https://archive.apache.org/dist/airflow/providers/>`_ of the
Apache
- Software Foundation, while they are removed from the
- `Downloads <https://downloads.apache.org/airflow/providers/>`_ .
- Also it remains in the Index of the Apache Airflow Providers documentation
at
- `Airflow Documentation <https://airflow.apache.org/docs/>`_ with note
``(not maintained)`` next to it.
-* The code of the provider is removed from ``main`` branch of the Apache
Airflow repository - including
- the tests and documentation. It is no longer built in CI and dependencies of
the provider no longer
- contribute to the CI image/constraints of Apache Airflow for development and
future ``MINOR`` release.
-* The provider is removed from the list of Apache Airflow extras in the next
``MINOR`` Airflow release
-* The dependencies of the provider are removed from the constraints of the
Apache Airflow
- (and the constraints are updated in the next ``MINOR`` release of Airflow)
-* In case of confirmed security issues that need fixing that are reported to
the provider after it has been
- removed, there are two options:
- * in case there is a viable alternative or in case the provider is anyhow
not useful to be installed, we
- might issue advisory to the users to remove the provider (and use
alternatives if applicable)
- * in case the users might still need the provider, we still might decide to
release new version of the
- provider with security issue fixed, starting from the source code in Git
history where the provider was
- last released. This however, should only be done in case there are no
viable alternatives for the users.
-* Removed provider might be re-instated as maintained provider, but it needs
to go through the regular process
- of accepting new provider described above.
-
-Provider distributions versioning
----------------------------------
-
-We are using the `SEMVER <https://semver.org/>`_ versioning scheme for the
Provider distributions. This is in order
-to give the users confidence about maintaining backwards compatibility in the
new releases of those
-packages.
-
-Details about maintaining the SEMVER version are going to be discussed and
implemented in
-`the related issue <https://github.com/apache/airflow/issues/11425>`_
-
-Possible states of Provider distributions
------------------------------------------
-
-The Provider distributions can be in one of several states.
-
-* The ``not-ready`` state is used when the provider has some in-progress
changes (usually API changes) that
- we do not want to release yet as part of the regular release cycle.
Providers in this state are excluded
- from being released as part of the regular release cycle (including
documentation building).
- The ``not-ready`` providers are treated as regular providers when it comes
to running tests and preparing
- and releasing packages in ``CI`` - as we want to make sure they are properly
releasable any time and we
- want them to contribute to dependencies and we want to test them. Also in
case of preinstalled providers,
- the ``not-ready`` providers are contributing their dependencies rather than
the provider package to
- requirements of Airflow.
-* The ``ready`` state is the usual state of the provider that is released in
the regular release cycle
- (including the documentation, package building and publishing). This is the
state most providers are in.
-* The ``suspended``` state is used when we have a good reason to suspend such
provider, following the devlist
- discussion and vote or "lazy consensus". The process of suspension is
described above.
- The ``suspended`` providers are excluded from being released as part of the
regular release cycle (including
- documentation building) but also they do not contribute dependencies to the
CI image and their tests are
- not run in CI process. The ``suspended`` providers are not released as part
of the regular release cycle.
-* The ``removed`` state is a temporary state after the provider has been voted
(or agreed in "lazy consensus")
- to be removed and it is only used for exactly one release cycle - in order
to produce the final version of
- the package - identical to the previous version with the exception of the
removal notice. The process
- of removal is described in [Provider's docs](../PROVIDERS.rst). The
difference between ``suspended``
- and ``removed`` providers is that additional information is added to their
documentation about the provider
- not being maintained any more by the community.
+See `3rd-Party Providers <providers/THIRD_PARTY_PROVIDERS.rst>`_.
diff --git a/providers/ACCEPTING_PROVIDERS.rst
b/providers/ACCEPTING_PROVIDERS.rst
new file mode 100644
index 00000000000..ca69f1ea11f
--- /dev/null
+++ b/providers/ACCEPTING_PROVIDERS.rst
@@ -0,0 +1,81 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+
+Accepting New Community Providers
+==================================
+
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
+
+The Airflow community welcomes new provider contributions. All new providers
enter through the
+**Incubation** stage (unless specifically accelerated by the PMC) to ensure
alignment with ASF
+guidelines and community principles.
+
+For details on the incubation stage and graduation criteria, see
+`Provider Governance <PROVIDER_GOVERNANCE.rst#incubation-stage>`_.
+
+Prerequisites for proposing a new provider
+------------------------------------------
+
+1. **Working codebase**: A functional implementation demonstrating the
integration
+2. **Stewardship commitment**: At least two individuals willing to serve as
stewards
+3. **Committer sponsorship**: At least one existing Airflow Committer willing
to sponsor the stewards
+4. **Quality standards**: Code, tests, and documentation meeting the
Contributing Guide standards
+
+Approval process
+-----------------
+
+Accepting new community providers requires a ``[DISCUSSION]`` followed by
``[VOTE]`` thread at the
+Airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_.
+
+For integrations with well-established open-source software (Apache Software
Foundation, Linux
+Foundation, or similar organizations with established governance), a ``[LAZY
CONSENSUS]`` process
+may be sufficient, provided the PR includes comprehensive test coverage and
documentation.
+
+The ``[DISCUSSION]`` thread should include:
+
+* Description of the integration and its value to the Airflow ecosystem
+* Identification of the proposed stewards and their sponsoring committer(s)
+* Commitment to meet incubation health metrics within 6 months
+* Plan for participating in quarterly governance updates
+
+The ``[VOTE]`` follows the usual Apache Software Foundation voting rules
concerning
+`Votes on Code Modification
<https://www.apache.org/foundation/voting.html#votes-on-code-modification>`_
+
+Alternative: 3rd-party managed providers
+-----------------------------------------
+
+For service providers or systems integrators with dedicated teams to manage
their provider and who wish to not participate
+in the Airflow community, we encourage considering 3rd-party management of
providers. The
+`Ecosystem page
<https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers>`_
+provides visibility for 3rd-party providers, and this approach allows service
providers or systems integrators to:
+
+* Synchronize releases with their service updates
+* Maintain direct control over the integration
+* Support older Airflow versions if needed
+
+There is no difference in technical capabilities between community and
3rd-party providers.
+See `3rd-party providers <THIRD_PARTY_PROVIDERS.rst>`_ for more details.
+
+Historical examples
+--------------------
+
+* Huawei Cloud provider - `Discussion
<https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc>`_
+* Cloudera provider - `Discussion
<https://lists.apache.org/thread/2z0lvgj466ksxxrbvofx41qvn03jrwwb>`_, `Vote
<https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm>`_
diff --git a/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
b/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
index e56c98537b0..2dcb5a70c89 100644
--- a/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
+++ b/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
@@ -15,83 +15,67 @@
specific language governing permissions and limitations
under the License.
-
**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
-Provider Governance Overview
-============================
+Managing Provider's Lifecycle
+==============================
-Before diving into the technical details of creating and managing providers,
please familiarize
-yourself with the governance framework that applies to all community providers:
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
-* `Provider Governance Framework
<../PROVIDERS.rst#provider-governance-framework>`_
+This document covers the **technical steps** for creating, releasing,
suspending, and removing
+community providers. For governance policies, lifecycle stages, and health
metrics, see
+`Provider Governance <PROVIDER_GOVERNANCE.rst>`_.
-Key governance concepts:
+Before proposing a new provider, review the `acceptance process
<ACCEPTING_PROVIDERS.rst>`_
+and ensure you have:
-* **Lifecycle stages**: Providers move through the following lifecycle stages:
Incubation → Production → Attic/Deprecation. To be precise,
- not all providers will move through all stages.
-* **Stewardship**: Each provider requires at least two stewards (who don't
have to be Airflow Committers themselves)
-* **Health metrics**: Quantitative criteria determine readiness for promotion
or deprecation
-* **Periodic reviews**: The PMC reviews provider health quarterly
+1. At least two individuals willing to serve as stewards
+2. Sponsorship from at least one existing Airflow Committer (if none of the
stewards are a committer)
+3. A commitment to meeting the incubation health metrics within 6 months
+4. A plan to participate in quarterly governance updates on the devlist
-All new providers must start in the **Incubation** stage (unless specifically
accelerated by the PMC)
-and meet the graduation criteria before moving to Production status.
-Existing providers will by default start in the Production stage with the
Airflow PMC as stewards, unless
-stewards have already been identified.
+Include this information in your ``[DISCUSSION]`` thread when proposing a new
provider.
-Stewardship Requirements for New Providers
-------------------------------------------
-When proposing a new community provider, you must:
+Creating a new community provider
+===================================
-1. Identify at least two individuals willing to serve as stewards
-2. Secure sponsorship from at least one existing Airflow Committer (if none of
the stewards are a committer)
-3. Commit to meeting the incubation health metrics within 6 months
-4. Participate in quarterly governance updates on the devlist
+This section gathers the necessary steps to create a new community provider
and also guidelines for
+updating existing ones. Providers may have distinctions not covered in this
guide. The sequence
+described was designed to meet the most linear flow possible in order to
develop a new provider.
-Include this information in your ``[DISCUSSION]`` thread when proposing a new
provider.
+We recommend looking at an existing provider similar to yours for reference on
tests, dependencies,
+and structure.
-Creating a new community provider
-=================================
-This document gathers the necessary steps to create a new community provider
and also guidelines for updating
-the existing ones. You should be aware that providers may have distinctions
that may not be covered in
-this guide. The sequence described was designed to meet the most linear flow
possible in order to develop a
-new provider.
+Setting up the development environment
+----------------------------------------
-Another recommendation that will help you is to look for a provider that works
similar to yours. That way it will
-help you to set up tests and other dependencies.
-
-First, you need to set up your local development environment. See
+First, set up your local development environment. See
`Contributors Quick Start
<../contributing-docs/03b_contributors_quick_start_seasoned_developers.rst>`_
-if you did not set up your local environment yet. We recommend using
``breeze`` to develop locally. This way you
-easily be able to have an environment more similar to the one executed by
GitHub CI workflow.
-
- .. code-block:: bash
+if you have not done so yet. We recommend using ``breeze`` to develop locally,
as it provides an
+environment similar to the one used by GitHub CI workflows.
- ./breeze
+.. code-block:: bash
-Using the code above you will set up Docker containers. These containers your
local code to internal volumes.
-In this way, the changes made in your IDE are already applied to the code
inside the container and tests can
-be carried out quickly.
+ ./breeze
-In this how-to guide our example provider name will be ``<PROVIDER>``.
-When you see this placeholder you must change for your provider name.
+This sets up Docker containers that mount your local code to internal volumes.
Changes made in your
+IDE are immediately available inside the container, allowing tests to be run
quickly.
+In this guide, ``<PROVIDER>`` is used as a placeholder for your provider name.
-Initial Code and Unit Tests
----------------------------
-Most likely you have developed a version of the provider using some local
customization and now you need to
-transfer this code to the Airflow project. Below is described all the initial
code structure that
-the provider may need. Understand that not all providers will need all the
components described in this structure.
-If you still have doubts about building your provider, we recommend that you
read the initial provider guide and
-open a issue on GitHub so the community can help you.
+Code structure
+---------------
-The folders are optional: example_dags, hooks, links, logs, notifications,
operators, secrets, sensors, transfers,
-triggers (and the list changes continuously).
+Below is the directory structure a provider may need. Not all providers
require all components ---
+the folders are optional: example_dags, hooks, links, logs, notifications,
operators, secrets,
+sensors, transfers, triggers (and the list changes continuously).
- .. code-block:: bash
+.. code-block:: bash
GIT apache/airflow/
└── providers/
@@ -121,7 +105,7 @@ triggers (and the list changes continuously).
│ ├──
__init__.py
│ └── *.py
└── tests/
- ├── unit\
+ ├── unit/
│ └── <PROVIDER>/
│ ├── __init__.py
│ ├── executors/
@@ -149,251 +133,162 @@ triggers (and the list changes continuously).
├── __init__.py
└── example_*.py
-Considering that you have already transferred your provider's code to the
above structure, it will now be necessary
-to create unit tests for each component you created. The example below I have
already set up an environment using
-breeze and I'll run unit tests for my Hook.
- .. code-block:: bash
+Unit tests
+-----------
+
+Create unit tests for each component of your provider. Example of running unit
tests inside breeze:
+
+.. code-block:: bash
+
+ [Breeze:3.10.19] root@fafd8d630e46:/opt/airflow# python -m pytest
providers/<PROVIDER>/tests/<PROVIDER>/hook/test_*.py
- [Breeze:3.10.19] root@fafd8d630e46:/opt/airflow# python -m pytest
providers/<PROVIDER>/tests/<PROVIDER>/hook/test_*.py
Integration tests
------------------
+------------------
See `Airflow Integration Tests
<../contributing-docs/testing/integration_tests.rst>`_
Documentation
--------------
-
-An important part of building a new provider is the documentation.
-Some steps for documentation occurs automatically by ``prek`` see
-`Installing prek guide
<../contributing-docs/03b_contributors_quick_start_seasoned_developers.rst#prek>`_
-
-Those are important files in the Airflow source tree that affect providers.
The ``pyproject.toml`` in root
-Airflow folder is automatically generated based on content of
``provider.yaml`` file in each provider
-when ``prek hook`` is run. Files such as ``extra-packages-ref.rst`` should be
manually updated because
-they are manually formatted for better layout and ``prek hook`` will just
verify if the information
-about provider is updated there. Files like ``commit.rst`` and ``CHANGELOG``
are automatically updated
-by ``breeze release-management`` command by release manager when providers are
released.
-
- .. code-block:: bash
-
- ├── pyproject.toml
- └── providers/<PROVIDER>/src/airflow/providers/
- ├── provider.yaml
- ├── pyproject.toml
- ├── CHANGELOG.rst
- │
- └── docs/
- ├── integration-logos
- │ └──
<PROVIDER>.png
- ├── index.rst
- ├── commits.rst
- ├── connections.rst
- └── operators/
- └── <PROVIDER>.rst
-
-There is a chance that your provider's name is not a common English word.
-In this case is necessary to add it to the file ``docs/spelling_wordlist.txt``.
-
-Add your provider dependencies into ``provider.yaml`` under ``dependencies``
key..
-If your provider doesn't have any dependency add a empty list.
+--------------
+
+An important part of building a new provider is the documentation. Some steps
for documentation
+occur automatically by ``prek`` --- see
+`Installing prek guide
<../contributing-docs/03b_contributors_quick_start_seasoned_developers.rst#prek>`_.
+
+Key files in the Airflow source tree that affect providers:
+
+* The ``pyproject.toml`` in root Airflow folder is automatically generated
based on content of
+ ``provider.yaml`` file in each provider when ``prek hook`` is run.
+* Files such as ``extra-packages-ref.rst`` should be manually updated because
they are manually
+ formatted for better layout and ``prek hook`` will just verify if the
information about provider
+ is updated there.
+* Files like ``commit.rst`` and ``CHANGELOG`` are automatically updated by
+ ``breeze release-management`` command by release manager when providers are
released.
+
+Documentation file structure:
+
+.. code-block:: bash
+
+ ├── pyproject.toml
+ └── providers/<PROVIDER>/src/airflow/providers/
+ ├── provider.yaml
+ ├── pyproject.toml
+ ├── CHANGELOG.rst
+ │
+ └── docs/
+ ├── integration-logos
+ │ └──
<PROVIDER>.png
+ ├── index.rst
+ ├── commits.rst
+ ├── connections.rst
+ └── operators/
+ └── <PROVIDER>.rst
+
+If your provider's name is not a common English word, add it to
``docs/spelling_wordlist.txt``.
+
+Add your provider dependencies into ``provider.yaml`` under ``dependencies``
key.
+If your provider doesn't have any dependency add an empty list.
In the ``docs/apache-airflow-providers-<PROVIDER>/connections.rst``:
- add information how to configure connection for your provider.
-In the provider's ``docs/operators/<PROVIDER>.rst`` add information
-how to use the Operator. It's important to add examples and additional
information if your
-Operator has extra-parameters.
-
- .. code-block:: RST
-
- .. _howto/operator:NewProviderOperator:
+In the provider's ``docs/operators/<PROVIDER>.rst`` add information how to use
the Operator.
+It's important to add examples and additional information if your Operator has
extra-parameters.
- NewProviderOperator
- ===================
+.. code-block:: RST
- Use the
:class:`~airflow.providers.<PROVIDER>.operators.NewProviderOperator` to do
something
- amazing with Airflow!
+ .. _howto/operator:NewProviderOperator:
- Using the Operator
- ^^^^^^^^^^^^^^^^^^
+ NewProviderOperator
+ ===================
- The NewProviderOperator requires a ``connection_id`` and this other
awesome parameter.
- You can see an example below:
+ Use the
:class:`~airflow.providers.<PROVIDER>.operators.NewProviderOperator` to do
something
+ amazing with Airflow!
- .. exampleinclude:: /../../<PROVIDER>/example_dags/example_<PROVIDER>.py
- :language: python
- :start-after: [START howto_operator_<PROVIDER>]
- :end-before: [END howto_operator_<PROVIDER>]
+ Using the Operator
+ ^^^^^^^^^^^^^^^^^^
+ The NewProviderOperator requires a ``connection_id`` and this other
awesome parameter.
+ You can see an example below:
-Copy from another, similar provider the docs: ``docs/*.rst``:
+ .. exampleinclude:: /../../<PROVIDER>/example_dags/example_<PROVIDER>.py
+ :language: python
+ :start-after: [START howto_operator_<PROVIDER>]
+ :end-before: [END howto_operator_<PROVIDER>]
-At least those docs should be present
+Copy from another, similar provider the docs: ``docs/*.rst``. At least these
docs should be present:
* security.rst
* changelog.rst
* commits.rst
* index.rst
* installing-providers-from-sources.rst
-* configurations-ref.rst - if your provider has ``config`` element in
provider.yaml with configuration options
- specific for your provider
+* configurations-ref.rst - if your provider has ``config`` element in
provider.yaml with configuration
+ options specific for your provider
Make sure to update/add all information that are specific for the new provider.
-In the ``providers/<PROVIDER>/src/airflow/providers/<PROVIDER>/provider.yaml``
add information of your provider:
-
- .. code-block:: yaml
-
- package-name: apache-airflow-providers-<PROVIDER>
- name: <PROVIDER>
- description: |
- `<PROVIDER> <https://example.io/>`__
- versions:
- - 1.0.0
- integrations:
- - integration-name: <PROVIDER>
- external-doc-url: https://www.example.io/
- logo: /docs/integration-logos/<PROVIDER>.png
- how-to-guide:
- -
/docs/apache-airflow-providers-<PROVIDER>/operators/<PROVIDER>.rst
- tags: [service]
+provider.yaml configuration
+-----------------------------
- operators:
- - integration-name: <PROVIDER>
- python-modules:
- - airflow.providers.<PROVIDER>.operators.<PROVIDER>
+In the ``providers/<PROVIDER>/src/airflow/providers/<PROVIDER>/provider.yaml``
add information of
+your provider:
- hooks:
- - integration-name: <PROVIDER>
- python-modules:
- - airflow.providers.<PROVIDER>.hooks.<PROVIDER>
+.. code-block:: yaml
- sensors:
- - integration-name: <PROVIDER>
- python-modules:
- - airflow.providers.<PROVIDER>.sensors.<PROVIDER>
+ package-name: apache-airflow-providers-<PROVIDER>
+ name: <PROVIDER>
+ description: |
+ `<PROVIDER> <https://example.io/>`__
+ versions:
+ - 1.0.0
- connection-types:
- - hook-class-name:
airflow.providers.<PROVIDER>.hooks.<PROVIDER>.NewProviderHook
- - connection-type: provider-connection-type
+ integrations:
+ - integration-name: <PROVIDER>
+ external-doc-url: https://www.example.io/
+ logo: /docs/integration-logos/<PROVIDER>.png
+ how-to-guide:
+ - /docs/apache-airflow-providers-<PROVIDER>/operators/<PROVIDER>.rst
+ tags: [service]
-After changing and creating these files you can build the documentation
locally. The two commands below will
-serve to accomplish this. The first will build your provider's documentation.
The second will ensure that the
-main Airflow documentation that involves some steps with the providers is also
working.
-
- .. code-block:: bash
-
- breeze build-docs <provider id>
- breeze build-docs apache-airflow
+ operators:
+ - integration-name: <PROVIDER>
+ python-modules:
+ - airflow.providers.<PROVIDER>.operators.<PROVIDER>
-Additional changes needed for cross-dependent providers
-=======================================================
+ hooks:
+ - integration-name: <PROVIDER>
+ python-modules:
+ - airflow.providers.<PROVIDER>.hooks.<PROVIDER>
-Those steps above are usually enough for most providers that are "standalone"
and not imported or used by
-other providers (in most cases we will not suspend such providers). However
some extra steps might be needed
-for providers that are used by other providers, or that are part of the
default PROD Dockerfile:
+ sensors:
+ - integration-name: <PROVIDER>
+ python-modules:
+ - airflow.providers.<PROVIDER>.sensors.<PROVIDER>
-* Most of the tests for the suspended provider, will be automatically excluded
by pytest collection. However,
- in case a provider is dependent on by another provider, the relevant tests
might fail to be collected or
- run by ``pytest``. In such cases you should skip the whole test module
failing to be collected by
- adding ``pytest.importorskip`` at the top of the test module.
- For example if your tests fail because they need to import
``apache.airflow.providers.google``
- and you have suspended it, you should add this line at the top of the test
module that fails.
+ connection-types:
+ - hook-class-name:
airflow.providers.<PROVIDER>.hooks.<PROVIDER>.NewProviderHook
+ - connection-type: provider-connection-type
-Example failing collection after ``google`` provider has been suspended:
-
- .. code-block:: txt
-
- _____ ERROR collecting
providers/apache/beam/tests/apache/beam/operators/test_beam.py ______
- ImportError while importing test module
'/opt/airflow/providers/apache/beam/tests/apache/beam/operators/test_beam.py'.
- Hint: make sure your test modules/packages have valid Python names.
- Traceback:
- /usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
- return _bootstrap._gcd_import(name[level:], package, level)
- providers/apache/beam/tests/apache/beam/operators/test_beam.py:25: in
<module>
- from airflow.providers.apache.beam.operators.beam import (
- airflow/providers/apache/beam/operators/beam.py:35: in <module>
- from airflow.providers.google.cloud.hooks.dataflow import (
- airflow/providers/google/cloud/hooks/dataflow.py:32: in <module>
- from google.cloud.dataflow_v1beta3 import GetJobRequest, Job,
JobState, JobsV1Beta3AsyncClient, JobView
- E ModuleNotFoundError: No module named 'google.cloud.dataflow_v1beta3'
- _ ERROR collecting
providers/microsoft/azure/tests/microsoft/azure/transfers/test_azure_blob_to_gcs.py
_
+Building documentation locally
+-------------------------------
-The fix is to add this line at the top of the
``providers/apache/beam/tests/apache/beam/operators/test_beam.py`` module:
+After creating and updating the files, build the documentation locally. The
first command builds your
+provider's documentation, the second ensures the main Airflow documentation is
also working:
- .. code-block:: python
+.. code-block:: bash
- pytest.importorskip("apache.airflow.providers.google")
-
-
-* Some of the other providers might also just import unconditionally the
suspended provider and they will
- fail during the provider verification step in CI. In this case you should
turn the provider imports
- into conditional imports. For example when import fails after ``amazon``
provider has been suspended:
-
- .. code-block:: txt
-
- Traceback (most recent call last):
- File "/opt/airflow/scripts/in_container/verify_providers.py", line
266, in import_all_classes
- _module = importlib.import_module(modinfo.name)
- File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in
import_module
- return _bootstrap._gcd_import(name, package, level)
- File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
- File "<frozen importlib._bootstrap>", line 983, in _find_and_load
- File "<frozen importlib._bootstrap>", line 967, in
_find_and_load_unlocked
- File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
- File "<frozen importlib._bootstrap_external>", line 728, in exec_module
- File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
- File
"/usr/local/lib/python3.8/site-packages/airflow/providers/mysql/transfers/s3_to_mysql.py",
line 23, in <module>
- from airflow.providers.amazon.aws.hooks.s3 import S3Hook
- ModuleNotFoundError: No module named 'airflow.providers.amazon'
-
-or:
-
- .. code-block:: txt
-
- Error: The ``airflow.providers.microsoft.azure.transfers.azure_blob_to_gcs``
object in transfers list in
- airflow/providers/microsoft/azure/provider.yaml does not exist or is not a
module:
- No module named 'gcloud.aio.storage'
-
-The fix for that is to turn the feature into an optional provider feature (in
the place where the excluded
-``airflow.providers`` import happens:
-
- .. code-block:: python
-
- try:
- from airflow.providers.amazon.aws.hooks.s3 import S3Hook
- except ImportError as e:
- from airflow.exceptions import AirflowOptionalProviderFeatureException
-
- raise AirflowOptionalProviderFeatureException(e)
-
-
-* In case we suspend an important provider, which is part of the default
Dockerfile you might want to
- update the tests for PROD docker image in
``docker-tests/tests/docker_tests/test_prod_image.py``.
-
-* Some of the suspended providers might also fail ``breeze`` unit tests that
expect a fixed set of providers.
- Those tests should be adjusted (but this is not very likely to happen,
because the tests are using only
- the most common providers that we will not be likely to suspend).
-
-Bumping min Airflow version
-===========================
-
-We regularly bump min Airflow version for all providers we release. This bump
is done according to our
-`Provider policies
<https://github.com/apache/airflow/blob/main/PROVIDERS.rst>`_ and it is only
applied
-to non-suspended/removed providers. We are running basic import compatibility
checks in our CI and
-the compatibility checks should be updated when min Airflow version is updated.
+ breeze build-docs <provider id>
+ breeze build-docs apache-airflow
-Details on how this should be done are described in
-`Provider policies
<https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md>`_
Conditional provider variants
-=============================
+==============================
Sometimes providers need to have different variants for different versions of
Airflow. This is done by:
@@ -417,8 +312,12 @@ The main reasons we are doing it in this way:
via prek hook ``check-imports-in-providers`` that will fail if the
``version_compat`` module is imported from another provider or from test
code.
+
+Releasing providers
+====================
+
Releasing pre-installed providers for the first time
-====================================================
+-----------------------------------------------------
When releasing providers for the first time, you need to release them in state
``not-ready``.
This will make it available for release management commands, but it will not
be added to airflow's
@@ -430,8 +329,9 @@ considered by the release management commands.
As soon as the provider is released, you should update the provider to
``state: ready``.
+
Releasing providers for past releases
-=====================================
+---------------------------------------
Sometimes we might want to release provider for previous MAJOR when new
release is already
released (or bumped in main). This is done by releasing them from
``providers-<PROVIDER>/vX-Y`` branch
@@ -442,14 +342,30 @@ The release process looks like usual, the only difference
is that the specific b
the provider and update all documentation, the changes and cherry-picking
should be targeting that
branch.
+
+Bumping min Airflow version
+-----------------------------
+
+We regularly bump min Airflow version for all providers we release. This bump
is done according to our
+`Provider policies
<PROVIDER_RELEASES.rst#upgrading-minimum-supported-version-of-airflow>`_ and it
is only applied
+to non-suspended/removed providers. We are running basic import compatibility
checks in our CI and
+the compatibility checks should be updated when min Airflow version is updated.
+
+Details on how this should be done are described in
+`Provider policies
<https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md>`_
+
+
Suspending providers
-====================
+=====================
As of April 2023, we have the possibility to suspend individual providers, so
that they are not holding
-back dependencies for Airflow and other providers. The process of suspending
providers is described
-in `description of the process
<https://github.com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers>`_
+back dependencies for Airflow and other providers. The criteria and process
for suspending providers is
+described in `Suspending releases for providers
<SUSPENDING_AND_REMOVING_PROVIDERS.rst#suspending-releases-for-providers>`_.
-Technically, suspending a provider is done by setting ``state: suspended``, in
the provider.yaml of the
+Technical steps
+----------------
+
+Suspending a provider is done by setting ``state: suspended`` in the
provider.yaml of the
provider. This should be followed by committing the change and either
automatically or manually running
prek hooks that will either update derived configuration files or ask you to
update them manually.
Note that you might need to run prek several times until all the static checks
pass,
@@ -465,9 +381,9 @@ and then run the static checks again.
If you want to be absolutely sure to run all static checks you can always do
this via
``prek --all-files``.
-Some of the manual modifications you will have to do (in both cases ``prek``
will guide you on what to do
+Some of the manual modifications you will have to do (in both cases ``prek``
will guide you on what to do):
-* You will have to run ``breeze setup regenerate-command-images`` to
regenerate breeze help files
+* You will have to run ``breeze setup regenerate-command-images`` to
regenerate breeze help files
* you will need to update ``extra-packages-ref.rst`` and in some cases - when
mentioned there explicitly -
``pyproject.toml`` to remove the provider from list of dependencies.
@@ -476,24 +392,115 @@ the information about available providers and their
dependencies and it is used
exclude suspended providers from all relevant parts of the build and CI system
(such as building CI image
with dependencies, building documentation, running tests, etc.)
+
+Handling cross-dependent providers
+------------------------------------
+
+The steps above are usually enough for most standalone providers. However,
extra steps might be needed
+for providers that are used by other providers, or that are part of the
default PROD Dockerfile.
+
+**Test collection failures**
+
+Most tests for the suspended provider will be automatically excluded by pytest
collection. However,
+if another provider depends on the suspended one, its tests might fail to be
collected. In such cases,
+add ``pytest.importorskip`` at the top of the failing test module.
+
+Example failing collection after ``google`` provider has been suspended:
+
+.. code-block:: txt
+
+ _____ ERROR collecting
providers/apache/beam/tests/apache/beam/operators/test_beam.py ______
+ ImportError while importing test module
'/opt/airflow/providers/apache/beam/tests/apache/beam/operators/test_beam.py'.
+ Hint: make sure your test modules/packages have valid Python names.
+ Traceback:
+ /usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
+ return _bootstrap._gcd_import(name[level:], package, level)
+ providers/apache/beam/tests/apache/beam/operators/test_beam.py:25: in
<module>
+ from airflow.providers.apache.beam.operators.beam import (
+ airflow/providers/apache/beam/operators/beam.py:35: in <module>
+ from airflow.providers.google.cloud.hooks.dataflow import (
+ airflow/providers/google/cloud/hooks/dataflow.py:32: in <module>
+ from google.cloud.dataflow_v1beta3 import GetJobRequest, Job,
JobState, JobsV1Beta3AsyncClient, JobView
+ E ModuleNotFoundError: No module named 'google.cloud.dataflow_v1beta3'
+ _ ERROR collecting
providers/microsoft/azure/tests/microsoft/azure/transfers/test_azure_blob_to_gcs.py
_
+
+Fix by adding this line at the top of the failing test module:
+
+.. code-block:: python
+
+ pytest.importorskip("apache.airflow.providers.google")
+
+
+**Import failures in provider verification**
+
+Some providers might unconditionally import the suspended provider and fail
during the provider
+verification step in CI. Turn these imports into conditional imports.
+
+Example when import fails after ``amazon`` provider has been suspended:
+
+.. code-block:: txt
+
+ Traceback (most recent call last):
+ File "/opt/airflow/scripts/in_container/verify_providers.py", line 266,
in import_all_classes
+ _module = importlib.import_module(modinfo.name)
+ ...
+ File
"/usr/local/lib/python3.8/site-packages/airflow/providers/mysql/transfers/s3_to_mysql.py",
line 23, in <module>
+ from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+ ModuleNotFoundError: No module named 'airflow.providers.amazon'
+
+or:
+
+.. code-block:: txt
+
+ Error: The
``airflow.providers.microsoft.azure.transfers.azure_blob_to_gcs`` object in
transfers list in
+ airflow/providers/microsoft/azure/provider.yaml does not exist or is not a
module:
+ No module named 'gcloud.aio.storage'
+
+Fix by turning the feature into an optional provider feature:
+
+.. code-block:: python
+
+ try:
+ from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+ except ImportError as e:
+ from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+ raise AirflowOptionalProviderFeatureException(e)
+
+
+**Other potential impacts**
+
+* If the suspended provider is part of the default Dockerfile, you might want
to update the tests for
+ PROD docker image in ``docker-tests/tests/docker_tests/test_prod_image.py``.
+* Some suspended providers might also fail ``breeze`` unit tests that expect a
fixed set of providers.
+ Those tests should be adjusted (but this is not very likely, because the
tests use only the most
+ common providers that we will not be likely to suspend).
+
+
Resuming providers
-==================
+===================
Resuming providers is done by reverting the original change that suspended it.
In case there are changes
needed to fix problems in the reverted provider, our CI will detect them and
you will have to fix them
as part of the PR reverting the suspension.
+
Removing providers
-==================
+===================
When removing providers from Airflow code, we need to make one last release
where we mark the provider as
-removed - in documentation and in description of the PyPI package. In order to
that release manager has to
-add "state: removed" flag in the provider yaml file and include the provider
in the next wave of the
-providers (and then remove all the code and documentation related to the
provider).
+removed - in documentation and in description of the PyPI package. For the
criteria and process, see
+`Removing community providers
<SUSPENDING_AND_REMOVING_PROVIDERS.rst#removing-community-providers>`_.
+
+Technical steps
+----------------
+
+The release manager has to add ``state: removed`` in the provider yaml file
and include the provider in
+the next wave of the providers (and then remove all the code and documentation
related to the provider).
-The "removed: removed" flag will cause the provider to be available for the
following commands (note that such
-provider has to be explicitly added as selected to the package - such provider
will not be included in
-the available list of providers or when documentation is built unless
--include-removed-providers
+The ``state: removed`` flag will cause the provider to be available for the
following commands (note that
+such provider has to be explicitly added as selected to the package - it will
not be included in
+the available list of providers or when documentation is built unless
``--include-removed-providers``
flag is used):
* ``breeze build-docs``
diff --git a/providers/PROVIDER_GOVERNANCE.rst
b/providers/PROVIDER_GOVERNANCE.rst
new file mode 100644
index 00000000000..b9852968b4e
--- /dev/null
+++ b/providers/PROVIDER_GOVERNANCE.rst
@@ -0,0 +1,215 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+
+Provider Governance
+====================
+
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
+
+This document describes the governance framework for community providers,
including lifecycle stages,
+stewardship model, and quantitative health metrics.
+
+For an overview of what providers are and how they fit into the Airflow
ecosystem, see
+`Apache Airflow Providers <../PROVIDERS.rst>`_.
+
+Governance framework
+--------------------
+
+Airflow's success is built on its extensive ecosystem of community-supported
integrations---over 1,600
+hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
+for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
+number of integrations while ensuring quality. Actual code acceptance and
release governance remains with the Airflow PMC.
+
+Stewardship model
+^^^^^^^^^^^^^^^^^
+
+Each provider or integration component must have designated **stewards**
responsible for ensuring
+the health criteria described below are met:
+
+* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
+* **Role definition**: Stewards are subject matter experts for the
integration. This could be expertise in the
+ service being integrated, the language being supported by the provider, or
the framework being integrated.
+ Stewardship is a responsibility, not an additional authority or privilege
+* **Committer sponsorship**: Each steward must be sponsored by at least one
Airflow Committer. The
+ sponsor ensures that stewards fulfill their responsibilities and provides
guidance on maintaining the
+ provider according to best practices. This includes regular dependency
updates, issue resolution, and
+ monitoring that the provider meets the health metrics required for its
current lifecycle stage (that is, incubation
+ or production). The sponsor is responsible for PR reviews and merges
(including ensuring coding standards are met), but
+ is NOT responsible for active maintenance of the provider's codebase, which
remains the responsibility of the stewards.
+ While sponsors should be accountable when it comes to reviews and merges,
it's also OK and welcome that other committers merge code providing it fulfill
the criteria.
+* **Accountability**: Ultimate accountability remains with the Airflow PMC and
Committers
+* **Transitions**: Neither sponsorship nor stewardship are roles in
perpetuity; they can be
+ transitioned to others based on mutual agreement and PMC approval
+
+.. note::
+
+ The quantitative criteria described below are aspirational. The PMC will
revisit these metrics
+ based on actual experience 6 months from the date of the first quarterly
review of the provider metrics being published.
+
+Lifecycle stages
+^^^^^^^^^^^^^^^^
+
+Providers generally follow a three-stage lifecycle: **Incubation**,
**Production**, and **Attic/Deprecation**. However,
+not all providers will move through all stages. Additionally, providers may be
designated as **Mature** under specific circumstances.
+
+Incubation stage
+""""""""""""""""
+
+All new providers or integration modules (such as Notifiers, Message
components, etc.) should start
+in incubation unless specifically accelerated by the PMC. Incubation ensures
that code, licenses, and
+processes align with ASF guidelines and community principles such as
community, collaborative
+development, and openness.
+
+Requirements for incubation:
+
+* Working codebase to bootstrap credibility and attract contributions
+* Visibility in the provider health dashboard (The provider health dashboard
is to be added)
+* Designated stewards with committer sponsorship
+
+Quantitative graduation criteria:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Minimum of 10 PRs in the last 6 months
+ * - PR review time
+ - PRs reviewed within 14 days
+ * - Issues reported
+ - Minimum of 15 unique issues filed in the last 6 months
+ * - Contributors
+ - At least 3 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - At least 50% of reported issues closed within 90 days
+ * - Security/release issues
+ - All release and security related issues closed within 60 days
+ * - Governance participation
+ - Demonstrated participation in project governance channels including
quarterly updates
+ * - Quality standards
+ - Meet quality criteria for code, documentation, and tests as listed in
the Contributing Guide
+
+Production stage
+""""""""""""""""
+
+All modules in production are expected to be well managed with prompt
resolution of issues and
+support for a consistent release cadence (at least monthly, but typically
every 2 weeks when changes
+exist). Production providers must:
+
+* Stay consistent with the main branch
+* Pass tests for main + all supported Airflow versions
+* Follow Airflow support guidelines, staying current with main Airflow releases
+
+Exceptions can be granted based on a PMC/devlist vote (PMC members only having
binding votes) for
+valid and one-off criteria.
+
+Quantitative criteria to maintain production status:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Minimum of 10 PRs in the last 6 months
+ * - PR review time
+ - PRs reviewed within 14 days
+ * - Issues reported
+ - Minimum of 20 unique issues filed in the last 6 months
+ * - Contributors
+ - At least 5 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - At least 60% of reported issues closed within 90 days
+ * - Security/release issues
+ - All release and security related issues closed within 30 days
+ * - Feature releases
+ - At least 1 feature release every 6 months
+ * - User engagement
+ - Maintain support activity with response to questions within 2 weeks on
average
+ * - Governance participation
+ - Demonstrated participation in project governance channels including
quarterly updates
+
+Attic / Deprecation stage
+"""""""""""""""""""""""""
+
+Modules should be moved into the Attic when relevance wanes, typically
measured by declining
+activity. This commonly occurs when the integrated solution has faded in
popularity and is replaced
+by more modern alternatives.
+
+Movement to the Attic must be communicated on the dev list and voted on by the
PMC. Exceptions can
+be granted based on the vote.
+
+Quantitative criteria triggering attic consideration:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Fewer than 5 PRs in the last 6 months
+ * - PR review time
+ - PRs not being reviewed in more than a month
+ * - Issues reported
+ - Fewer than 10 unique issues filed in the last 6 months
+ * - Contributors
+ - Fewer than 3 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - Less than 30% of reported issues closed within 90 days
+ * - Security/release issues
+ - Release and security related issues not getting closed within 30 days
+ * - Feature releases
+ - No feature releases in the last 6 months
+
+Consequences of attic status:
+
+* Modules remain readable but do not receive active maintenance
+* Module is not actively tested in "main" in Airflow CI, its dependencies are
not checked for conflicts
+ with other main providers, and common refactorings are not applied to it.
+* After a period of at least 6 months in the attic, modules can be chosen for
removal with
+ appropriate communication (see `Removing community providers
<SUSPENDING_AND_REMOVING_PROVIDERS.rst#removing-community-providers>`_)
+* It is possible for the provider to be resurrected from the attic as long as
there is confidence that there is a
+ clear need for the provider and the (possibly new) stewards are able to
maintain it actively on this go around.
+ It should be noted that significant effort may be required to resurrect a
provider from the attic.
+
+Mature providers
+""""""""""""""""
+
+Some providers may have very stable interfaces requiring minimal changes on a
regular basis (e.g.,
+HTTP provider integration). At the discretion of the Airflow PMC, certain
providers can be tagged
+as **"mature providers"**, which will not automatically be deprecated and
moved into the attic due
+to lack of activity alone.
+
+Periodic reviews
+^^^^^^^^^^^^^^^^
+
+The Airflow PMC is responsible for reviewing the health status of integrations
on a **quarterly
+basis** and making decisions such as:
+
+* Graduating providers from incubation to production
+* Moving providers from production to the attic
+* Granting exceptions for specific providers
+
+These discussions will be held in public and subsequently summarized and
shared on the Airflow devlist.
diff --git a/providers/PROVIDER_RELEASES.rst b/providers/PROVIDER_RELEASES.rst
new file mode 100644
index 00000000000..f54da65d3e4
--- /dev/null
+++ b/providers/PROVIDER_RELEASES.rst
@@ -0,0 +1,103 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+
+Provider Releases and Versioning
+=================================
+
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
+
+Release process
+---------------
+
+The community providers are released regularly (usually every 2 weeks) in
batches consisting of any providers
+that need to be released because they changed since last release. The release
manager decides which providers
+to include and whether some or all providers should be released (see the next
chapter about upgrading the
+minimum version of Airflow for example the case where we release all active
meaning non-suspended providers,
+together in a single batch). Also Release Manager decides on the version bump
of the provider (depending on
+classification, whether there are breaking changes, new features or just bugs
comparing to previous version).
+
+Versioning scheme
+-----------------
+
+We are using the `SEMVER <https://semver.org/>`_ versioning scheme for the
Provider distributions. This is in order
+to give the users confidence about maintaining backwards compatibility in the
new releases of those
+packages.
+
+Details about maintaining the SEMVER version are going to be discussed and
implemented in
+`the related issue <https://github.com/apache/airflow/issues/11425>`_
+
+Provider distribution states
+-----------------------------
+
+The Provider distributions can be in one of several states.
+
+* The ``not-ready`` state is used when the provider has some in-progress
changes (usually API changes) that
+ we do not want to release yet as part of the regular release cycle.
Providers in this state are excluded
+ from being released as part of the regular release cycle (including
documentation building).
+ The ``not-ready`` providers are treated as regular providers when it comes
to running tests and preparing
+ and releasing packages in ``CI`` - as we want to make sure they are properly
releasable any time and we
+ want them to contribute to dependencies and we want to test them. Also in
case of preinstalled providers,
+ the ``not-ready`` providers are contributing their dependencies rather than
the provider package to
+ requirements of Airflow.
+* The ``ready`` state is the usual state of the provider that is released in
the regular release cycle
+ (including the documentation, package building and publishing). This is the
state most providers are in.
+* The ``suspended`` state is used when we have a good reason to suspend such
provider, following the devlist
+ discussion and vote or "lazy consensus". The process of suspension is
described in
+ `Suspending releases for providers
<SUSPENDING_AND_REMOVING_PROVIDERS.rst#suspending-releases-for-providers>`_.
+ The ``suspended`` providers are excluded from being released as part of the
regular release cycle (including
+ documentation building) but also they do not contribute dependencies to the
CI image and their tests are
+ not run in CI process. The ``suspended`` providers are not released as part
of the regular release cycle.
+* The ``removed`` state is a temporary state after the provider has been voted
(or agreed in "lazy consensus")
+ to be removed and it is only used for exactly one release cycle - in order
to produce the final version of
+ the package - identical to the previous version with the exception of the
removal notice. The process
+ of removal is described in `Removing community providers
<SUSPENDING_AND_REMOVING_PROVIDERS.rst#removing-community-providers>`_.
+ The difference between ``suspended`` and ``removed`` providers is that
additional information is added to
+ their documentation about the provider not being maintained any more by the
community.
+
+Upgrading minimum supported version of Airflow
+-----------------------------------------------
+
+.. note::
+
+ The following policy applies for Airflow 2. It has not yet been finalized
for Airflow 3 and is subject to changes.
+
+One of the important limitations of the Providers released by the community is
that we introduce the limit
+of a minimum supported version of Airflow. The minimum version of Airflow is
the ``MINOR`` version (2.4, 2.5 etc.)
+indicating that the providers might use features that appeared in this
release. The default support timespan
+for the minimum version of Airflow (there could be justified exceptions) is
that we increase the minimum
+Airflow version to the next MINOR release, when 12 months passed since the
first release for the
+MINOR version of Airflow.
+
+For example this means that by default we upgrade the minimum version of
Airflow supported by providers
+to 3.1.0 in the first Provider's release after 20th of May 2026. 20th of May
2025 was the date when the
+first ``PATCHLEVEL`` version of 2.11 (2.11.0) was released and since Airflow
3.0 was released in April 2025,
+we go straight to Airflow 3.1 as minimum supported version of Airflow for
providers in May 2026.
+
+When we increase the minimum Airflow version, this is not a reason to bump
``MAJOR`` version of the providers
+(unless there are other breaking changes in the provider). The reason for that
is that people who use
+older version of Airflow will not be able to use that provider (so it is not a
breaking change for them)
+and for people who are using supported version of Airflow this is not a
breaking change on its own - they
+will be able to use the new version without breaking their workflows. When we
upgraded min-version to
+2.2+, our approach was different but as of 2.3+ upgrade (November 2022) we
only bump ``MINOR`` version of the
+provider when we increase minimum Airflow version.
+
+Increasing the minimum version of the Providers is one of the reasons why
3rd-party provider maintainers
+might want to maintain their own providers - as they can decide to support
older versions of Airflow.
diff --git a/providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst
b/providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst
new file mode 100644
index 00000000000..31c3d88a771
--- /dev/null
+++ b/providers/SUSPENDING_AND_REMOVING_PROVIDERS.rst
@@ -0,0 +1,110 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+
+Suspending and Removing Providers
+==================================
+
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
+
+For technical details on how to perform these operations, see
+`Managing provider's lifecycle <MANAGING_PROVIDERS_LIFECYCLE.rst>`_.
+
+Suspending releases for providers
+----------------------------------
+
+In case a provider is found to require old dependencies that are not
compatible with upcoming versions of
+the Apache Airflow or with newer dependencies required by other providers, the
provider's release
+process can be suspended.
+
+This means:
+
+* The provider's state in ``provider.yaml`` is set to "suspended"
+* No new releases of the provider will be made until the problem with
dependencies is solved
+* Sources of the provider remain in the repository for now (in the future we
might add process to remove them)
+* No new changes will be accepted for the provider (other than the ones that
fix the dependencies)
+* The provider will be removed from the list of Apache Airflow extras in the
next Airflow release
+ (including patch-level release if it is possible/easy to cherry-pick the
suspension change)
+* Tests of the provider will not be run on our CI (in main branch)
+* Dependencies of the provider will not be installed in our main branch CI
image nor included in constraints
+* We can still decide to apply security fixes to released providers - by
adding fixes to the main branch
+ but cherry-picking, testing and releasing them in the patch-level branch of
the provider similar to the
+ mixed governance model described in `3rd-party providers
<THIRD_PARTY_PROVIDERS.rst#mixed-governance-model>`_.
+
+The suspension may be triggered by any committer after the following criteria
are met:
+
+* The maintainers of dependencies of the provider are notified about the issue
and are given a reasonable
+ time to resolve it (at least 1 week)
+* Other options to resolve the issue have been exhausted and there are good
reasons for upgrading
+ the old dependencies in question
+* Explanation why we need to suspend the provider is stated in a public
discussion in the devlist. Followed
+ by ``[LAZY CONSENSUS]`` or ``[VOTE]`` discussion at the devlist (with the
majority of the binding votes
+ agreeing that we should suspend the provider)
+
+The suspension will be lifted when the dependencies of the provider are made
compatible with the Apache
+Airflow and with other providers - by merging a PR that removes the suspension
and succeeds.
+
+Removing community providers
+------------------------------
+
+After a Provider has been deprecated, as described in the
+`Provider Governance <PROVIDER_GOVERNANCE.rst#attic-deprecation-stage>`_
document with a ``[VOTE]`` thread, it can
+be removed from main branch of Airflow when the community agrees that there
should be no
+more updates to the providers done by the community - except maybe potentially
security fixes found. There
+might be various reasons for the providers to be removed:
+
+* the service they connect to is no longer available
+* the dependencies for the provider are not maintained anymore and there is no
viable alternative
+* there is another, more popular provider that supersedes community provider
+* etc. etc.
+
+Generally speaking a discussion thread ``[DISCUSS]`` is advised before such
removal and
+sufficient time should pass (at least a week) to give a chance for community
members to express their
+opinion on the removal.
+
+Consequences of removal:
+
+* One last release of the provider is done with documentation updated
informing that the provider is no
+ longer maintained by the Apache Airflow community - linking to this page.
This information should also
+ find its way to the package documentation and consequently - to the
description of the package in PyPI.
+* An ``[ANNOUNCE]`` thread is sent to the devlist and user list announcing
removal of the provider
+* The released providers remain available on PyPI and in the
+ `Archives <https://archive.apache.org/dist/airflow/providers/>`_ of the
Apache
+ Software Foundation, while they are removed from the
+ `Downloads <https://downloads.apache.org/airflow/providers/>`_ .
+ Also it remains in the Index of the Apache Airflow Providers documentation at
+ `Airflow Documentation <https://airflow.apache.org/docs/>`_ with note ``(not
maintained)`` next to it.
+* The code of the provider is removed from ``main`` branch of the Apache
Airflow repository - including
+ the tests and documentation. It is no longer built in CI and dependencies of
the provider no longer
+ contribute to the CI image/constraints of Apache Airflow for development and
future ``MINOR`` release.
+* The provider is removed from the list of Apache Airflow extras in the next
``MINOR`` Airflow release
+* The dependencies of the provider are removed from the constraints of the
Apache Airflow
+ (and the constraints are updated in the next ``MINOR`` release of Airflow)
+* In case of confirmed security issues that need fixing that are reported to
the provider after it has been
+ removed, there are two options:
+
+ * in case there is a viable alternative or in case the provider is anyhow
not useful to be installed, we
+ might issue advisory to the users to remove the provider (and use
alternatives if applicable)
+ * in case the users might still need the provider, we still might decide to
release new version of the
+ provider with security issue fixed, starting from the source code in Git
history where the provider was
+ last released. This however, should only be done in case there are no
viable alternatives for the users.
+
+* Removed provider might be re-instated as maintained provider, but it needs
to go through the regular process
+ of accepting new provider described in `Accepting new community providers
<ACCEPTING_PROVIDERS.rst>`_.
diff --git a/providers/THIRD_PARTY_PROVIDERS.rst
b/providers/THIRD_PARTY_PROVIDERS.rst
new file mode 100644
index 00000000000..3394fd6bff0
--- /dev/null
+++ b/providers/THIRD_PARTY_PROVIDERS.rst
@@ -0,0 +1,54 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+
+3rd-Party Providers
+====================
+
+.. contents:: Table of Contents
+ :depth: 3
+ :local:
+
+Relation to community providers
+---------------------------------
+
+Providers, can also be maintained and released by 3rd parties (service
providers or systems integrators).
+There is no difference between the community and 3rd party providers - they
have all the same capabilities
+and limitations.
+
+This is especially in case the provider concerns 3rd-party service that has a
team that can manage provider
+on their own, has a rapidly evolving live service, and believe they need a
faster release cycle than the community
+can provide.
+
+Information about such 3rd-party providers are usually published at the
+`Ecosystem: plugins and providers
<https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers>`_
+page of the Airflow website and we encourage the service providers to publish
their providers there. You can also
+find a 3rd-party registries of such providers, that you can use if you search
for existing providers (they
+are also listed at the "Ecosystem" page in the same chapter)
+
+While we already have - historically - a number of 3rd-party service providers
managed by the community,
+most of those services have dedicated teams that keep an eye on the community
providers and not only take
+active part in managing them (see mixed-governance model below), but also
provide a way that we can
+verify whether the provider works with the latest version of the service via
dashboards that show
+status of System Tests for the provider. This allows us to have a high level
of confidence that when we
+release the provider it works with the latest version of the service. System
Tests are part of the Airflow
+code, but they are executed and verified by those 3rd party service teams. We
are working with the 3rd
+party service teams (who are often important stakeholders of the Apache
Airflow project) to add dashboards
+for the historical providers that are managed by the community, and current
set of Dashboards can be also
+found at the
+`Ecosystem: system test dashboards
<https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards>`_
