Hello here, We have been, over the last few months. struggling with an approach for bumping versions of providers, when there are inter-dependencies between providers. We have a very good example of it now when we want to move Airflow 3 SDK compatibility code to common.compat provider (example for standard provider here https://github.com/apache/airflow/pull/56867) and I would like to use that opportunity to discuss possible approaches and settle on a good solution that Elad and potentially other release manager could follow.
Also that might be an input later to the "potential providers separation" discussion we might have later this year as it describes one of the issues that happens when we isolate things to separate repos (but this is something we will deal with in the future). # Problem We have provider A depending on B and we want to add a code to provider B that A will use. In this case, we add the "sdk" module to provider A. Example: we add "compatibility" code to "common.compat" provider (currently released version 1.7.4) and in the upcoming "standard" provider we want to use that code. This means that when we release standard provider with that code we have to have this dependency: "apache-airflow-providers-common-compat >= FUTURE VERSION OF COMMON COMPAT". Due to the way Python dependency system works and development installation of dependent providers work in development workspace, we cannot just say (without actually bumping the common.compat in our code to 1.8.0), this will cause conflict. ""apache-airflow-providers-common-compat >= 1.8.0" # Possible solutions We tried two different things: 1) having contributors bump "common.compat" when they add new things. This is problematic, because bumping the version is a release manager's work, and it gets in the way of preparing release notes. Release notes are prepared at the time of release preparation and we have automation that helps the release manager to decide what version we should bump provider to - based on types of merged changes. Imagine that we have a breaking change, then the new version would be a "2.0.0" not "1.8.0" - but the release manager knows that only after all changes since last release are merged. Also our automation properly prepares release notes in this case, grouping the changes by type, and it is part of the "bumping version" process - but once the version has been bumped, that is already not easy to reconciiliate. 2) Do NOT bump the version by contributors - but leave a TODO to do so by Release manager: "apache-airflow-providers-common-compat>=1.7.4", # + TODO: bump to next version While this is a bit "hacky" - it kind of works and we can even easily automate it during the release process. Say - we change if "TODO: bump" is present as comment and we automatically bump it a release preparation time (we would have to make topological sort and first bump dependent providers and then those that depend on them (to know the version to bump to) but this should be **easy"". Also - we could make the same approach for breaking changes (contributors currently are supposed to add migration comments at the top of release notes when the next version of the provider is going to be breaking, the contributor could then add a "# TODO: bump major version of provider as there are breaking changes" comment. 3) Maybe there are other ways we have not thought about ? # Question I think - personally - that option 2) is better as it completely leaves the "version bump" in the hands of the release manager - which seems more robust and less brittle. But it has a small problem that contributors need a bit of education and explanation on how to do it (it's not obvious). We could add some comments to the pyproject.toml to explain it - but no-one reads those comments anyway - so it will then be up to committers to learn it as well and make it part of the review process to make sure this is happening. I would love to hear your thoughts about it - especially when we have discussion to pick-up about making possible to accept more providers - clarifying this part of the process and settling down on how we approach this use case (which is actually quite common - not only in common.compat and common.*, but also in some cases where for example Google provider depends on some features in cncf.kubernetes etc. WDYT? J.
