Hello Airflow Community, I decided to start a thread (just to avoid some surprises) that will help us to be well aware of some of the consequences of the currently agreed policy we have for providers version support. I think it's good we re-iterate it now.
Just to explain my view: I am personally, 100% ok with the current policies and their consequences. I was fully aware of them when we discussed them and tried to make sure they are well understood before we got them agreed. And I am NOT calling for a change. I think the policy we have is good and drives faster adoption of new releases of Airflow while leaving plenty of time for everyone - Enterprise, Small business users and Managed Services users to prepare and migrate their Airflow in case they want to make sure they will be able to install new providers (including security fixes). But I also realise that some of the community members could have not seen fully the consequences when we agreed it. Since we have a "fresh" set of examples that show it - using those examples is always easier than explaining policies. And I think it might be easier for others to understand and raise concerns, and if there will be enough concern/consensus in a direction to change/adjust/adapt such policy - it might make it easier for them to potentially raise concrete proposals of changes - also basing themon the "real life" examples. Some facts: 1) We released the new provider wave with new providers compatible with 2.3+ according to our 12 months policy of supporting a MINOR version of Airflow in providers. 2) We have 4 providers with Remote Code Execution for CVEs (now public) that are 2.3+ only. This basically means that if anyone uses Spark, Hive, Pig, Pinot provider, other than preparing a patched version on their own based on our CVE announcements, they have to upgrade to Airflow 2.3+. Note that for 3 out of those 4 providers, those are breaking changes because it was not really feasible to fix the vulnerabilities in a non-breaking fashion. 3) This situation might repeat again in providers released after 30th of April 2023 - any providers released in May, will have Airflow 2.4+ compatibility and users who would like to use those providers (for example because they will contain security fixes) will have to upgrade Airflow to 2.4+. Again - I am not calling for a change. I think the policy is good and it will help avoiding the pitfalls of Airlfow 1.10 "loooong tail" of upgrades and pairing with SemvVer and the way we made it easier and more rebust ways to upgrade/downgrade, it is a great policy for the community as a whole, Yes, it strongly encourages changes in users behaviours when it comes to more frequent and proactive upgrades, but this is precisely what I think, is better for the community as a whole. But if someone has some other proposals and arguments - this is a great example that we can use during the discussion, now when the consequences of our agreed policies are easy visible and easy to grasp. To make it easy for others - here are past discussions about it and consensus agreed: * Discussion in December 2021 about the policy https://lists.apache.org/thread/csczm7xmnntdz9wtjbod8pqgt13zoggo where we seemed to agree about the consensus * Calling for the lazy consensus in February 2022 with TL;DR; after the discussion above: https://lists.apache.org/thread/43v1dnww37t0o924mzo365ot2p58k864 * Related discussion about removing deprecated code from providers in May 2022: https://lists.apache.org/thread/6px7f0r3mmocbox45prlso7zor7wzjhx * Vote for the 2.2+ release of providers (with explanation of the policy - first time when we applied it) from June 2022 https://lists.apache.org/thread/7f1ljqm96no4kpbrydfrlcpnr1w9q120 * Vote for the 2.3+ release of providers (again with explanation) from November 2022 - second time when we applied it https://lists.apache.org/thread/w0hnwqb8ftp1lsjm5cfcmx5t9hqzq2yb J.
