I thought a bit that I think if we agree on some "earlier than end of life" policy, that might be a good thing. There are a few packages that are dropping Python support generally 3-6 months before EOL. Notably a number of "development tools" that sometimes hold us back from upgrading our CI/dev env (can't remember exactly which ones, but I definitely recall a few times where we could not upgrade to the latest version of some of those).
I think what really matters is that we have a "wide-enough" range of support. In a number of corporate environments, there are some old libraries and services that will have several years lags before they are supported in a newer version of Python. However, also I think (but this is more of an anecdotal evidence, but something that Jen's comment also hints about) - the rate of adoption of new versions of Python had significantly increased. There are multiple reasons for that - almost 100% backwards compatibility of 3.8 - 3.12 compared to relatively significant breaking changes in 3.5 - 3.8 played a major role. I can't remember for the last 3 years that we had a breaking change because of different Python versions (in the python language itself) - and if they were any, it was because of some obscure library feature that was usually very quickly fixed when found. If you look at the state of Python https://blog.jetbrains.com/pycharm/2024/12/the-state-of-python/ from December 2024, Python 3.10+ was already at > 75% back then. When you look at pypistats of Airflow https://pypistats.org/packages/apache-airflow it's quite a bit less: Python 3.10+ is ~60% ... But.... I think supporting EOL python is very important for low level libraries that might be used by many in different contexts. For example I know `pip` is not only looking at EOL but also at the usage stats. But In our case, I think we can be more of a "driver" of a change. If Airflow 3 supports only Python 3.10+, that might actually make people consider changing. I can't imagine many of those old libraries I mention to support 3.9 but not support 3.10. The backwards-compatibility problems mentioned in 3.5 - 3.8 could have held those libraries back to support 3.9, but if you have something supporting 3.9 - after initial ~6 months of catching up on all the libraries, moving to 3.10 should be pretty painless. Maybe...... Just maybe.... We could introduce a rule that we drop support for Python version 6 months before it reaches EOL ? We could also do "12 months" if we are a bit "aggressive". That would generally shift our dropping Python to ~ March time from ~ October time and that would nicely coincide with our plans to release Airflow 3 ? Yes. It's arbitrary, but for me 6 months is quite "enough" to see that "the end is coming soon" and also 6 months Is the fastest time of any upgrade I can imagine in any corporate environment that has a problem with such an upgrade. The "6 months" and "12 months" are both a bit of a magical number but we already use them in some places: > The version of the base OS image is the stable version of Debian. Airflow supports using all currently active stable versions - as soon as all Airflow dependencies support building, and we set up the CI pipeline for building and testing the OS version. Approximately 6 months before the end-of-regular support of a previous stable version of the OS, Airflow switches the images released to use the latest supported version of the OS. (that will be end of of 2027 BTW) > One of the important limitations of the Providers released by the community is that we introduce the limit of a minimum supported version of Airflow. The minimum version of Airflow is the ``MINOR`` version (2.4, 2.5 etc.) indicating that the providers might use features that appeared in this release. The default support timespan for the minimum version of Airflow (there could be justified exceptions) is that we increase the minimum Airflow version to the next MINOR release, when 12 months passed since the first release for the MINOR version of Airflow. (April 2025 is when we drop Airflow 2.9 support in providers) 6 months is also my "rule of thumb "for when I **think** we can start supporting a new version of Python as well after its releases, because that gives time to the vast majority of our libraries to support it. I was about to create a PR with Python 3.13 support and we can also see how we fare there. If we could introduce such policy, I'd be warmly supporting either a 6 months or 12 months shift from EOL. J. On Fri, Feb 7, 2025 at 9:22 PM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > I am also +1 for dropping 3.9 support early. That can save a lot of > boilerplate for __future__ for type hints as well. > > As well as Ash sais we would most probably anyway only support 3.0 and > 3.1 with 3.9 > > And I can say, migrating Python from 3.8 straight into 3.12 was only > generating one bug in our environment, else all was un-noticed. > > On 07.02.25 13:51, Ash Berlin-Taylor wrote: > > https://endoflife.date/python for a visual of the versions and their > support status for anyone following along. > > > > We do have a published get-out-of-jail card: > > > >> This policy is best-effort which means there may be situations where we > might terminate support earlier if circumstances require it. > > But yes, it shouldn’t be willy-nilly. The only sort of reason/rule I can > think of would be “At major version releases, we will support any Python > with more than 2 years left of security support” but that feels very > arbitrary. > > > > The module I want to use is https://docs.cadwyn.dev/ (which I’ve used > on other non OSS projects), but thinking about it a bit more, we are likely > only going to have one or two versions of the Task Execution API (one with > 3.0, and one with 3.1), maybe a couple of more in some patch releases but > that is unlikely and I can manually emulate Cadwyn for now and swap over to > it around the 3.2/3.3 timeframe. > > > > > > > > > >> On 7 Feb 2025, at 12:30, Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> I would be for it - but this should be accompanied with a clear > proposal of > >> the policy we are going to use forward for Airflow 3. We cannot make > such > >> "ad-hoc" decisions based on "I want to use that package". We need to > have > >> solid reasoning and clear indication for our users what kind of support > >> they can expect for different python versions. They cannot be surprised. > >> So far our policy was clear - we drop support when Python EOL is > reached. > >> That was easy to explain and justify. I don't have a concrete proposal, > but > >> maybe someone can come up with a rule that we codify and use from now > on. > >> > >> J. > >> > >> On Fri, Feb 7, 2025 at 1:25 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > >> > >>> Hi all, > >>> > >>> I have a proposal that we increase the minimum required python version > to > >>> 3.10, in a slight departure from our published python version req > >>> > https://github.com/apache/airflow?tab=readme-ov-file#support-for-python-and-kubernetes-versions > >>> > >>> As a reminder, Python 3.9 is already in security only fixes, and is > >>> supported only until October 2025, or for roughly 7 months after the > >>> release of Airflow 3.0. We’d be bringing forward the requirement on > Python > >>> 3.10 from (likely) Airflow 3.2 to 3.0. > >>> > >>> We discussed this briefly last July on the Airflow 3 dev calls and the > >>> conclusion then was to follow the our policy, but I would like to make > use > >>> of a python module in the Task Execution API server that only supports > >>> Python 3.10+. > >>> > >>> Pros of this: We’re more up to date, and I can use this module, rather > >>> than having to write something myself to handle API versioning > >>> Cons of this: it might make it harder for some users to update to 3.0 > as > >>> some users would also have to update the Python version before they can > >>> update the Airflow version. > >>> > >>> Looking at the analytics data we had previously collected (which only > gets > >>> data from 2.10.x so is imperfect to be sure), I can see that Python > 3.10+ > >>> accounts for about 77% of the data. So it’s certainly not nothing. (5% > are > >>> still on 3.8, and 17% are on 3.9) > >>> > >>> I’m really not sold one way or another on this, so I thought I’d > discuss > >>> it with the wider community. > >>> > >>> What do you all think? > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
