FYI. Python 3.12 is now fully back - ready for 2.9.0. We even got some of the providers that were excluded, back in:
* databricks is back in after the issue we had was determined as "Python coverage" tool not working well yet for Python 3.12 (we disabled coverage for Python 3.12 now) * cassandra is about to be re-enabled https://github.com/apache/airflow/pull/38314 after they responded to our issue and released a binary driver today with libev support compiled in. The two remaining providers that do not work for Python 3.12 are papermill and apache-beam. * Papermill - technically support is merged but we are waiting for the new release of their package https://github.com/nteract/papermill/pull/771 - I ping them today and there is some reaction, let's see. * Beam (traditionally they have most complex dependencies and are lagging behind) - issue here and I inquired them when support is planned https://github.com/apache/beam/issues/29149 Just as a reminder: Airflow 2.9.0 default images will be *Python 3.12* - according to the rule we agreed here: https://lists.apache.org/thread/0oxnvct24xlqsj76z42w2ttw2d043oy3 - this means that papermill and beam will not be installable in "apache/airflow" image or "apache/airflow:2.9.0" - users will have to use "apache/airflow:2.9.0-python3.11" for example. J. On Mon, Mar 11, 2024 at 10:02 AM Jarek Potiuk <ja...@potiuk.com> wrote: > There were strange errors in the canary build which was not showing up in > the canary - so I need to revert and investigate. > > On Mon, Mar 11, 2024 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> And merged :) >> >> On Sun, Mar 10, 2024 at 6:52 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> Hey here, >>> >>> FINALLY - after 5 months (a little more than I initially anticipated - I >>> thought it will take 3-4 months) we can finally add Python 3.12. >>> >>> The time is about right - a day that we plan to cut the >>> Airflow `v2-9-test` branch !! >>> >>> Thanks to Bolke for the final push on the Universal Pathlib migration >>> (and Andreas Poehlmann who is the Universal Pathlib Maintainer). That was >>> the final blocker that kept us from adding the support for 3.12 !!!! >>> >>> The PR is all "GREEN" https://github.com/apache/airflow/pull/36755 and >>> waits for reviews :) >>> >>> ------------------- >>> >>> A bit more details on 3.12 support: >>> >>> >>> While Airflow fully supports 3.12, we had to exclude 3 providers from >>> Python 3.12 support (temporarily - all of them will be included back when >>> they support 3.12): >>> >>> * *apache.beam* (beam has no 3.12 support yet) and looking at the state >>> of the ticket here we will wait a bit more >>> https://github.com/apache/beam/issues/29149 >>> >>> * *apache.cassandra* (the default setup for cassandra does not work >>> with 3.12 and requires custom compilation) - they are working on releasing >>> a build that works (the problem is their binary driver does not have the >>> right library compiled in). Should be fixed in the next cassandra-driver >>> release (3.30.0). Either because they fix their build environment to >>> compile the libev support in >>> https://datastax-oss.atlassian.net/browse/PYTHON-1378 or when they >>> promote asyncio reactor to be "production ready": >>> https://datastax-oss.atlassian.net/browse/PYTHON-1375 >>> >>> * *papermill *(not sure if they will release a new version with 3.12 >>> support any time soon. The fix is already merged >>> https://github.com/nteract/papermill/pull/771 - but the last release of >>> papermill happened in November, and there is not much activity in the >>> project. >>> >>> All the other providers seem to happily work in the Python 3.12 >>> environment. >>> >>> J. >>> >>>
