Thanks for spotting this and explaining it so clearly, appreciate it, Ash. Agreed, no objection. After your explanation, I realised the fault. It's more than just "pandas 3 doesn't work with xcom" - people could've been using pandas 3 functionally and not for xcom.
Forcing pandas 2 on people already running pandas 3 fine on 3.3.0 is worse than the rollback risk we were worried about. So let's revert 70791 and merge 70558 (after a review). The cross version error message is probably a separate issue or we fold it in 70558. Thanks & Regards, Amogh Desai On Tue, Aug 4, 2026 at 4:47 PM Ash Berlin-Taylor <[email protected]> wrote: > So my issue is 70791 is that it itself is (or will be) a regression on > 3.3.0 > > By limiting Pandas to < 3 in this change in a point release, we will end > up breaking any dag that was using Pandas 3.0 features (because 3.0.5 will > happily install with 3.3.0), but with the proposed version limitation, it > would get forcibly downgraded. Either this will result in unsatisfiable > constraints if the users has `pandas>=3.0`, or perhaps worse it would > silently downgrade. > > And as for the worry about downgrade breaking things, yes, but Pandas 3.0 > was released on 21st January 2026. By forcing a downgrade now we are > re-breaking anything people may have already dealt with this. To me it's > not really any different to "I changed my dag to push something new to > xcom, if I roll back the dag I can't use those XComs anymore”. > > In this regard Airflow should behave as a library, not an application — > and be lenient in the versions we work with. > > If we’d applied this version constraint 6 months ago/just after pandas 3.0 > was released it’s fine, but almost 7 months after the fact it seems > actively harmful for compat/expectations. > > I think we should revert 70791 and merge > https://github.com/apache/airflow/pull/70558/ in instead. If everyone > agrees/doesn’t disagree, then I can handle these changes as you are > travelling Jarek. Perhaps we need to extend it with. “When attempting to > load Pandas2 df when Pandas3 is the active version throw a nice error”. > TBD? (Unless anyone already knows how this will behave?) > > -ash > > > On 4 Aug 2026, at 08:04, Jarek Potiuk <[email protected]> wrote: > > > >> A (temporary) -1 on anything including > > https://github.com/apache/airflow/pull/70791 please — I want to catch > up > > with the logic and thinking there (I understand this was discussed on the > > dev call) > > > > FYI Ash - I think this is important to mention that it also impacts > Airlfow > > -> not only providers, so if we have other proposals - I think it would > be > > great to get it sorted before RC. I am flying in about few hours, and > then > > i will also be available "sometimes" so it would be great to know if we > > need to do some follow-ups - might delay 3.1.1 as currently there is a > > runtime check in Airflow preventing pandas3 to be used **when** > > serialization is used. > > > > > > J. > > > > > > On Tue, Aug 4, 2026 at 6:16 AM Steve Ahn <[email protected]> wrote: > > > >> +1 (non-binding) on airbyte 6.0.1rc1. > >> > >> Tested against the RC wheel on 3.3.0: a cancelled Airbyte sync now > yields a > >> "cancelled" event rather than falling through the success branch > (#69786). > >> > >> I also verified my changes in amazon 9.34.0rc1 (#69676, deferred > Redshift > >> statement > >> is cancelled on kill) and common.sql 2.1.0rc1 (#69736, value-check > >> tolerance no > >> longer inverts for negative expected values) — both behave correctly > >> against their > >> RC wheels. Not voting on those two since they carry #70791, which Ash > has > >> asked to > >> hold. > >> > >> -- > >> Steve Ahn > >> > >> On Mon, Aug 3, 2026 at 4:17 AM Jarek Potiuk <[email protected]> wrote: > >> > >>> Correction: (We discussed these at the dev call and in the > PRs/issues—not > >>> on the dev list, though.) Perhaps it's a good idea that we are doing > this > >>> now. But if you have any thoughts, perhaps it's better to discuss them > >> in a > >>> separate thread :). > >>> > >>> On Mon, Aug 3, 2026 at 1:15 PM Jarek Potiuk <[email protected]> wrote: > >>> > >>>> Sure. Take your time Ash. > >>>> > >>>> Just to give you more context (we can discuss this on Slack if you > have > >>>> questions or doubts, rather than here). > >>>> > >>>> I think there are no "perfect" solutions here due to how "pandas" is > >>> built > >>>> into our XCom processing and potential migration issues. There are > >>> **only** > >>>> trade-offs. We discussed a few of those at the devlist. I'm also not > >>>> entirely happy with what we came up with, but I think the issue here > is > >>>> choosing the "lesser evil". > >>>> > >>>> Unless of course we can find a solution that covers all potential > >> issues > >>>> here: > >>>> > >>>> a) risk of accidental migration issues to Pandas 3 by people without > >>>> realising that it breaks both XCom **and** incompatibilities with > >> Pandas > >>> 3 > >>>> b) risk of "no easy downgrade" path when people - even knowingly - > >>> migrate > >>>> to Pandas 3, only to realize later they need to downgrade. > >>>> c) risk of not supporting Pandas 3 "today" for those who need it > >>>> > >>>> We must weigh all these risks. The trade-off we developed considers > >> risks > >>>> (a) and (b) more impactful than (c). Our approach is to deliberately > >> add > >>>> support for Pandas 3 in Airflow 3.4 while temporarily mitigating (a) > >> and > >>>> (b). > >>>> But other trade-offs here are possible of course. > >>>> > >>>> J, > >>>> > >>>> > >>>> On Mon, Aug 3, 2026 at 12:33 PM Ash Berlin-Taylor <[email protected]> > >>> wrote: > >>>> > >>>>> A (temporary) -1 on anything including > >>>>> https://github.com/apache/airflow/pull/70791 please — I want to > >> catch > >>>>> up with the logic and thinking there (I understand this was discussed > >> on > >>>>> the dev call) > >>>>> > >>>>> -ash > >>>>> > >>>>> > >>>>>> On 3 Aug 2026, at 08:43, Andrew Chang <[email protected]> > >> wrote: > >>>>>> > >>>>>> Thanks for preparing the release, Jarek! > >>>>>> > >>>>>> +1 (non-binding) > >>>>>> > >>>>>> Tested apache-airflow-providers-microsoft-azure 14.1.0rc1 on > >> released > >>>>>> Airflow 3.3.0 with breeze --use-airflow-version. I focused on my > >>> change > >>>>>> #70301, which registers the wasb remote logging scheme. > >>>>>> > >>>>>> The wasb scheme resolves to WasbRemoteLogIO through provider > >> dispatch, > >>>>> and > >>>>>> the legacy discovery path is not called. from_config() builds the IO > >>>>> from > >>>>>> the [logging] section as expected. I also ran a real upload and read > >>>>>> round-trip against Azurite, and the log content matched. > >>>>>> > >>>>>> Best, > >>>>>> Andrew > >>>>>> > >>>>>> > >>>>>> > >>>>>> Jarek Potiuk <[email protected]> 於 2026年8月3日週一 上午8:37寫道： > >>>>>> > >>>>>>> Hey all, > >>>>>>> > >>>>>>> I have just cut the new wave Airflow Providers packages with > >> release > >>>>>>> preparation date 2026-08-01. This email is calling a vote on the > >>>>>>> release, > >>>>>>> which will last for 72 hours - which means that it will end on > >>>>>>> 2026-08-06 00:40 UTC and until 3 binding +1 votes have been > >> received. > >>>>>>> > >>>>>>> Consider this my (binding) +1. > >>>>>>> > >>>>>>> This wave contains 48 providers. There are no breaking changes in > >> any > >>>>>>> of them - all bumps are minor or patch level. > >>>>>>> > >>>>>>> Four of the providers are re-cuts of release candidates that were > >>>>>>> prepared in the 2026-07-22 wave but never published, so they are at > >>>>>>> rc2 rather than rc1: > >>>>>>> > >>>>>>> - apache-airflow-providers-clickhousedb 1.0.1rc2 > >>>>>>> - apache-airflow-providers-common-ai 0.7.0rc2 > >>>>>>> - apache-airflow-providers-google 22.3.0rc2 > >>>>>>> - apache-airflow-providers-openlineage 2.20.0rc2 > >>>>>>> > >>>>>>> For those four, the testing status issue carries over the check > >> marks > >>>>>>> from the previous wave, so only the genuinely new commits are left > >>>>>>> unchecked. > >>>>>>> > >>>>>>> Airflow Providers are available at: > >>>>>>> > >> https://dist.apache.org/repos/dist/dev/airflow/providers/2026-08-01 > >>>>>>> > >>>>>>> *apache_airflow_providers-2026-08-01-source.tar.gz* is the full > >>> source > >>>>>>> tarball of airflow repo - snapshot taken at the moment of > >> provider's > >>>>>>> release. > >>>>>>> > >>>>>>> *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the convenience > >>>>>>> python "sdist" distributions that we publish in PyPI > >>>>>>> > >>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the convenience > >> Python > >>>>>>> "wheel" distributions that we publish in PyPI. > >>>>>>> > >>>>>>> The test procedure for PMC members is described in > >>>>>>> > >>>>>>> > >>>>> > >>> > >> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-pmc-members > >>>>>>> > >>>>>>> The test procedure for and Contributors who would like to test this > >>> RC > >>>>>>> is described in: > >>>>>>> > >>>>>>> > >>>>> > >>> > >> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-contributors > >>>>>>> > >>>>>>> > >>>>>>> Public keys are available at: > >>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS > >>>>>>> > >>>>>>> Please vote accordingly: > >>>>>>> > >>>>>>> [ ] +1 approve > >>>>>>> [ ] +0 no opinion > >>>>>>> [ ] -1 disapprove with the reason > >>>>>>> > >>>>>>> Only votes from PMC members are binding, but members of the > >> community > >>>>> are > >>>>>>> encouraged to test the release and vote with "(non-binding)". > >>>>>>> > >>>>>>> Please note that the version number excludes the 'rcX' string. > >>>>>>> This will allow us to rename the artifact without modifying > >>>>>>> the artifact checksums when we actually release it. > >>>>>>> > >>>>>>> The status of testing the providers by the community is kept here: > >>>>>>> https://github.com/apache/airflow/issues/70953 > >>>>>>> > >>>>>>> The issue is also the easiest way to see important PRs included in > >>> the > >>>>>>> RC candidates. > >>>>>>> Detailed changelog for the providers will be published in the > >>>>>>> documentation after the > >>>>>>> RC candidates are released. > >>>>>>> > >>>>>>> You can find the RC packages in PyPI following these links: > >>>>>>> > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-airbyte/6.0.1rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-alibaba/3.4.0rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-amazon/9.34.0rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-anthropic/0.2.1rc1/ > >>>>>>> > >>>>> > >>> > https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.12.2rc1/ > >>>>>>> > >>>>> > >> https://pypi.org/project/apache-airflow-providers-apache-hive/9.6.1rc1/ > >>>>>>> > >>>>> > >>> > >> > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.15.1rc1/ > >>>>>>> > >>>>> > >> https://pypi.org/project/apache-airflow-providers-apache-livy/4.6.0rc1/ > >>>>>>> > >>>>> > >>> > https://pypi.org/project/apache-airflow-providers-apache-spark/6.3.1rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-arangodb/2.9.6rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-celery/3.23.1rc1/ > >>>>>>> > >>>>> > >>> > https://pypi.org/project/apache-airflow-providers-clickhousedb/1.0.1rc2/ > >>>>>>> > >>>>>>> > >>>>> > >>> > >> > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/10.21.0rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-cohere/1.6.7rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-common-ai/0.7.0rc2/ > >>>>>>> > >>>>> > >>> > >> > https://pypi.org/project/apache-airflow-providers-common-compat/1.18.0rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-common-sql/2.1.0rc1/ > >>>>>>> > >>>>> > >> https://pypi.org/project/apache-airflow-providers-databricks/7.18.1rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-dbt-cloud/4.9.3rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-docker/4.5.9rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-edge3/4.3.0rc1/ > >>>>>>> > >>>>> > >>> > >> > https://pypi.org/project/apache-airflow-providers-elasticsearch/6.9.0rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-exasol/4.10.5rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-fab/3.8.0rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-git/0.4.2rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-google/22.3.0rc2/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-hashicorp/4.8.0rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-imap/3.12.1rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-keycloak/0.9.0rc1/ > >>>>>>> > >>>>>>> > >>>>> > >>> > >> > https://pypi.org/project/apache-airflow-providers-microsoft-azure/14.1.0rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/3.12.1rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-openai/1.8.2rc1/ > >>>>>>> > >>>>> > >>> > https://pypi.org/project/apache-airflow-providers-openlineage/2.20.0rc2/ > >>>>>>> > >>>>> > >> https://pypi.org/project/apache-airflow-providers-opensearch/1.11.2rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-papermill/3.13.2rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-postgres/7.0.1rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-presto/5.12.1rc1/ > >>>>>>> > >>>>> > >> https://pypi.org/project/apache-airflow-providers-salesforce/5.14.2rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/6.0.1rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-smtp/3.0.3rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-snowflake/6.16.0rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/6.0.1rc1/ > >>>>>>> > >>> https://pypi.org/project/apache-airflow-providers-standard/1.17.0rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-telegram/4.9.6rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-teradata/3.6.2rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-trino/6.6.1rc1/ > >>>>>>> > >> https://pypi.org/project/apache-airflow-providers-weaviate/3.4.1rc1/ > >>>>>>> https://pypi.org/project/apache-airflow-providers-yandex/4.5.1rc1/ > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Jarek Potiuk > >>>>>>> > >>>>>>> > >> --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>> For additional commands, e-mail: [email protected] > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >> > >
