Hello here,
*Tl;DR; We had an interesting discussion on Slack today and I wanted to
bring it here to discuss what we should do - whether we should keep devel
dependencies in our PyPI constraints or not.*
*A bit of context:*
In our CI we generate several types of constraints. one of them are
"source" constraints that we use during our CI testing - including all our
testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
which is more <What should be the "golden" versions of dependencies when
you install Airflow using `pip install` for production>.
The main difference between the two (roughly) is that "source constraints"
only uses dependencies from "branch tip" ("main", "v3-0-test"), where "PyPI
constraints" uses "Airflow core" branch-tip dependencies - but all the
providers are installed from PyPI. There are few nuances when some packages
are being released but basically - for example for Airflow 3.0.3, "PyPI"
constraints is "what is the current set of dependencies and providers when
we install airflow from PyPI at the moment of release".
So target for "source" constraints are "Airflow contributors" where target
for "PyPI constraints" are "Airflow users".
*The problem*
In the slack discussion
https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 -
slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
constraints also contain Airflow development dependencies listed (like ruff
or mypy). That made me think a bit and I quickly came up with PR that could
"fix" it - i.e. only keep the "real" constraints in.
https://github.com/apache/airflow/pull/53631 - it's as easy as adding the
`--exact` (and `--strict` for verification) flag to the `uv pip install`
command we use.
I agree it's a bit confusing to have the devel dependencies there - it is
not the "intention", but there is very little problem with it - the fact
that mypy or ruff are listed with a version in constraints, does not mean
it must be installed - constraints mainly say "if you install this, it
should be this version" . And there is no problem whatsoever (and it is
even recommended by us) to have "airflow with its deps" installed with
constraints, and then anything else you need to install or update -
without. So other than potential confusion, it usually does not cause harm.
Now... It might seem that removing devel deps has no side effects and we
should **just do it** - but it's not that easy. Main problem is that those
devel dependencies might "hold" other dependencies (directly or
transitively) from being upgraded - and when we remove the devel
dependencies and do the resolution again - we might have a DIFFERENT set of
dependencies, simply because removing the devel dependencies, might "free"
the other deps and they might get upgraded. Yeah - I know it's not obvious
for someone who did not spend 3 years solving dependency issues :).
And the problem with it is that because we need those devel deps to test
things - the upgraded dependencies might have never been tested in our CI -
because we never upgraded to later versions. It's a bit of
Heisenberg-effect. By using test tools we are changing the state of the
tested thing. Currently our "PyPI" constraints reflect the state of Airflow
"when the testing tools are installed" - and we are generally not able to
test the state of Airflow "without the testing tools" - by the sheer fact
that we do not have those tools :)
Just to illustrate the case - you can see what happens when we removed
devel deps:
https://github.com/apache/airflow/actions/runs/16444553234 - in summary you
can see a colored version that is a bit clearer but I dumped the diff below
from an example - Python 3.10 case. Left side < is "what was with testing
tools" and > is without them. You can see a LOT of test tools removed
(pytest, ruff and others - including dependencies used only by those tools)
But also you can see (likely some of the devel/test deps limited those)
* Authlib - upgraded from 1.3.1 to 1.6.1
* Google - genai upgraded from 1.2.0 to 1.20.0
* Httpx - upgraded from 0.27.0 to 0.28.1
* Pinotdb - upgraded from 5.6.0 to 5.7.0
* Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
it has just been released and new version was picked)
* validators - upgraded form 0.34.0 to 0.35.0
* weaviate-client - upgraded from 4.9.0 to 4.16.3
* websockets - upgraded from 14.2 to 15.0.1
This means that all the "upgraded" packages above have not been - likely
ever tested in our CI.
And when we put them in "versioned" constraints, it might mean that they
will be obviously failing.
So ... there is a risk connected with just generating the "non-devel"
constraints without additional tests. But that also means that the user who
does not use constraints might get those versions anyway - default
behaviour of `pip install airflow` will be to install those upgraded
versions above if you do not use constraints. But also that is no different
than installing a released airflow version a few weeks or months after
release without constraints - there will be tens of packages that were
released since our release and installing airflow without constraints will
pick those.
However - when we manually install RC candidates - say by running airflow
from rc1 image, we will use those "upgraded" dependencies - so some
"obvious" problems might be caught during the rc manual testing - where you
do not use test tools but run airflow "as if" it was installed in
production.
*The solution ? *
There are few ways we can proceed with it:
a) leave it as is - no big harm, potential confusion, by people who see it
- potential differences
b) remove devel following my PR and take the risk that we never tested (in
CI) the dependencies we set in the constraints. That's a bit risky, but we
can always reactively downgrade some of those packages after the fact if
users will start reporting the issues
c) potentially most complex thing - trying to remove devel dependencies
somewhat combining the two - for example removing all the deps that would
be removed in b) from a) - but that has also some edge cases - when
different versions of dependencies can add or upgrade their dependencies,
we are very likely to occasionally remove or add too much.
I wonder (for those who managed to read that far - likely not many - what
do you think here? What would be the best option ? Maybe there are other
options I have not thought about ?
J
-----
The diff for Python 3.10
34c34
< Authlib==1.3.1
---
> Authlib==1.6.1
64d63
< Sphinx==8.2.3
78d76
< aioresponses==0.7.8
82d79
< alabaster==1.0.0
198d194
< arro3-core==0.5.1
202d197
< astroid==3.3.11
211,212d205
< aws-sam-translator==1.99.0
< aws-xray-sdk==2.14.0
258,259d250
< cfgv==3.4.0
< cfn-lint==1.38.0
262d252
< checksumdir==1.2.0
277d266
< coverage==7.9.2
291d279
< deltalake==1.1.0
293d280
< diagrams==0.24.4
301,302d287
< docutils==0.21.2
< duckdb==1.3.2
308d292
< eralchemy2==1.4.1
311d294
< execnet==2.1.1
321,322d303
< flit==3.12.0
< flit_core==3.12.0
386c367
< google-genai==1.2.0
---
> google-genai==1.20.0
389d369
< graphql-core==3.2.6
398d377
< grpcio-tools==1.62.3
403,404d381
< hatch==1.14.1
< hatchling==1.27.0
413c390
< httpx==0.27.0
---
> httpx==0.28.1
418d394
< hyperlink==21.0.0
421,423d396
< icdiff==2.0.7
< id==1.5.0
< identify==2.6.12
426d398
< imagesize==1.4.1
430d401
< incremental==24.7.2
433,435d403
< iniconfig==2.1.0
< inputimeout==1.0.4
< ipdb==0.13.13
449d416
< joserfc==1.2.2
451d417
< jsonpatch==1.33
455,456d420
< jsonpointer==3.0.0
< jsonschema-path==0.3.4
463d426
< kerberos==1.3.1
465,466d427
< keyrings.alt==5.0.2
< kgb==7.2
494,495d454
< mmh3==5.1.0
< mongomock==4.3.0
497,498d455
< moto==5.1.8
< mpmath==1.3.0
508,512d464
< mypy-boto3-appflow==1.39.0
< mypy-boto3-rds==1.39.1
< mypy-boto3-redshift-data==1.39.0
< mypy-boto3-s3==1.39.5
< mypy==1.17.0
514c466
< mysql-connector-python==9.3.0
---
> mysql-connector-python==9.4.0
521,523d472
< networkx==3.5
< nh3==0.3.0
< nodeenv==1.9.1
528,529d476
< openapi-schema-validator==0.6.3
< openapi-spec-validator==0.7.2
560d506
< pathable==0.4.4
563d508
< pdbr==0.9.2
569,570c514
< pinotdb==5.6.0
< pipdeptree==2.28.0
---
> pinotdb==5.7.0
574d517
< plyvel==1.5.1
577,579d519
< pprintpp==0.4.0
< pre-commit-uv==4.1.4
< pre_commit==4.2.0
592d531
< py-partiql-parser==0.6.1
608d546
< pyenchant==3.2.2
610,611d547
< pygraphviz==1.14
< pyiceberg==0.9.1
622,632d557
< pytest-asyncio==0.25.0
< pytest-cov==6.2.1
< pytest-custom-exit-code==0.3.0
< pytest-icdiff==0.9
< pytest-instafail==0.5.0
< pytest-mock==3.14.1
< pytest-rerunfailures==15.1
< pytest-timeouts==1.2.1
< pytest-unordered==0.7.0
< pytest-xdist==3.8.0
< pytest==8.4.1
642d566
< python-on-whales==0.78.0
652d575
< readme_renderer==44.0
659d581
< requests-mock==1.12.1
664,665d585
< responses==0.25.7
< restructuredtext_lint==1.4.0
667,668d586
< rfc3339-validator==0.1.4
< rfc3986==2.0.0
670d587
< rich-click==1.8.9
674d590
< roman-numerals-py==3.1.0
679d594
< ruff==0.12.3
688d602
< semver==3.0.4
690,691c604
< sentinels==1.0.0
< sentry-sdk==2.33.1
---
> sentry-sdk==2.33.2
703d615
< snowballstemmer==3.0.1
709,725d620
< sphinx-argparse==0.5.2
< sphinx-autoapi==3.6.0
< sphinx-autobuild==2024.10.3
< sphinx-copybutton==0.5.2
< sphinx-jinja==2.0.2
< sphinx-rtd-theme==3.0.2
< sphinx_design==0.6.1
< sphinxcontrib-applehelp==2.0.0
< sphinxcontrib-devhelp==2.0.0
< sphinxcontrib-htmlhelp==2.1.0
< sphinxcontrib-httpdomain==1.8.1
< sphinxcontrib-jquery==4.1
< sphinxcontrib-jsmath==1.0.1
< sphinxcontrib-qthelp==2.0.0
< sphinxcontrib-redoc==1.6.0
< sphinxcontrib-serializinghtml==2.0.0
< sphinxcontrib-spelling==8.0.1
737d631
< strictyaml==1.7.3
740d633
< sympy==1.14.0
752d644
< time-machine==2.16.0
755d646
< tomli_w==1.2.0
758d648
< towncrier==24.8.0
762,763d651
< trove-classifiers==2025.5.9.12
< twine==6.1.0
765,774d652
< types-Deprecated==1.2.15.20250304
< types-Markdown==3.8.0.20250708
< types-PyMySQL==1.1.0.20250711
< types-PyYAML==6.0.12.20250516
< types-aiofiles==24.1.0.20250708
< types-certifi==2021.10.8.3
< types-cffi==1.17.0.20250523
< types-croniter==6.0.0.20250626
< types-docutils==0.21.0.20250722
< types-paramiko==3.5.0.20250708
776,778d653
< types-pyOpenSSL==24.1.0.20240722
< types-python-dateutil==2.9.0.20250708
< types-python-slugify==8.0.2.20240310
780d654
< types-redis==4.6.0.20241004
782,784d655
< types-setuptools==80.9.0.20250529
< types-tabulate==0.9.0.20241207
< types-toml==0.10.8.20240310
794d664
< userpath==1.9.2
799c669
< validators==0.34.0
---
> validators==0.35.0
806c676
< weaviate-client==4.9.6
---
> weaviate-client==4.16.3
809c679
< websockets==14.2
---
> websockets==15.0.1
815d684
< yamllint==1.37.1
