As I user of Airflow I have ended up implementing this for my environments:
pip install apache-airflow[all or select-extras] -c {official constraints} >
smaller-run-time-constraints.txt
pip-compile {all my requirements including development}.txt -c
smaller-run-time-constraints.txt
Exactly to remove the dev requirements airflow has from interfering with our
own dev requirements. Though this approach notably doesn't change any of the
used non-dev requirements, it's not clear to me (and I'm probably just missing
something) why your diff includes non-dev requirements.
Damian
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, July 22, 2025 11:04 AM
To: dev@airflow.apache.org
Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Hello here,
*Tl;DR; We had an interesting discussion on Slack today and I wanted to bring
it here to discuss what we should do - whether we should keep devel
dependencies in our PyPI constraints or not.*
*A bit of context:*
In our CI we generate several types of constraints. one of them are "source"
constraints that we use during our CI testing - including all our testing
tools, mypy, ruff etc., and another type is "PyPI" constraint - which is more
<What should be the "golden" versions of dependencies when you install Airflow
using `pip install` for production>.
The main difference between the two (roughly) is that "source constraints"
only uses dependencies from "branch tip" ("main", "v3-0-test"), where "PyPI
constraints" uses "Airflow core" branch-tip dependencies - but all the
providers are installed from PyPI. There are few nuances when some packages are
being released but basically - for example for Airflow 3.0.3, "PyPI"
constraints is "what is the current set of dependencies and providers when we
install airflow from PyPI at the moment of release".
So target for "source" constraints are "Airflow contributors" where target for
"PyPI constraints" are "Airflow users".
*The problem*
In the slack discussion
https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 - slack
user @dashmug (sorry - do not know name) noticed that the "PyPI"
constraints also contain Airflow development dependencies listed (like ruff or
mypy). That made me think a bit and I quickly came up with PR that could "fix"
it - i.e. only keep the "real" constraints in.
https://github.com/apache/airflow/pull/53631 - it's as easy as adding the
`--exact` (and `--strict` for verification) flag to the `uv pip install`
command we use.
I agree it's a bit confusing to have the devel dependencies there - it is not
the "intention", but there is very little problem with it - the fact that mypy
or ruff are listed with a version in constraints, does not mean it must be
installed - constraints mainly say "if you install this, it should be this
version" . And there is no problem whatsoever (and it is even recommended by
us) to have "airflow with its deps" installed with constraints, and then
anything else you need to install or update - without. So other than potential
confusion, it usually does not cause harm.
Now... It might seem that removing devel deps has no side effects and we should
**just do it** - but it's not that easy. Main problem is that those devel
dependencies might "hold" other dependencies (directly or
transitively) from being upgraded - and when we remove the devel dependencies
and do the resolution again - we might have a DIFFERENT set of dependencies,
simply because removing the devel dependencies, might "free"
the other deps and they might get upgraded. Yeah - I know it's not obvious for
someone who did not spend 3 years solving dependency issues :).
And the problem with it is that because we need those devel deps to test things
- the upgraded dependencies might have never been tested in our CI - because we
never upgraded to later versions. It's a bit of Heisenberg-effect. By using
test tools we are changing the state of the tested thing. Currently our "PyPI"
constraints reflect the state of Airflow "when the testing tools are installed"
- and we are generally not able to test the state of Airflow "without the
testing tools" - by the sheer fact that we do not have those tools :)
Just to illustrate the case - you can see what happens when we removed devel
deps:
https://github.com/apache/airflow/actions/runs/16444553234 - in summary you can
see a colored version that is a bit clearer but I dumped the diff below from an
example - Python 3.10 case. Left side < is "what was with testing tools" and >
is without them. You can see a LOT of test tools removed (pytest, ruff and
others - including dependencies used only by those tools)
But also you can see (likely some of the devel/test deps limited those)
* Authlib - upgraded from 1.3.1 to 1.6.1
* Google - genai upgraded from 1.2.0 to 1.20.0
* Httpx - upgraded from 0.27.0 to 0.28.1
* Pinotdb - upgraded from 5.6.0 to 5.7.0
* Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as it
has just been released and new version was picked)
* validators - upgraded form 0.34.0 to 0.35.0
* weaviate-client - upgraded from 4.9.0 to 4.16.3
* websockets - upgraded from 14.2 to 15.0.1
This means that all the "upgraded" packages above have not been - likely ever
tested in our CI.
And when we put them in "versioned" constraints, it might mean that they will
be obviously failing.
So ... there is a risk connected with just generating the "non-devel"
constraints without additional tests. But that also means that the user who
does not use constraints might get those versions anyway - default behaviour of
`pip install airflow` will be to install those upgraded versions above if you
do not use constraints. But also that is no different than installing a
released airflow version a few weeks or months after release without
constraints - there will be tens of packages that were released since our
release and installing airflow without constraints will pick those.
However - when we manually install RC candidates - say by running airflow from
rc1 image, we will use those "upgraded" dependencies - so some "obvious"
problems might be caught during the rc manual testing - where you do not use
test tools but run airflow "as if" it was installed in production.
*The solution ? *
There are few ways we can proceed with it:
a) leave it as is - no big harm, potential confusion, by people who see it
- potential differences
b) remove devel following my PR and take the risk that we never tested (in
CI) the dependencies we set in the constraints. That's a bit risky, but we can
always reactively downgrade some of those packages after the fact if users will
start reporting the issues
c) potentially most complex thing - trying to remove devel dependencies
somewhat combining the two - for example removing all the deps that would be
removed in b) from a) - but that has also some edge cases - when different
versions of dependencies can add or upgrade their dependencies, we are very
likely to occasionally remove or add too much.
I wonder (for those who managed to read that far - likely not many - what do
you think here? What would be the best option ? Maybe there are other options I
have not thought about ?
J
-----
The diff for Python 3.10
34c34
< Authlib==1.3.1
---
> Authlib==1.6.1
64d63
< Sphinx==8.2.3
78d76
< aioresponses==0.7.8
82d79
< alabaster==1.0.0
198d194
< arro3-core==0.5.1
202d197
< astroid==3.3.11
211,212d205
< aws-sam-translator==1.99.0
< aws-xray-sdk==2.14.0
258,259d250
< cfgv==3.4.0
< cfn-lint==1.38.0
262d252
< checksumdir==1.2.0
277d266
< coverage==7.9.2
291d279
< deltalake==1.1.0
293d280
< diagrams==0.24.4
301,302d287
< docutils==0.21.2
< duckdb==1.3.2
308d292
< eralchemy2==1.4.1
311d294
< execnet==2.1.1
321,322d303
< flit==3.12.0
< flit_core==3.12.0
386c367
< google-genai==1.2.0
---
> google-genai==1.20.0
389d369
< graphql-core==3.2.6
398d377
< grpcio-tools==1.62.3
403,404d381
< hatch==1.14.1
< hatchling==1.27.0
413c390
< httpx==0.27.0
---
> httpx==0.28.1
418d394
< hyperlink==21.0.0
421,423d396
< icdiff==2.0.7
< id==1.5.0
< identify==2.6.12
426d398
< imagesize==1.4.1
430d401
< incremental==24.7.2
433,435d403
< iniconfig==2.1.0
< inputimeout==1.0.4
< ipdb==0.13.13
449d416
< joserfc==1.2.2
451d417
< jsonpatch==1.33
455,456d420
< jsonpointer==3.0.0
< jsonschema-path==0.3.4
463d426
< kerberos==1.3.1
465,466d427
< keyrings.alt==5.0.2
< kgb==7.2
494,495d454
< mmh3==5.1.0
< mongomock==4.3.0
497,498d455
< moto==5.1.8
< mpmath==1.3.0
508,512d464
< mypy-boto3-appflow==1.39.0
< mypy-boto3-rds==1.39.1
< mypy-boto3-redshift-data==1.39.0
< mypy-boto3-s3==1.39.5
< mypy==1.17.0
514c466
< mysql-connector-python==9.3.0
---
> mysql-connector-python==9.4.0
521,523d472
< networkx==3.5
< nh3==0.3.0
< nodeenv==1.9.1
528,529d476
< openapi-schema-validator==0.6.3
< openapi-spec-validator==0.7.2
560d506
< pathable==0.4.4
563d508
< pdbr==0.9.2
569,570c514
< pinotdb==5.6.0
< pipdeptree==2.28.0
---
> pinotdb==5.7.0
574d517
< plyvel==1.5.1
577,579d519
< pprintpp==0.4.0
< pre-commit-uv==4.1.4
< pre_commit==4.2.0
592d531
< py-partiql-parser==0.6.1
608d546
< pyenchant==3.2.2
610,611d547
< pygraphviz==1.14
< pyiceberg==0.9.1
622,632d557
< pytest-asyncio==0.25.0
< pytest-cov==6.2.1
< pytest-custom-exit-code==0.3.0
< pytest-icdiff==0.9
< pytest-instafail==0.5.0
< pytest-mock==3.14.1
< pytest-rerunfailures==15.1
< pytest-timeouts==1.2.1
< pytest-unordered==0.7.0
< pytest-xdist==3.8.0
< pytest==8.4.1
642d566
< python-on-whales==0.78.0
652d575
< readme_renderer==44.0
659d581
< requests-mock==1.12.1
664,665d585
< responses==0.25.7
< restructuredtext_lint==1.4.0
667,668d586
< rfc3339-validator==0.1.4
< rfc3986==2.0.0
670d587
< rich-click==1.8.9
674d590
< roman-numerals-py==3.1.0
679d594
< ruff==0.12.3
688d602
< semver==3.0.4
690,691c604
< sentinels==1.0.0
< sentry-sdk==2.33.1
---
> sentry-sdk==2.33.2
703d615
< snowballstemmer==3.0.1
709,725d620
< sphinx-argparse==0.5.2
< sphinx-autoapi==3.6.0
< sphinx-autobuild==2024.10.3
< sphinx-copybutton==0.5.2
< sphinx-jinja==2.0.2
< sphinx-rtd-theme==3.0.2
< sphinx_design==0.6.1
< sphinxcontrib-applehelp==2.0.0
< sphinxcontrib-devhelp==2.0.0
< sphinxcontrib-htmlhelp==2.1.0
< sphinxcontrib-httpdomain==1.8.1
< sphinxcontrib-jquery==4.1
< sphinxcontrib-jsmath==1.0.1
< sphinxcontrib-qthelp==2.0.0
< sphinxcontrib-redoc==1.6.0
< sphinxcontrib-serializinghtml==2.0.0
< sphinxcontrib-spelling==8.0.1
737d631
< strictyaml==1.7.3
740d633
< sympy==1.14.0
752d644
< time-machine==2.16.0
755d646
< tomli_w==1.2.0
758d648
< towncrier==24.8.0
762,763d651
< trove-classifiers==2025.5.9.12
< twine==6.1.0
765,774d652
< types-Deprecated==1.2.15.20250304
< types-Markdown==3.8.0.20250708
< types-PyMySQL==1.1.0.20250711
< types-PyYAML==6.0.12.20250516
< types-aiofiles==24.1.0.20250708
< types-certifi==2021.10.8.3
< types-cffi==1.17.0.20250523
< types-croniter==6.0.0.20250626
< types-docutils==0.21.0.20250722
< types-paramiko==3.5.0.20250708
776,778d653
< types-pyOpenSSL==24.1.0.20240722
< types-python-dateutil==2.9.0.20250708
< types-python-slugify==8.0.2.20240310
780d654
< types-redis==4.6.0.20241004
782,784d655
< types-setuptools==80.9.0.20250529
< types-tabulate==0.9.0.20241207
< types-toml==0.10.8.20240310
794d664
< userpath==1.9.2
799c669
< validators==0.34.0
---
> validators==0.35.0
806c676
< weaviate-client==4.9.6
---
> weaviate-client==4.16.3
809c679
< websockets==14.2
---
> websockets==15.0.1
815d684
< yamllint==1.37.1
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of companies.
Strike is a technology solutions provider, and is not a broker or dealer and
does not transact any securities related business directly whatsoever. This
communication is the property of Strike and its affiliates, and does not
constitute an offer to sell or the solicitation of an offer to buy any security
in any jurisdiction. It is intended only for the person to whom it is addressed
and may contain information that is privileged, confidential, or otherwise
protected from disclosure. Distribution or copying of this communication, or
the information contained herein, by anyone other than the intended recipient
is prohibited. If you have received this communication in error, please
immediately notify Strike at i...@striketechnologies.com, and delete and
destroy any copies hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are
intended solely for the addressee. This transmission is covered by the
Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information
contained in this transmission is confidential in nature and protected from
further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338
(1999), and may be subject to attorney-client or other legal privilege. Your
use or disclosure of this information for any purpose other than that intended
by its transmittal is strictly prohibited, and may subject you to fines and/or
penalties under federal and state law. If you are not the intended recipient of
this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction
to the sender via return transmittal.
