Thanks Jarek, I am fine with option b as far as if we find anything while testing we can downgrade.
Pavan On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Would love to hear more if anyone has an opinion :). Yep. It's an advanced > topic and I can make arbitrary decision :) ... > > I am thinking now that it should be quite "OK" to accept those "potentially > untested" versions - especially that they will be tested during manual > testing. Unless someone objects i will likely update PR with some changelog > / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS > soon if I don't hear any opposition. > > J. > > > On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > Exactly to remove the dev requirements airflow has from interfering > with > > our own dev requirements. Though this approach notably doesn't change any > > of the used non-dev requirements, it's not clear to me (and I'm probably > > just missing something) why your diff includes non-dev requirements. > > > > The thing is that I do not "remove" devel features - because due to > > transitive dependencies I would have to go deep down the rabbit hole > which > > dependencies are "devel" and which not. So what I do, I ask `uv` to do > the > > job for me and repeat the resolution but with the `--exact` flag > (combined > > with the already used `--resolution higest` and `--no-sources`). > > > > Apparently in this case `uv` will sometimes decide that it's ok to bump > > some dependencies that are left after initial resolution. The thing is > that > > uv will not install some versions of some packages if they are > conflicting > > with already installed packages. What I believe `uv` does when I use the > *--exact > > *flag of `uv pip install` it "fixes" the initial resolution. It finds out > > what is needed, removes those packages that are not needed, and then it > > resolves again and improves the resolution (that would be my guess at > least > > - it could be done slightly differently and in a single step I guess. > > > > Essentially the difference between the left and right side is: > > > > a) install absolutely everything from the workspace > > > > b) do this: > > left: uv pip install --no-sources 'apache-airflow[all]' > > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl > > --reinstall --resolution highest --find-links file:///dist > > right: uv pip install --no-sources *--exact --strict* > > 'apache-airflow[all]' 'apache-airflow-core[all]' apache-airflow-task-sdk > > ./airflow-ctl --reinstall --resolution highest --find-links file:///dist > > > > In the current state content of the fille://dist should be the same in > > both cases: > > > > apache-airflow-providers-edge3 (1.1.2): wheel - > > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl > > apache-airflow-providers-keycloak (1.0.0): wheel - > > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl > > apache-airflow-providers-snowflake (6.5.1): wheel - > > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl > > > > This is a bit of in-process quirk of the "PyPI" resolution - those are > the > > three packages that we have updated versions of in the sources, but they > > are not released yet. But they should be essentially the same in both > > cases. So the difference between left and right is `--exact --strict` > > (strict I added to make sure that we are not removing something that is > > still needed). > > > > J. > > > > > > > > > > > > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw < > ds...@striketechnologies.com> > > wrote: > > > >> As I user of Airflow I have ended up implementing this for my > >> environments: > >> > >> pip install apache-airflow[all or select-extras] -c {official > >> constraints} > smaller-run-time-constraints.txt > >> pip-compile {all my requirements including development}.txt -c > >> smaller-run-time-constraints.txt > >> > >> Exactly to remove the dev requirements airflow has from interfering with > >> our own dev requirements. Though this approach notably doesn't change > any > >> of the used non-dev requirements, it's not clear to me (and I'm probably > >> just missing something) why your diff includes non-dev requirements. > >> > >> Damian > >> > >> -----Original Message----- > >> From: Jarek Potiuk <ja...@potiuk.com> > >> Sent: Tuesday, July 22, 2025 11:04 AM > >> To: dev@airflow.apache.org > >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ? > >> > >> Hello here, > >> > >> > >> *Tl;DR; We had an interesting discussion on Slack today and I wanted to > >> bring it here to discuss what we should do - whether we should keep > devel > >> dependencies in our PyPI constraints or not.* > >> > >> *A bit of context:* > >> In our CI we generate several types of constraints. one of them are > >> "source" constraints that we use during our CI testing - including all > our > >> testing tools, mypy, ruff etc., and another type is "PyPI" constraint - > >> which is more <What should be the "golden" versions of dependencies when > >> you install Airflow using `pip install` for production>. > >> > >> The main difference between the two (roughly) is that "source > constraints" > >> only uses dependencies from "branch tip" ("main", "v3-0-test"), where > >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all > >> the providers are installed from PyPI. There are few nuances when some > >> packages are being released but basically - for example for Airflow > 3.0.3, > >> "PyPI" > >> constraints is "what is the current set of dependencies and providers > >> when we install airflow from PyPI at the moment of release". > >> > >> So target for "source" constraints are "Airflow contributors" where > >> target for "PyPI constraints" are "Airflow users". > >> > >> > >> *The problem* > >> In the slack discussion > >> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 > >> - slack user @dashmug (sorry - do not know name) noticed that the > "PyPI" > >> constraints also contain Airflow development dependencies listed (like > >> ruff or mypy). That made me think a bit and I quickly came up with PR > that > >> could "fix" it - i.e. only keep the "real" constraints in. > >> https://github.com/apache/airflow/pull/53631 - it's as easy as adding > >> the `--exact` (and `--strict` for verification) flag to the `uv pip > >> install` command we use. > >> > >> I agree it's a bit confusing to have the devel dependencies there - it > is > >> not the "intention", but there is very little problem with it - the fact > >> that mypy or ruff are listed with a version in constraints, does not > mean > >> it must be installed - constraints mainly say "if you install this, it > >> should be this version" . And there is no problem whatsoever (and it is > >> even recommended by us) to have "airflow with its deps" installed with > >> constraints, and then anything else you need to install or update - > >> without. So other than potential confusion, it usually does not cause > harm. > >> > >> Now... It might seem that removing devel deps has no side effects and we > >> should **just do it** - but it's not that easy. Main problem is that > those > >> devel dependencies might "hold" other dependencies (directly or > >> transitively) from being upgraded - and when we remove the devel > >> dependencies and do the resolution again - we might have a DIFFERENT > set of > >> dependencies, simply because removing the devel dependencies, might > "free" > >> the other deps and they might get upgraded. Yeah - I know it's not > >> obvious for someone who did not spend 3 years solving dependency issues > :). > >> > >> And the problem with it is that because we need those devel deps to test > >> things - the upgraded dependencies might have never been tested in our > CI - > >> because we never upgraded to later versions. It's a bit of > >> Heisenberg-effect. By using test tools we are changing the state of the > >> tested thing. Currently our "PyPI" constraints reflect the state of > Airflow > >> "when the testing tools are installed" - and we are generally not able > to > >> test the state of Airflow "without the testing tools" - by the sheer > fact > >> that we do not have those tools :) > >> > >> Just to illustrate the case - you can see what happens when we removed > >> devel deps: > >> https://github.com/apache/airflow/actions/runs/16444553234 - in summary > >> you can see a colored version that is a bit clearer but I dumped the > diff > >> below from an example - Python 3.10 case. Left side < is "what was with > >> testing tools" and > is without them. You can see a LOT of test tools > >> removed (pytest, ruff and others - including dependencies used only by > >> those tools) > >> > >> But also you can see (likely some of the devel/test deps limited those) > >> > >> * Authlib - upgraded from 1.3.1 to 1.6.1 > >> * Google - genai upgraded from 1.2.0 to 1.20.0 > >> * Httpx - upgraded from 0.27.0 to 0.28.1 > >> * Pinotdb - upgraded from 5.6.0 to 5.7.0 > >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive > as > >> it has just been released and new version was picked) > >> * validators - upgraded form 0.34.0 to 0.35.0 > >> * weaviate-client - upgraded from 4.9.0 to 4.16.3 > >> * websockets - upgraded from 14.2 to 15.0.1 > >> > >> > >> This means that all the "upgraded" packages above have not been - likely > >> ever tested in our CI. > >> > >> And when we put them in "versioned" constraints, it might mean that they > >> will be obviously failing. > >> > >> So ... there is a risk connected with just generating the "non-devel" > >> constraints without additional tests. But that also means that the user > >> who does not use constraints might get those versions anyway - default > >> behaviour of `pip install airflow` will be to install those upgraded > >> versions above if you do not use constraints. But also that is no > different > >> than installing a released airflow version a few weeks or months after > >> release without constraints - there will be tens of packages that were > >> released since our release and installing airflow without constraints > will > >> pick those. > >> > >> However - when we manually install RC candidates - say by running > airflow > >> from rc1 image, we will use those "upgraded" dependencies - so some > >> "obvious" problems might be caught during the rc manual testing - where > you > >> do not use test tools but run airflow "as if" it was installed in > >> production. > >> > >> *The solution ? * > >> > >> There are few ways we can proceed with it: > >> > >> a) leave it as is - no big harm, potential confusion, by people who see > >> it > >> - potential differences > >> > >> b) remove devel following my PR and take the risk that we never tested > (in > >> CI) the dependencies we set in the constraints. That's a bit risky, but > >> we can always reactively downgrade some of those packages after the > fact if > >> users will start reporting the issues > >> > >> c) potentially most complex thing - trying to remove devel dependencies > >> somewhat combining the two - for example removing all the deps that > would > >> be removed in b) from a) - but that has also some edge cases - when > >> different versions of dependencies can add or upgrade their > dependencies, > >> we are very likely to occasionally remove or add too much. > >> > >> > >> I wonder (for those who managed to read that far - likely not many - > what > >> do you think here? What would be the best option ? Maybe there are other > >> options I have not thought about ? > >> > >> > >> J > >> > >> > >> ----- > >> > >> The diff for Python 3.10 > >> > >> > >> > >> 34c34 > >> < Authlib==1.3.1 > >> --- > >> > Authlib==1.6.1 > >> 64d63 > >> < Sphinx==8.2.3 > >> 78d76 > >> < aioresponses==0.7.8 > >> 82d79 > >> < alabaster==1.0.0 > >> 198d194 > >> < arro3-core==0.5.1 > >> 202d197 > >> < astroid==3.3.11 > >> 211,212d205 > >> < aws-sam-translator==1.99.0 > >> < aws-xray-sdk==2.14.0 > >> 258,259d250 > >> < cfgv==3.4.0 > >> < cfn-lint==1.38.0 > >> 262d252 > >> < checksumdir==1.2.0 > >> 277d266 > >> < coverage==7.9.2 > >> 291d279 > >> < deltalake==1.1.0 > >> 293d280 > >> < diagrams==0.24.4 > >> 301,302d287 > >> < docutils==0.21.2 > >> < duckdb==1.3.2 > >> 308d292 > >> < eralchemy2==1.4.1 > >> 311d294 > >> < execnet==2.1.1 > >> 321,322d303 > >> < flit==3.12.0 > >> < flit_core==3.12.0 > >> 386c367 > >> < google-genai==1.2.0 > >> --- > >> > google-genai==1.20.0 > >> 389d369 > >> < graphql-core==3.2.6 > >> 398d377 > >> < grpcio-tools==1.62.3 > >> 403,404d381 > >> < hatch==1.14.1 > >> < hatchling==1.27.0 > >> 413c390 > >> < httpx==0.27.0 > >> --- > >> > httpx==0.28.1 > >> 418d394 > >> < hyperlink==21.0.0 > >> 421,423d396 > >> < icdiff==2.0.7 > >> < id==1.5.0 > >> < identify==2.6.12 > >> 426d398 > >> < imagesize==1.4.1 > >> 430d401 > >> < incremental==24.7.2 > >> 433,435d403 > >> < iniconfig==2.1.0 > >> < inputimeout==1.0.4 > >> < ipdb==0.13.13 > >> 449d416 > >> < joserfc==1.2.2 > >> 451d417 > >> < jsonpatch==1.33 > >> 455,456d420 > >> < jsonpointer==3.0.0 > >> < jsonschema-path==0.3.4 > >> 463d426 > >> < kerberos==1.3.1 > >> 465,466d427 > >> < keyrings.alt==5.0.2 > >> < kgb==7.2 > >> 494,495d454 > >> < mmh3==5.1.0 > >> < mongomock==4.3.0 > >> 497,498d455 > >> < moto==5.1.8 > >> < mpmath==1.3.0 > >> 508,512d464 > >> < mypy-boto3-appflow==1.39.0 > >> < mypy-boto3-rds==1.39.1 > >> < mypy-boto3-redshift-data==1.39.0 > >> < mypy-boto3-s3==1.39.5 > >> < mypy==1.17.0 > >> 514c466 > >> < mysql-connector-python==9.3.0 > >> --- > >> > mysql-connector-python==9.4.0 > >> 521,523d472 > >> < networkx==3.5 > >> < nh3==0.3.0 > >> < nodeenv==1.9.1 > >> 528,529d476 > >> < openapi-schema-validator==0.6.3 > >> < openapi-spec-validator==0.7.2 > >> 560d506 > >> < pathable==0.4.4 > >> 563d508 > >> < pdbr==0.9.2 > >> 569,570c514 > >> < pinotdb==5.6.0 > >> < pipdeptree==2.28.0 > >> --- > >> > pinotdb==5.7.0 > >> 574d517 > >> < plyvel==1.5.1 > >> 577,579d519 > >> < pprintpp==0.4.0 > >> < pre-commit-uv==4.1.4 > >> < pre_commit==4.2.0 > >> 592d531 > >> < py-partiql-parser==0.6.1 > >> 608d546 > >> < pyenchant==3.2.2 > >> 610,611d547 > >> < pygraphviz==1.14 > >> < pyiceberg==0.9.1 > >> 622,632d557 > >> < pytest-asyncio==0.25.0 > >> < pytest-cov==6.2.1 > >> < pytest-custom-exit-code==0.3.0 > >> < pytest-icdiff==0.9 > >> < pytest-instafail==0.5.0 > >> < pytest-mock==3.14.1 > >> < pytest-rerunfailures==15.1 > >> < pytest-timeouts==1.2.1 > >> < pytest-unordered==0.7.0 > >> < pytest-xdist==3.8.0 > >> < pytest==8.4.1 > >> 642d566 > >> < python-on-whales==0.78.0 > >> 652d575 > >> < readme_renderer==44.0 > >> 659d581 > >> < requests-mock==1.12.1 > >> 664,665d585 > >> < responses==0.25.7 > >> < restructuredtext_lint==1.4.0 > >> 667,668d586 > >> < rfc3339-validator==0.1.4 > >> < rfc3986==2.0.0 > >> 670d587 > >> < rich-click==1.8.9 > >> 674d590 > >> < roman-numerals-py==3.1.0 > >> 679d594 > >> < ruff==0.12.3 > >> 688d602 > >> < semver==3.0.4 > >> 690,691c604 > >> < sentinels==1.0.0 > >> < sentry-sdk==2.33.1 > >> --- > >> > sentry-sdk==2.33.2 > >> 703d615 > >> < snowballstemmer==3.0.1 > >> 709,725d620 > >> < sphinx-argparse==0.5.2 > >> < sphinx-autoapi==3.6.0 > >> < sphinx-autobuild==2024.10.3 > >> < sphinx-copybutton==0.5.2 > >> < sphinx-jinja==2.0.2 > >> < sphinx-rtd-theme==3.0.2 > >> < sphinx_design==0.6.1 > >> < sphinxcontrib-applehelp==2.0.0 > >> < sphinxcontrib-devhelp==2.0.0 > >> < sphinxcontrib-htmlhelp==2.1.0 > >> < sphinxcontrib-httpdomain==1.8.1 > >> < sphinxcontrib-jquery==4.1 > >> < sphinxcontrib-jsmath==1.0.1 > >> < sphinxcontrib-qthelp==2.0.0 > >> < sphinxcontrib-redoc==1.6.0 > >> < sphinxcontrib-serializinghtml==2.0.0 > >> < sphinxcontrib-spelling==8.0.1 > >> 737d631 > >> < strictyaml==1.7.3 > >> 740d633 > >> < sympy==1.14.0 > >> 752d644 > >> < time-machine==2.16.0 > >> 755d646 > >> < tomli_w==1.2.0 > >> 758d648 > >> < towncrier==24.8.0 > >> 762,763d651 > >> < trove-classifiers==2025.5.9.12 > >> < twine==6.1.0 > >> 765,774d652 > >> < types-Deprecated==1.2.15.20250304 > >> < types-Markdown==3.8.0.20250708 > >> < types-PyMySQL==1.1.0.20250711 > >> < types-PyYAML==6.0.12.20250516 > >> < types-aiofiles==24.1.0.20250708 > >> < types-certifi==2021.10.8.3 > >> < types-cffi==1.17.0.20250523 > >> < types-croniter==6.0.0.20250626 > >> < types-docutils==0.21.0.20250722 > >> < types-paramiko==3.5.0.20250708 > >> 776,778d653 > >> < types-pyOpenSSL==24.1.0.20240722 > >> < types-python-dateutil==2.9.0.20250708 > >> < types-python-slugify==8.0.2.20240310 > >> 780d654 > >> < types-redis==4.6.0.20241004 > >> 782,784d655 > >> < types-setuptools==80.9.0.20250529 > >> < types-tabulate==0.9.0.20241207 > >> < types-toml==0.10.8.20240310 > >> 794d664 > >> < userpath==1.9.2 > >> 799c669 > >> < validators==0.34.0 > >> --- > >> > validators==0.35.0 > >> 806c676 > >> < weaviate-client==4.9.6 > >> --- > >> > weaviate-client==4.16.3 > >> 809c679 > >> < websockets==14.2 > >> --- > >> > websockets==15.0.1 > >> 815d684 > >> < yamllint==1.37.1 > >> ________________________________ > >> Strike Technologies, LLC (“Strike”) is part of the GTS family of > >> companies. Strike is a technology solutions provider, and is not a > broker > >> or dealer and does not transact any securities related business directly > >> whatsoever. This communication is the property of Strike and its > >> affiliates, and does not constitute an offer to sell or the > solicitation of > >> an offer to buy any security in any jurisdiction. It is intended only > for > >> the person to whom it is addressed and may contain information that is > >> privileged, confidential, or otherwise protected from disclosure. > >> Distribution or copying of this communication, or the information > contained > >> herein, by anyone other than the intended recipient is prohibited. If > you > >> have received this communication in error, please immediately notify > Strike > >> at i...@striketechnologies.com, and delete and destroy any copies > hereof. > >> ________________________________ > >> > >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > attachments > >> are intended solely for the addressee. This transmission is covered by > the > >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > >> information contained in this transmission is confidential in nature and > >> protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > >> legal privilege. Your use or disclosure of this information for any > purpose > >> other than that intended by its transmittal is strictly prohibited, and > may > >> subject you to fines and/or penalties under federal and state law. If > you > >> are not the intended recipient of this transmission, please DESTROY ALL > >> COPIES RECEIVED and confirm destruction to the sender via return > >> transmittal. > >> > > >