OK. I will run LAZY_CONSENSUS then On Fri, Jul 25, 2025 at 9:22 AM Pavankumar Gopidesu <[email protected]> wrote:
> Thanks Jarek, > > I am fine with option b as far as if we find anything while testing we can > downgrade. > > Pavan > > On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <[email protected]> wrote: > > > Would love to hear more if anyone has an opinion :). Yep. It's an > advanced > > topic and I can make arbitrary decision :) ... > > > > I am thinking now that it should be quite "OK" to accept those > "potentially > > untested" versions - especially that they will be tested during manual > > testing. Unless someone objects i will likely update PR with some > changelog > > / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS > > soon if I don't hear any opposition. > > > > J. > > > > > > On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <[email protected]> wrote: > > > > > > Exactly to remove the dev requirements airflow has from interfering > > with > > > our own dev requirements. Though this approach notably doesn't change > any > > > of the used non-dev requirements, it's not clear to me (and I'm > probably > > > just missing something) why your diff includes non-dev requirements. > > > > > > The thing is that I do not "remove" devel features - because due to > > > transitive dependencies I would have to go deep down the rabbit hole > > which > > > dependencies are "devel" and which not. So what I do, I ask `uv` to do > > the > > > job for me and repeat the resolution but with the `--exact` flag > > (combined > > > with the already used `--resolution higest` and `--no-sources`). > > > > > > Apparently in this case `uv` will sometimes decide that it's ok to bump > > > some dependencies that are left after initial resolution. The thing is > > that > > > uv will not install some versions of some packages if they are > > conflicting > > > with already installed packages. What I believe `uv` does when I use > the > > *--exact > > > *flag of `uv pip install` it "fixes" the initial resolution. It finds > out > > > what is needed, removes those packages that are not needed, and then it > > > resolves again and improves the resolution (that would be my guess at > > least > > > - it could be done slightly differently and in a single step I guess. > > > > > > Essentially the difference between the left and right side is: > > > > > > a) install absolutely everything from the workspace > > > > > > b) do this: > > > left: uv pip install --no-sources 'apache-airflow[all]' > > > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl > > > --reinstall --resolution highest --find-links file:///dist > > > right: uv pip install --no-sources *--exact --strict* > > > 'apache-airflow[all]' 'apache-airflow-core[all]' > apache-airflow-task-sdk > > > ./airflow-ctl --reinstall --resolution highest --find-links > file:///dist > > > > > > In the current state content of the fille://dist should be the same in > > > both cases: > > > > > > apache-airflow-providers-edge3 (1.1.2): wheel - > > > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl > > > apache-airflow-providers-keycloak (1.0.0): wheel - > > > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl > > > apache-airflow-providers-snowflake (6.5.1): wheel - > > > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl > > > > > > This is a bit of in-process quirk of the "PyPI" resolution - those are > > the > > > three packages that we have updated versions of in the sources, but > they > > > are not released yet. But they should be essentially the same in both > > > cases. So the difference between left and right is `--exact --strict` > > > (strict I added to make sure that we are not removing something that is > > > still needed). > > > > > > J. > > > > > > > > > > > > > > > > > > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw < > > [email protected]> > > > wrote: > > > > > >> As I user of Airflow I have ended up implementing this for my > > >> environments: > > >> > > >> pip install apache-airflow[all or select-extras] -c {official > > >> constraints} > smaller-run-time-constraints.txt > > >> pip-compile {all my requirements including development}.txt -c > > >> smaller-run-time-constraints.txt > > >> > > >> Exactly to remove the dev requirements airflow has from interfering > with > > >> our own dev requirements. Though this approach notably doesn't change > > any > > >> of the used non-dev requirements, it's not clear to me (and I'm > probably > > >> just missing something) why your diff includes non-dev requirements. > > >> > > >> Damian > > >> > > >> -----Original Message----- > > >> From: Jarek Potiuk <[email protected]> > > >> Sent: Tuesday, July 22, 2025 11:04 AM > > >> To: [email protected] > > >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ? > > >> > > >> Hello here, > > >> > > >> > > >> *Tl;DR; We had an interesting discussion on Slack today and I wanted > to > > >> bring it here to discuss what we should do - whether we should keep > > devel > > >> dependencies in our PyPI constraints or not.* > > >> > > >> *A bit of context:* > > >> In our CI we generate several types of constraints. one of them are > > >> "source" constraints that we use during our CI testing - including all > > our > > >> testing tools, mypy, ruff etc., and another type is "PyPI" constraint > - > > >> which is more <What should be the "golden" versions of dependencies > when > > >> you install Airflow using `pip install` for production>. > > >> > > >> The main difference between the two (roughly) is that "source > > constraints" > > >> only uses dependencies from "branch tip" ("main", "v3-0-test"), where > > >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but > all > > >> the providers are installed from PyPI. There are few nuances when some > > >> packages are being released but basically - for example for Airflow > > 3.0.3, > > >> "PyPI" > > >> constraints is "what is the current set of dependencies and providers > > >> when we install airflow from PyPI at the moment of release". > > >> > > >> So target for "source" constraints are "Airflow contributors" where > > >> target for "PyPI constraints" are "Airflow users". > > >> > > >> > > >> *The problem* > > >> In the slack discussion > > >> > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 > > >> - slack user @dashmug (sorry - do not know name) noticed that the > > "PyPI" > > >> constraints also contain Airflow development dependencies listed (like > > >> ruff or mypy). That made me think a bit and I quickly came up with PR > > that > > >> could "fix" it - i.e. only keep the "real" constraints in. > > >> https://github.com/apache/airflow/pull/53631 - it's as easy as adding > > >> the `--exact` (and `--strict` for verification) flag to the `uv pip > > >> install` command we use. > > >> > > >> I agree it's a bit confusing to have the devel dependencies there - it > > is > > >> not the "intention", but there is very little problem with it - the > fact > > >> that mypy or ruff are listed with a version in constraints, does not > > mean > > >> it must be installed - constraints mainly say "if you install this, it > > >> should be this version" . And there is no problem whatsoever (and it > is > > >> even recommended by us) to have "airflow with its deps" installed with > > >> constraints, and then anything else you need to install or update - > > >> without. So other than potential confusion, it usually does not cause > > harm. > > >> > > >> Now... It might seem that removing devel deps has no side effects and > we > > >> should **just do it** - but it's not that easy. Main problem is that > > those > > >> devel dependencies might "hold" other dependencies (directly or > > >> transitively) from being upgraded - and when we remove the devel > > >> dependencies and do the resolution again - we might have a DIFFERENT > > set of > > >> dependencies, simply because removing the devel dependencies, might > > "free" > > >> the other deps and they might get upgraded. Yeah - I know it's not > > >> obvious for someone who did not spend 3 years solving dependency > issues > > :). > > >> > > >> And the problem with it is that because we need those devel deps to > test > > >> things - the upgraded dependencies might have never been tested in our > > CI - > > >> because we never upgraded to later versions. It's a bit of > > >> Heisenberg-effect. By using test tools we are changing the state of > the > > >> tested thing. Currently our "PyPI" constraints reflect the state of > > Airflow > > >> "when the testing tools are installed" - and we are generally not able > > to > > >> test the state of Airflow "without the testing tools" - by the sheer > > fact > > >> that we do not have those tools :) > > >> > > >> Just to illustrate the case - you can see what happens when we removed > > >> devel deps: > > >> https://github.com/apache/airflow/actions/runs/16444553234 - in > summary > > >> you can see a colored version that is a bit clearer but I dumped the > > diff > > >> below from an example - Python 3.10 case. Left side < is "what was > with > > >> testing tools" and > is without them. You can see a LOT of test tools > > >> removed (pytest, ruff and others - including dependencies used only by > > >> those tools) > > >> > > >> But also you can see (likely some of the devel/test deps limited > those) > > >> > > >> * Authlib - upgraded from 1.3.1 to 1.6.1 > > >> * Google - genai upgraded from 1.2.0 to 1.20.0 > > >> * Httpx - upgraded from 0.27.0 to 0.28.1 > > >> * Pinotdb - upgraded from 5.6.0 to 5.7.0 > > >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive > > as > > >> it has just been released and new version was picked) > > >> * validators - upgraded form 0.34.0 to 0.35.0 > > >> * weaviate-client - upgraded from 4.9.0 to 4.16.3 > > >> * websockets - upgraded from 14.2 to 15.0.1 > > >> > > >> > > >> This means that all the "upgraded" packages above have not been - > likely > > >> ever tested in our CI. > > >> > > >> And when we put them in "versioned" constraints, it might mean that > they > > >> will be obviously failing. > > >> > > >> So ... there is a risk connected with just generating the "non-devel" > > >> constraints without additional tests. But that also means that the > user > > >> who does not use constraints might get those versions anyway - default > > >> behaviour of `pip install airflow` will be to install those upgraded > > >> versions above if you do not use constraints. But also that is no > > different > > >> than installing a released airflow version a few weeks or months after > > >> release without constraints - there will be tens of packages that were > > >> released since our release and installing airflow without constraints > > will > > >> pick those. > > >> > > >> However - when we manually install RC candidates - say by running > > airflow > > >> from rc1 image, we will use those "upgraded" dependencies - so some > > >> "obvious" problems might be caught during the rc manual testing - > where > > you > > >> do not use test tools but run airflow "as if" it was installed in > > >> production. > > >> > > >> *The solution ? * > > >> > > >> There are few ways we can proceed with it: > > >> > > >> a) leave it as is - no big harm, potential confusion, by people who > see > > >> it > > >> - potential differences > > >> > > >> b) remove devel following my PR and take the risk that we never tested > > (in > > >> CI) the dependencies we set in the constraints. That's a bit risky, > but > > >> we can always reactively downgrade some of those packages after the > > fact if > > >> users will start reporting the issues > > >> > > >> c) potentially most complex thing - trying to remove devel > dependencies > > >> somewhat combining the two - for example removing all the deps that > > would > > >> be removed in b) from a) - but that has also some edge cases - when > > >> different versions of dependencies can add or upgrade their > > dependencies, > > >> we are very likely to occasionally remove or add too much. > > >> > > >> > > >> I wonder (for those who managed to read that far - likely not many - > > what > > >> do you think here? What would be the best option ? Maybe there are > other > > >> options I have not thought about ? > > >> > > >> > > >> J > > >> > > >> > > >> ----- > > >> > > >> The diff for Python 3.10 > > >> > > >> > > >> > > >> 34c34 > > >> < Authlib==1.3.1 > > >> --- > > >> > Authlib==1.6.1 > > >> 64d63 > > >> < Sphinx==8.2.3 > > >> 78d76 > > >> < aioresponses==0.7.8 > > >> 82d79 > > >> < alabaster==1.0.0 > > >> 198d194 > > >> < arro3-core==0.5.1 > > >> 202d197 > > >> < astroid==3.3.11 > > >> 211,212d205 > > >> < aws-sam-translator==1.99.0 > > >> < aws-xray-sdk==2.14.0 > > >> 258,259d250 > > >> < cfgv==3.4.0 > > >> < cfn-lint==1.38.0 > > >> 262d252 > > >> < checksumdir==1.2.0 > > >> 277d266 > > >> < coverage==7.9.2 > > >> 291d279 > > >> < deltalake==1.1.0 > > >> 293d280 > > >> < diagrams==0.24.4 > > >> 301,302d287 > > >> < docutils==0.21.2 > > >> < duckdb==1.3.2 > > >> 308d292 > > >> < eralchemy2==1.4.1 > > >> 311d294 > > >> < execnet==2.1.1 > > >> 321,322d303 > > >> < flit==3.12.0 > > >> < flit_core==3.12.0 > > >> 386c367 > > >> < google-genai==1.2.0 > > >> --- > > >> > google-genai==1.20.0 > > >> 389d369 > > >> < graphql-core==3.2.6 > > >> 398d377 > > >> < grpcio-tools==1.62.3 > > >> 403,404d381 > > >> < hatch==1.14.1 > > >> < hatchling==1.27.0 > > >> 413c390 > > >> < httpx==0.27.0 > > >> --- > > >> > httpx==0.28.1 > > >> 418d394 > > >> < hyperlink==21.0.0 > > >> 421,423d396 > > >> < icdiff==2.0.7 > > >> < id==1.5.0 > > >> < identify==2.6.12 > > >> 426d398 > > >> < imagesize==1.4.1 > > >> 430d401 > > >> < incremental==24.7.2 > > >> 433,435d403 > > >> < iniconfig==2.1.0 > > >> < inputimeout==1.0.4 > > >> < ipdb==0.13.13 > > >> 449d416 > > >> < joserfc==1.2.2 > > >> 451d417 > > >> < jsonpatch==1.33 > > >> 455,456d420 > > >> < jsonpointer==3.0.0 > > >> < jsonschema-path==0.3.4 > > >> 463d426 > > >> < kerberos==1.3.1 > > >> 465,466d427 > > >> < keyrings.alt==5.0.2 > > >> < kgb==7.2 > > >> 494,495d454 > > >> < mmh3==5.1.0 > > >> < mongomock==4.3.0 > > >> 497,498d455 > > >> < moto==5.1.8 > > >> < mpmath==1.3.0 > > >> 508,512d464 > > >> < mypy-boto3-appflow==1.39.0 > > >> < mypy-boto3-rds==1.39.1 > > >> < mypy-boto3-redshift-data==1.39.0 > > >> < mypy-boto3-s3==1.39.5 > > >> < mypy==1.17.0 > > >> 514c466 > > >> < mysql-connector-python==9.3.0 > > >> --- > > >> > mysql-connector-python==9.4.0 > > >> 521,523d472 > > >> < networkx==3.5 > > >> < nh3==0.3.0 > > >> < nodeenv==1.9.1 > > >> 528,529d476 > > >> < openapi-schema-validator==0.6.3 > > >> < openapi-spec-validator==0.7.2 > > >> 560d506 > > >> < pathable==0.4.4 > > >> 563d508 > > >> < pdbr==0.9.2 > > >> 569,570c514 > > >> < pinotdb==5.6.0 > > >> < pipdeptree==2.28.0 > > >> --- > > >> > pinotdb==5.7.0 > > >> 574d517 > > >> < plyvel==1.5.1 > > >> 577,579d519 > > >> < pprintpp==0.4.0 > > >> < pre-commit-uv==4.1.4 > > >> < pre_commit==4.2.0 > > >> 592d531 > > >> < py-partiql-parser==0.6.1 > > >> 608d546 > > >> < pyenchant==3.2.2 > > >> 610,611d547 > > >> < pygraphviz==1.14 > > >> < pyiceberg==0.9.1 > > >> 622,632d557 > > >> < pytest-asyncio==0.25.0 > > >> < pytest-cov==6.2.1 > > >> < pytest-custom-exit-code==0.3.0 > > >> < pytest-icdiff==0.9 > > >> < pytest-instafail==0.5.0 > > >> < pytest-mock==3.14.1 > > >> < pytest-rerunfailures==15.1 > > >> < pytest-timeouts==1.2.1 > > >> < pytest-unordered==0.7.0 > > >> < pytest-xdist==3.8.0 > > >> < pytest==8.4.1 > > >> 642d566 > > >> < python-on-whales==0.78.0 > > >> 652d575 > > >> < readme_renderer==44.0 > > >> 659d581 > > >> < requests-mock==1.12.1 > > >> 664,665d585 > > >> < responses==0.25.7 > > >> < restructuredtext_lint==1.4.0 > > >> 667,668d586 > > >> < rfc3339-validator==0.1.4 > > >> < rfc3986==2.0.0 > > >> 670d587 > > >> < rich-click==1.8.9 > > >> 674d590 > > >> < roman-numerals-py==3.1.0 > > >> 679d594 > > >> < ruff==0.12.3 > > >> 688d602 > > >> < semver==3.0.4 > > >> 690,691c604 > > >> < sentinels==1.0.0 > > >> < sentry-sdk==2.33.1 > > >> --- > > >> > sentry-sdk==2.33.2 > > >> 703d615 > > >> < snowballstemmer==3.0.1 > > >> 709,725d620 > > >> < sphinx-argparse==0.5.2 > > >> < sphinx-autoapi==3.6.0 > > >> < sphinx-autobuild==2024.10.3 > > >> < sphinx-copybutton==0.5.2 > > >> < sphinx-jinja==2.0.2 > > >> < sphinx-rtd-theme==3.0.2 > > >> < sphinx_design==0.6.1 > > >> < sphinxcontrib-applehelp==2.0.0 > > >> < sphinxcontrib-devhelp==2.0.0 > > >> < sphinxcontrib-htmlhelp==2.1.0 > > >> < sphinxcontrib-httpdomain==1.8.1 > > >> < sphinxcontrib-jquery==4.1 > > >> < sphinxcontrib-jsmath==1.0.1 > > >> < sphinxcontrib-qthelp==2.0.0 > > >> < sphinxcontrib-redoc==1.6.0 > > >> < sphinxcontrib-serializinghtml==2.0.0 > > >> < sphinxcontrib-spelling==8.0.1 > > >> 737d631 > > >> < strictyaml==1.7.3 > > >> 740d633 > > >> < sympy==1.14.0 > > >> 752d644 > > >> < time-machine==2.16.0 > > >> 755d646 > > >> < tomli_w==1.2.0 > > >> 758d648 > > >> < towncrier==24.8.0 > > >> 762,763d651 > > >> < trove-classifiers==2025.5.9.12 > > >> < twine==6.1.0 > > >> 765,774d652 > > >> < types-Deprecated==1.2.15.20250304 > > >> < types-Markdown==3.8.0.20250708 > > >> < types-PyMySQL==1.1.0.20250711 > > >> < types-PyYAML==6.0.12.20250516 > > >> < types-aiofiles==24.1.0.20250708 > > >> < types-certifi==2021.10.8.3 > > >> < types-cffi==1.17.0.20250523 > > >> < types-croniter==6.0.0.20250626 > > >> < types-docutils==0.21.0.20250722 > > >> < types-paramiko==3.5.0.20250708 > > >> 776,778d653 > > >> < types-pyOpenSSL==24.1.0.20240722 > > >> < types-python-dateutil==2.9.0.20250708 > > >> < types-python-slugify==8.0.2.20240310 > > >> 780d654 > > >> < types-redis==4.6.0.20241004 > > >> 782,784d655 > > >> < types-setuptools==80.9.0.20250529 > > >> < types-tabulate==0.9.0.20241207 > > >> < types-toml==0.10.8.20240310 > > >> 794d664 > > >> < userpath==1.9.2 > > >> 799c669 > > >> < validators==0.34.0 > > >> --- > > >> > validators==0.35.0 > > >> 806c676 > > >> < weaviate-client==4.9.6 > > >> --- > > >> > weaviate-client==4.16.3 > > >> 809c679 > > >> < websockets==14.2 > > >> --- > > >> > websockets==15.0.1 > > >> 815d684 > > >> < yamllint==1.37.1 > > >> ________________________________ > > >> Strike Technologies, LLC (“Strike”) is part of the GTS family of > > >> companies. Strike is a technology solutions provider, and is not a > > broker > > >> or dealer and does not transact any securities related business > directly > > >> whatsoever. This communication is the property of Strike and its > > >> affiliates, and does not constitute an offer to sell or the > > solicitation of > > >> an offer to buy any security in any jurisdiction. It is intended only > > for > > >> the person to whom it is addressed and may contain information that is > > >> privileged, confidential, or otherwise protected from disclosure. > > >> Distribution or copying of this communication, or the information > > contained > > >> herein, by anyone other than the intended recipient is prohibited. If > > you > > >> have received this communication in error, please immediately notify > > Strike > > >> at [email protected], and delete and destroy any copies > > hereof. > > >> ________________________________ > > >> > > >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > > attachments > > >> are intended solely for the addressee. This transmission is covered by > > the > > >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > > >> information contained in this transmission is confidential in nature > and > > >> protected from further use or disclosure under U.S. Pub. L. 106-102, > 113 > > >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > > >> legal privilege. Your use or disclosure of this information for any > > purpose > > >> other than that intended by its transmittal is strictly prohibited, > and > > may > > >> subject you to fines and/or penalties under federal and state law. If > > you > > >> are not the intended recipient of this transmission, please DESTROY > ALL > > >> COPIES RECEIVED and confirm destruction to the sender via return > > >> transmittal. > > >> > > > > > >
