Thanks Jarek,

I am fine with option b as far as if we find anything while testing we can
downgrade.

Pavan

On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Would love to hear more if anyone has an opinion :). Yep. It's an advanced
> topic and I can make arbitrary decision :)  ...
>
> I am thinking now that it should be quite "OK" to accept those "potentially
> untested" versions - especially that they will be tested during manual
> testing. Unless someone objects i will likely update PR with some changelog
> / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
> soon if I don't hear any opposition.
>
> J.
>
>
> On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > Exactly to remove the dev requirements airflow has from interfering
> with
> > our own dev requirements. Though this approach notably doesn't change any
> > of the used non-dev requirements, it's not clear to me (and I'm probably
> > just missing something) why your diff includes non-dev requirements.
> >
> > The thing is that I do not "remove" devel features - because due to
> > transitive dependencies I would have to go deep down the rabbit hole
> which
> > dependencies are "devel" and which not. So what I do, I ask `uv` to do
> the
> > job for me and repeat the resolution but with the `--exact` flag
> (combined
> > with the already used `--resolution higest` and `--no-sources`).
> >
> > Apparently in this case `uv` will sometimes decide that it's ok to bump
> > some dependencies that are left after initial resolution. The thing is
> that
> > uv will not install some versions of some packages if they are
> conflicting
> > with already installed packages. What I believe `uv` does when I use the
> *--exact
> > *flag of `uv pip install` it "fixes" the initial resolution. It finds out
> > what is needed, removes those packages that are not needed, and then it
> > resolves again and improves the resolution (that would be my guess at
> least
> > - it could be done slightly differently and in a single step I guess.
> >
> > Essentially the difference between the left and right side is:
> >
> > a) install absolutely everything from the workspace
> >
> > b) do this:
> > left: uv pip install --no-sources 'apache-airflow[all]'
> > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> > --reinstall --resolution highest --find-links file:///dist
> > right: uv pip install --no-sources *--exact --strict*
> > 'apache-airflow[all]' 'apache-airflow-core[all]' apache-airflow-task-sdk
> > ./airflow-ctl --reinstall --resolution highest --find-links file:///dist
> >
> > In the current state content of the fille://dist should be the same in
> > both cases:
> >
> > apache-airflow-providers-edge3 (1.1.2): wheel -
> > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> > apache-airflow-providers-keycloak (1.0.0): wheel -
> > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> > apache-airflow-providers-snowflake (6.5.1): wheel -
> > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
> >
> > This is a bit of in-process quirk of the "PyPI" resolution - those are
> the
> > three packages that we have updated versions of in the sources, but they
> > are not released yet. But they should be essentially the same in both
> > cases. So the difference between left and right is `--exact --strict`
> > (strict I added to make sure that we are not removing something that is
> > still needed).
> >
> > J.
> >
> >
> >
> >
> >
> > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <
> ds...@striketechnologies.com>
> > wrote:
> >
> >> As I user of Airflow I have ended up implementing this for my
> >> environments:
> >>
> >> pip install apache-airflow[all or select-extras] -c {official
> >> constraints} > smaller-run-time-constraints.txt
> >> pip-compile {all my requirements including development}.txt -c
> >> smaller-run-time-constraints.txt
> >>
> >> Exactly to remove the dev requirements airflow has from interfering with
> >> our own dev requirements. Though this approach notably doesn't change
> any
> >> of the used non-dev requirements, it's not clear to me (and I'm probably
> >> just missing something) why your diff includes non-dev requirements.
> >>
> >> Damian
> >>
> >> -----Original Message-----
> >> From: Jarek Potiuk <ja...@potiuk.com>
> >> Sent: Tuesday, July 22, 2025 11:04 AM
> >> To: dev@airflow.apache.org
> >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
> >>
> >> Hello here,
> >>
> >>
> >> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
> >> bring it here to discuss what we should do - whether we should keep
> devel
> >> dependencies in our PyPI constraints or not.*
> >>
> >> *A bit of context:*
> >> In our CI we generate several types of constraints. one of them are
> >> "source" constraints that we use during our CI testing - including all
> our
> >> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
> >> which is more <What should be the "golden" versions of dependencies when
> >> you install Airflow using `pip install` for production>.
> >>
> >> The main difference between the two (roughly) is that "source
> constraints"
> >> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
> >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
> >> the providers are installed from PyPI. There are few nuances when some
> >> packages are being released but basically - for example for Airflow
> 3.0.3,
> >> "PyPI"
> >> constraints is "what is the current set of dependencies and providers
> >> when we install airflow from PyPI at the moment of release".
> >>
> >> So target for "source" constraints are "Airflow contributors" where
> >> target for "PyPI constraints" are "Airflow users".
> >>
> >>
> >> *The problem*
> >> In the slack discussion
> >> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
> >> - slack user @dashmug  (sorry - do not know name) noticed that the
> "PyPI"
> >> constraints also contain Airflow development dependencies listed (like
> >> ruff or mypy). That made me think a bit and I quickly came up with PR
> that
> >> could "fix" it - i.e. only keep the "real" constraints in.
> >> https://github.com/apache/airflow/pull/53631 - it's as easy as adding
> >> the `--exact` (and `--strict` for verification) flag to the `uv pip
> >> install` command we use.
> >>
> >> I agree it's a bit confusing to have the devel dependencies there - it
> is
> >> not the "intention", but there is very little problem with it - the fact
> >> that mypy or ruff are listed with a version in constraints, does not
> mean
> >> it must be installed - constraints mainly say "if you install this, it
> >> should be this version" . And there is no problem whatsoever (and it is
> >> even recommended by us) to have "airflow with its deps" installed with
> >> constraints, and then anything else you need to install or update -
> >> without. So other than potential confusion, it usually does not cause
> harm.
> >>
> >> Now... It might seem that removing devel deps has no side effects and we
> >> should **just do it** - but it's not that easy. Main problem is that
> those
> >> devel dependencies might  "hold" other dependencies (directly or
> >> transitively) from being upgraded - and when we remove the devel
> >> dependencies and do the resolution again - we might have a DIFFERENT
> set of
> >> dependencies, simply because removing the devel dependencies, might
> "free"
> >> the other deps and they might get upgraded. Yeah - I know it's not
> >> obvious for someone who did not spend 3 years solving dependency issues
> :).
> >>
> >> And the problem with it is that because we need those devel deps to test
> >> things - the upgraded dependencies might have never been tested in our
> CI -
> >> because we never upgraded to later versions. It's a bit of
> >> Heisenberg-effect. By using test tools we are changing the state of the
> >> tested thing. Currently our "PyPI" constraints reflect the state of
> Airflow
> >> "when the testing tools are installed" - and we are generally not able
> to
> >> test the state of Airflow "without the testing tools" - by the sheer
> fact
> >> that we do not have those tools :)
> >>
> >> Just to illustrate the case - you can see what happens when we removed
> >> devel deps:
> >> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
> >> you can see a colored version that is a bit clearer but I dumped the
> diff
> >> below from an example - Python 3.10 case. Left side < is "what was with
> >> testing tools" and > is without them. You can see a LOT of test tools
> >> removed (pytest, ruff and others - including dependencies used only by
> >> those tools)
> >>
> >> But also you can see (likely some of the devel/test deps limited those)
> >>
> >> * Authlib - upgraded from 1.3.1 to 1.6.1
> >> * Google - genai upgraded from 1.2.0 to 1.20.0
> >> * Httpx - upgraded from 0.27.0 to 0.28.1
> >> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2  (this is false positive
> as
> >> it has just been released and new version was picked)
> >> * validators - upgraded form 0.34.0 to 0.35.0
> >> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> >> * websockets - upgraded from 14.2 to 15.0.1
> >>
> >>
> >> This means that all the "upgraded" packages above have not been - likely
> >> ever tested in our CI.
> >>
> >> And when we put them in "versioned" constraints, it might mean that they
> >> will be obviously failing.
> >>
> >> So ... there is a risk connected with just generating the "non-devel"
> >> constraints without additional tests. But that also means that the user
> >> who does not use constraints might get those versions anyway - default
> >> behaviour of `pip install airflow` will be to install those upgraded
> >> versions above if you do not use constraints. But also that is no
> different
> >> than installing a released airflow version a few weeks or months after
> >> release without constraints - there will be tens of packages that were
> >> released since our release and installing airflow without constraints
> will
> >> pick those.
> >>
> >> However - when we manually install RC candidates - say by running
> airflow
> >> from rc1 image, we will use those "upgraded" dependencies - so some
> >> "obvious" problems might be caught during the rc manual testing - where
> you
> >> do not use test tools but run airflow "as if" it was installed in
> >> production.
> >>
> >> *The solution ? *
> >>
> >> There are few ways we can proceed with it:
> >>
> >> a) leave it as is  - no big harm, potential confusion, by people who see
> >> it
> >> - potential differences
> >>
> >> b) remove devel following my PR and take the risk that we never tested
> (in
> >> CI) the dependencies we set in the constraints. That's a bit risky, but
> >> we can always reactively downgrade some of those packages after the
> fact if
> >> users will start reporting the issues
> >>
> >> c) potentially most complex thing - trying to remove devel dependencies
> >> somewhat combining the two - for example removing all the deps that
> would
> >> be removed in b) from a) - but that has also some edge cases - when
> >> different versions of dependencies can add or upgrade their
> dependencies,
> >> we are very likely to occasionally remove or add too much.
> >>
> >>
> >> I wonder (for those who managed to read that far - likely not many -
> what
> >> do you think here? What would be the best option ? Maybe there are other
> >> options I have not thought about ?
> >>
> >>
> >> J
> >>
> >>
> >> -----
> >>
> >> The diff for Python 3.10
> >>
> >>
> >>
> >> 34c34
> >> < Authlib==1.3.1
> >> ---
> >> > Authlib==1.6.1
> >> 64d63
> >> < Sphinx==8.2.3
> >> 78d76
> >> < aioresponses==0.7.8
> >> 82d79
> >> < alabaster==1.0.0
> >> 198d194
> >> < arro3-core==0.5.1
> >> 202d197
> >> < astroid==3.3.11
> >> 211,212d205
> >> < aws-sam-translator==1.99.0
> >> < aws-xray-sdk==2.14.0
> >> 258,259d250
> >> < cfgv==3.4.0
> >> < cfn-lint==1.38.0
> >> 262d252
> >> < checksumdir==1.2.0
> >> 277d266
> >> < coverage==7.9.2
> >> 291d279
> >> < deltalake==1.1.0
> >> 293d280
> >> < diagrams==0.24.4
> >> 301,302d287
> >> < docutils==0.21.2
> >> < duckdb==1.3.2
> >> 308d292
> >> < eralchemy2==1.4.1
> >> 311d294
> >> < execnet==2.1.1
> >> 321,322d303
> >> < flit==3.12.0
> >> < flit_core==3.12.0
> >> 386c367
> >> < google-genai==1.2.0
> >> ---
> >> > google-genai==1.20.0
> >> 389d369
> >> < graphql-core==3.2.6
> >> 398d377
> >> < grpcio-tools==1.62.3
> >> 403,404d381
> >> < hatch==1.14.1
> >> < hatchling==1.27.0
> >> 413c390
> >> < httpx==0.27.0
> >> ---
> >> > httpx==0.28.1
> >> 418d394
> >> < hyperlink==21.0.0
> >> 421,423d396
> >> < icdiff==2.0.7
> >> < id==1.5.0
> >> < identify==2.6.12
> >> 426d398
> >> < imagesize==1.4.1
> >> 430d401
> >> < incremental==24.7.2
> >> 433,435d403
> >> < iniconfig==2.1.0
> >> < inputimeout==1.0.4
> >> < ipdb==0.13.13
> >> 449d416
> >> < joserfc==1.2.2
> >> 451d417
> >> < jsonpatch==1.33
> >> 455,456d420
> >> < jsonpointer==3.0.0
> >> < jsonschema-path==0.3.4
> >> 463d426
> >> < kerberos==1.3.1
> >> 465,466d427
> >> < keyrings.alt==5.0.2
> >> < kgb==7.2
> >> 494,495d454
> >> < mmh3==5.1.0
> >> < mongomock==4.3.0
> >> 497,498d455
> >> < moto==5.1.8
> >> < mpmath==1.3.0
> >> 508,512d464
> >> < mypy-boto3-appflow==1.39.0
> >> < mypy-boto3-rds==1.39.1
> >> < mypy-boto3-redshift-data==1.39.0
> >> < mypy-boto3-s3==1.39.5
> >> < mypy==1.17.0
> >> 514c466
> >> < mysql-connector-python==9.3.0
> >> ---
> >> > mysql-connector-python==9.4.0
> >> 521,523d472
> >> < networkx==3.5
> >> < nh3==0.3.0
> >> < nodeenv==1.9.1
> >> 528,529d476
> >> < openapi-schema-validator==0.6.3
> >> < openapi-spec-validator==0.7.2
> >> 560d506
> >> < pathable==0.4.4
> >> 563d508
> >> < pdbr==0.9.2
> >> 569,570c514
> >> < pinotdb==5.6.0
> >> < pipdeptree==2.28.0
> >> ---
> >> > pinotdb==5.7.0
> >> 574d517
> >> < plyvel==1.5.1
> >> 577,579d519
> >> < pprintpp==0.4.0
> >> < pre-commit-uv==4.1.4
> >> < pre_commit==4.2.0
> >> 592d531
> >> < py-partiql-parser==0.6.1
> >> 608d546
> >> < pyenchant==3.2.2
> >> 610,611d547
> >> < pygraphviz==1.14
> >> < pyiceberg==0.9.1
> >> 622,632d557
> >> < pytest-asyncio==0.25.0
> >> < pytest-cov==6.2.1
> >> < pytest-custom-exit-code==0.3.0
> >> < pytest-icdiff==0.9
> >> < pytest-instafail==0.5.0
> >> < pytest-mock==3.14.1
> >> < pytest-rerunfailures==15.1
> >> < pytest-timeouts==1.2.1
> >> < pytest-unordered==0.7.0
> >> < pytest-xdist==3.8.0
> >> < pytest==8.4.1
> >> 642d566
> >> < python-on-whales==0.78.0
> >> 652d575
> >> < readme_renderer==44.0
> >> 659d581
> >> < requests-mock==1.12.1
> >> 664,665d585
> >> < responses==0.25.7
> >> < restructuredtext_lint==1.4.0
> >> 667,668d586
> >> < rfc3339-validator==0.1.4
> >> < rfc3986==2.0.0
> >> 670d587
> >> < rich-click==1.8.9
> >> 674d590
> >> < roman-numerals-py==3.1.0
> >> 679d594
> >> < ruff==0.12.3
> >> 688d602
> >> < semver==3.0.4
> >> 690,691c604
> >> < sentinels==1.0.0
> >> < sentry-sdk==2.33.1
> >> ---
> >> > sentry-sdk==2.33.2
> >> 703d615
> >> < snowballstemmer==3.0.1
> >> 709,725d620
> >> < sphinx-argparse==0.5.2
> >> < sphinx-autoapi==3.6.0
> >> < sphinx-autobuild==2024.10.3
> >> < sphinx-copybutton==0.5.2
> >> < sphinx-jinja==2.0.2
> >> < sphinx-rtd-theme==3.0.2
> >> < sphinx_design==0.6.1
> >> < sphinxcontrib-applehelp==2.0.0
> >> < sphinxcontrib-devhelp==2.0.0
> >> < sphinxcontrib-htmlhelp==2.1.0
> >> < sphinxcontrib-httpdomain==1.8.1
> >> < sphinxcontrib-jquery==4.1
> >> < sphinxcontrib-jsmath==1.0.1
> >> < sphinxcontrib-qthelp==2.0.0
> >> < sphinxcontrib-redoc==1.6.0
> >> < sphinxcontrib-serializinghtml==2.0.0
> >> < sphinxcontrib-spelling==8.0.1
> >> 737d631
> >> < strictyaml==1.7.3
> >> 740d633
> >> < sympy==1.14.0
> >> 752d644
> >> < time-machine==2.16.0
> >> 755d646
> >> < tomli_w==1.2.0
> >> 758d648
> >> < towncrier==24.8.0
> >> 762,763d651
> >> < trove-classifiers==2025.5.9.12
> >> < twine==6.1.0
> >> 765,774d652
> >> < types-Deprecated==1.2.15.20250304
> >> < types-Markdown==3.8.0.20250708
> >> < types-PyMySQL==1.1.0.20250711
> >> < types-PyYAML==6.0.12.20250516
> >> < types-aiofiles==24.1.0.20250708
> >> < types-certifi==2021.10.8.3
> >> < types-cffi==1.17.0.20250523
> >> < types-croniter==6.0.0.20250626
> >> < types-docutils==0.21.0.20250722
> >> < types-paramiko==3.5.0.20250708
> >> 776,778d653
> >> < types-pyOpenSSL==24.1.0.20240722
> >> < types-python-dateutil==2.9.0.20250708
> >> < types-python-slugify==8.0.2.20240310
> >> 780d654
> >> < types-redis==4.6.0.20241004
> >> 782,784d655
> >> < types-setuptools==80.9.0.20250529
> >> < types-tabulate==0.9.0.20241207
> >> < types-toml==0.10.8.20240310
> >> 794d664
> >> < userpath==1.9.2
> >> 799c669
> >> < validators==0.34.0
> >> ---
> >> > validators==0.35.0
> >> 806c676
> >> < weaviate-client==4.9.6
> >> ---
> >> > weaviate-client==4.16.3
> >> 809c679
> >> < websockets==14.2
> >> ---
> >> > websockets==15.0.1
> >> 815d684
> >> < yamllint==1.37.1
> >> ________________________________
> >>  Strike Technologies, LLC (“Strike”) is part of the GTS family of
> >> companies. Strike is a technology solutions provider, and is not a
> broker
> >> or dealer and does not transact any securities related business directly
> >> whatsoever. This communication is the property of Strike and its
> >> affiliates, and does not constitute an offer to sell or the
> solicitation of
> >> an offer to buy any security in any jurisdiction. It is intended only
> for
> >> the person to whom it is addressed and may contain information that is
> >> privileged, confidential, or otherwise protected from disclosure.
> >> Distribution or copying of this communication, or the information
> contained
> >> herein, by anyone other than the intended recipient is prohibited. If
> you
> >> have received this communication in error, please immediately notify
> Strike
> >> at i...@striketechnologies.com, and delete and destroy any copies
> hereof.
> >> ________________________________
> >>
> >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> attachments
> >> are intended solely for the addressee. This transmission is covered by
> the
> >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> >> information contained in this transmission is confidential in nature and
> >> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> >> legal privilege. Your use or disclosure of this information for any
> purpose
> >> other than that intended by its transmittal is strictly prohibited, and
> may
> >> subject you to fines and/or penalties under federal and state law. If
> you
> >> are not the intended recipient of this transmission, please DESTROY ALL
> >> COPIES RECEIVED and confirm destruction to the sender via return
> >> transmittal.
> >>
> >
>

Reply via email to