As a small pointer - the logic is implemented here: https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py :
It also handles installing airflow from git url and pr # (added recently by Liu Zhe You) and compiles assets in case we install from those. It has now a good 1000+ lines of code to handle all cases :D :D. Good luck :) J. On Wed, Nov 5, 2025 at 11:08 AM Jarek Potiuk <[email protected]> wrote: > BTW. We probably (and feel free :) could make it work when I think of it. > currently --use-installation-from-dist works in two steps:. > > a) installs specified airflow (or airflow from dist if found in dist) in a > single operation without constraints > > b) installs all other distributions without constraints > > > We could probably make it three step and handle this edge case by > installing airflow separately (with constraints) then providers and all > other things > > I will leave it as exercise for the readers though - the logic there is > already very complex and handles quite a few other cases and this one might > break some of them. :) > > śr., 5 lis 2025, 10:57 użytkownik Jarek Potiuk <[email protected]> napisał: > >> The problem is `--use-distributions-from-dist`. The >> `--use-airflow-version` and --use-distributions-from-dist does not use the >> `--use-airflow-version` constraints because it will then limit whatever is >> instlalled from dist (constraints from 3.1.1 have older version of edge >> provider and if you try to install new edge provider from "dist" and >> constraints pointing to the 3.1.1 provider they will conflict. >> >> This is our classic installation pattern where you install airflow + >> whatever was released with constraints and **then** in a separate step you >> install any updates. Using both constraints and something that deviates >> from them in the same command is not going to work. >> >> So the right way of installing the provider for old versions is: >> >> * breeze start-airflow --use-airflow-version 3.1.1 -> uses constraints >> * install new provider in terminal inside >> * restart the components in other tmux terminals >> >> The --use-distributions-from-dist also works but by the fact that we want >> to install all the distributions together (including one that deviates from >> original constraints) - we cannot use constraints for it. >> >> J. >> >> >> >> On Tue, Nov 4, 2025 at 11:09 PM Jens Scheffler <[email protected]> >> wrote: >> >>> Hi Jarek, >>> >>> what I actually did to be preceise: >>> >>> 1) Checked out to tag "providers-edge3/1.4.1rc2" >>> >>> 2) Build edge3 and docker on that tag (I need docker as well for an >>> extended test...) via >>> breeze release-management prepare-provider-distributions edge3 docker >>> --skip-tag-check >>> >>> 3) Checkout (latest) main to ensure we use feshest breeze >>> >>> 4) Start system via breeze using: >>> breeze start-airflow --python 3.12 --load-example-dags --backend >>> postgres --executor EdgeExecutor --answer y --use-airflow-version 3. >>> 1.1 --use-distributions-from-dist >>> >>> Re tested tonight again, results: >>> >>> * 3.0.6 - OK >>> * 3.1.0 - not compatible with edge3 1.4.1 :-D >>> * 3.1.1 - fails in structlog >>> * 3.1.2rc1 - OK >>> * 3.1.2rc2 - OK >>> >>> Made no manual install. But as 3.1.2 is OK I am okay anyway. >>> >>> On 11/4/25 15:36, Jarek Potiuk wrote: >>> >> Don't know why I catched the "wrong" constraints" via installing with >>> > breeze - might be constraints are not considered when installed a >>> specific >>> > airflow version and then providers are upgraded? >>> > >>> > It could be that later when you upgraded some providers, this also get >>> > bumped or maybe some uv sync in the image happened. When I used >>> > `--use-airflow-version 3.1.1` and **then** installed just >>> > `apache-airflow-providers-edge3==1.4.1rc2` in breeze - everything >>> worked >>> > fine. It also can be you have some customisation scripts in breeze >>> that do >>> > some upgrades afterwards and you have not noticed. >>> > >>> > On Tue, Nov 4, 2025 at 3:34 PM Jarek Potiuk<[email protected]> wrote: >>> > >>> >> It's actually already fixed in main and cherry-picked to 3.1.2 - > so >>> in >>> >> 3.1.2 it's OK to have structlog 25.5.0. See >>> >> https://github.com/apache/airflow/pull/57335 and it's been >>> cherry-picked >>> >> and merged as part ofhttps://github.com/apache/airflow/pull/57640 >>> >> >>> >> J, >>> >> >>> >> >>> >> On Tue, Nov 4, 2025 at 3:04 PM Scheffler Jens (XC-AS/EAE-ADA-T) via >>> dev < >>> >> [email protected]> wrote: >>> >> >>> >>> Hi Amough, >>> >>> >>> >>> you are right but constraints in Airflow 3.1.2rc[1|2] both pin >>> >>> structlog==25.5.0 >>> >>> see >>> >>> >>> https://raw.githubusercontent.com/apache/airflow/constraints-3.1.2rc2/constraints-3.10.txt >>> >>> >>> >>> So wanted to raise this if not detected that we need to fix >>> constraints >>> >>> before releasing 3.1.2 as well. >>> >>> Don't know why I catched the "wrong" constraints" via installing with >>> >>> breeze - might be constraints are not considered when installed a >>> specific >>> >>> airflow version and then providers are upgraded? >>> >>> >>> >>> Jens >>> >>> >>> >>> -----Original Message----- >>> >>> From: Amogh Desai<[email protected]> >>> >>> Sent: Dienstag, 4. November 2025 10:07 >>> >>> To:[email protected] >>> >>> Subject: Re: [VOTE] Airflow Providers prepared on November 03, 2025 >>> >>> >>> >>> Jens, we have constrained structlog to disallow 25.5.0: >>> >>> >>> >>> >>> https://raw.githubusercontent.com/apache/airflow/constraints-3.1.0/constraints-3.10.txt >>> >>> >>> >>> So, that should not be a problem for you :), maybe it was installed >>> in >>> >>> your environment. >>> >>> >>> >>> Thanks & Regards, >>> >>> Amogh Desai >>> >>> >>> >>> >>> >>> On Tue, Nov 4, 2025 at 3:59 AM Jens Scheffler<[email protected]> >>> >>> wrote: >>> >>> >>> >>>> +1 (binding) - Checked SVN, Check in Docker, Reproducible package >>> >>>> +build, >>> >>>> Licenses, Signatures >>> >>>> >>> >>>> Installed the edge provider with current common-compat in Airflow >>> >>>> 2.11.0, 3.0.6, 3.1.2rc1 and main and tested with EdgeExecutor and >>> >>>> Integration Test Dag, all loking good. >>> >>>> >>> >>>> Will test detailled issue resolution (subpath etc) tomorrow but >>> until >>> >>>> now all good to release. Best release we had for Edge3 ever :-D >>> >>>> >>> >>>> Note: Was not able to install providers with Airflow 3.1.1 as >>> >>>> structlog==25.5.0 installed by breeze has some incompatability and >>> >>>> breze start failed with - assume this is not blocking providers >>> >>>> release - assume this is because constraints are installed from main >>> >>>> and not 3.1.1 in breeze: >>> >>>> >>> >>>> Traceback (most recent call last): >>> >>>> File "/usr/python/bin/airflow", line 4, in <module> >>> >>>> from airflow.__main__ import main >>> >>>> File >>> >>>> "/usr/python/lib/python3.12/site-packages/airflow/__init__.py", >>> >>>> line 79, in <module> >>> >>>> settings.initialize() >>> >>>> File >>> >>>> "/usr/python/lib/python3.12/site-packages/airflow/settings.py", >>> >>>> line 698, in initialize >>> >>>> LOGGING_CLASS_PATH = configure_logging() >>> >>>> ^^^^^^^^^^^^^^^^^^^ >>> >>>> File >>> >>>> >>> "/usr/python/lib/python3.12/site-packages/airflow/logging_config.py", >>> >>>> line 88, in configure_logging >>> >>>> from airflow._shared.logging import configure_logging, >>> >>>> init_log_folder, translate_config_values >>> >>>> File >>> >>>> >>> "/usr/python/lib/python3.12/site-packages/airflow/_shared/logging/__in >>> >>>> it__.py", >>> >>>> >>> >>>> line 27, in <module> >>> >>>> from .structlog import configure_logging, init_log_file, >>> >>>> init_log_folder >>> >>>> File >>> >>>> >>> "/usr/python/lib/python3.12/site-packages/airflow/_shared/logging/stru >>> >>>> ctlog.py", >>> >>>> >>> >>>> line 38, in <module> >>> >>>> from .percent_formatter import PercentFormatRender >>> >>>> File >>> >>>> >>> "/usr/python/lib/python3.12/site-packages/airflow/_shared/logging/perc >>> >>>> ent_formatter.py", >>> >>>> >>> >>>> line 29, in <module> >>> >>>> from structlog.dev import ConsoleRenderer, Styles >>> >>>> ImportError: cannot import name 'Styles' from 'structlog.dev' >>> >>>> (/usr/python/lib/python3.12/site-packages/structlog/dev.py) >>> >>>> >>> >>>> Error: check_environment returned 1. Exiting. >>> >>>> >>> >>>> Call that I used for testing: breeze start-airflow --python 3.12 >>> >>>> --load-example-dags --backend postgres --executor EdgeExecutor >>> >>>> --answer y --use-airflow-version 3.1.1 --use-distributions-from-dist >>> >>>> >>> >>>> Tested also with Airflow 3.1.0 and saw as intended that edge3 >>> provider >>> >>>> 1.4.1rc2 can not installed with Airflow 3.1.0 as UI plugin >>> incompatible. >>> >>>> >>> >>>> Using Python 3.12.12 environment at: /usr/python >>> >>>> × No solution found when resolving dependencies: >>> >>>> ╰─▶ Because only apache-airflow-providers-edge3==1.4.1 is >>> available >>> >>>> and apache-airflow-providers-edge3==1.4.1 depends on one of: >>> >>>> apache-airflow>=2.10.0,<3.1.0 >>> >>>> apache-airflow>3.1.0 >>> >>>> we can conclude that all versions of >>> >>>> apache-airflow-providers-edge3 depend on one of: >>> >>>> apache-airflow>=2.10.0,<3.1.0 >>> >>>> apache-airflow>3.1.0 >>> >>>> >>> >>>> And because you require apache-airflow==3.1.0 and >>> >>>> apache-airflow-providers-edge3, we can conclude that your >>> requirements >>> >>>> are unsatisfiable. >>> >>>> >>> >>>> On 11/3/25 13:06, Elad Kalif wrote: >>> >>>>> Hey all, >>> >>>>> >>> >>>>> I have just cut the ad-hoc wave Airflow Providers packages. This >>> >>>>> email is calling a vote on the release, which will last for 72 >>> hours >>> >>>>> - which means that it will end on November 06, 2025 12:05 PM UTC >>> and >>> >>>>> until 3 binding +1 votes have been received. >>> >>>>> >>> >>>>> Consider this my (binding) +1. >>> >>>>> >>> >>>>> Airflow Providers are available at: >>> >>>>> https://di/ >>> >>>>> st.apache.org >>> %2Frepos%2Fdist%2Fdev%2Fairflow%2Fproviders%2F&data=05% >>> >>>>> 7C02%7CJens.Scheffler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1b819 >>> >>>>> >>> e4d%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681481189% >>> >>>>> >>> 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIs >>> >>>>> >>> IlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata= >>> >>>>> u%2BGuEPt6aFORkaSjATUG2TVoDJiUJmNDiTM9Txe4cQc%3D&reserved=0 >>> >>>>> >>> >>>>> *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the binary >>> >>>>> Python "sdist" release - they are also official "sources" for >>> the >>> >>>> Provider >>> >>>>> distributions. >>> >>>>> >>> >>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary >>> >>>>> Python "wheel" release. >>> >>>>> >>> >>>>> The test procedure for PMC members is described in >>> >>>>> >>> >>>> https://gith/ >>> >>>> ub.com >>> %2Fapache%2Fairflow%2Fblob%2Fmain%2Fdev%2FREADME_RELEASE_PROVIDE >>> >>>> >>> RS.md%23verify-the-release-candidate-by-pmc-members&data=05%7C02%7CJen >>> >>>> s.Scheffler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1b819e4d%7C0ae51e >>> >>>> >>> 1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681516854%7CUnknown%7CTW >>> >>>> >>> FpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs >>> >>>> >>> IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6sgJ88Z0t%2F1Qnhxr >>> >>>> wPTnej7SfgLNf6hI7qYxnJBl6RU%3D&reserved=0 >>> >>>>> The test procedure for and Contributors who would like to test this >>> >>>>> RC is described in: >>> >>>>> >>> >>>> https://gith/ >>> >>>> ub.com >>> %2Fapache%2Fairflow%2Fblob%2Fmain%2Fdev%2FREADME_RELEASE_PROVIDE >>> >>>> >>> RS.md%23verify-the-release-candidate-by-contributors&data=05%7C02%7CJe >>> >>>> ns.Scheffler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1b819e4d%7C0ae51 >>> >>>> >>> e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681546634%7CUnknown%7CT >>> >>>> >>> WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI >>> >>>> >>> sIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=j%2B4anx9P7T5uTdc >>> >>>> omIOr%2BOr2%2FoDHQnntn1Pvhn33BHs%3D&reserved=0 >>> >>>>> >>> >>>>> Public keys are available at: >>> >>>>> https://di/ >>> >>>>> st.apache.org >>> %2Frepos%2Fdist%2Frelease%2Fairflow%2FKEYS&data=05%7C02 >>> >>>>> %7CJens.Scheffler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1b819e4d% >>> >>>>> >>> 7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681574562%7CUn >>> >>>>> >>> known%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAi >>> >>>>> >>> OiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=enB4 >>> >>>>> BNSldvybyXLUzIJIz5jE2tCnZ904nXCrgzoZp%2Bg%3D&reserved=0 >>> >>>>> >>> >>>>> Please vote accordingly: >>> >>>>> >>> >>>>> [ ] +1 approve >>> >>>>> [ ] +0 no opinion >>> >>>>> [ ] -1 disapprove with the reason >>> >>>>> >>> >>>>> Only votes from PMC members are binding, but members of the >>> >>>>> community are encouraged to test the release and vote with >>> >>> "(non-binding)". >>> >>>>> Please note that the version number excludes the 'rcX' string. >>> >>>>> This will allow us to rename the artifact without modifying the >>> >>>>> artifact checksums when we actually release. >>> >>>>> >>> >>>>> The status of testing the providers by the community is kept here: >>> >>>>> https://gi/ >>> >>>>> thub.com >>> %2Fapache%2Fairflow%2Fissues%2F57739&data=05%7C02%7CJens.Sch >>> >>>>> effler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1b819e4d%7C0ae51e190 >>> >>>>> >>> 7c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681602068%7CUnknown%7CTWF >>> >>>>> >>> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI >>> >>>>> >>> sIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PPDrnZgPIfBJWRp >>> >>>>> 5VyEq1Ij2NGJXs9qK3b9mJA7Cwjk%3D&reserved=0 >>> >>>>> >>> >>>>> The issue is also the easiest way to see important PRs included in >>> >>>>> the RC candidates. >>> >>>>> Detailed changelog for the providers will be published in the >>> >>>> documentation >>> >>>>> after the >>> >>>>> RC candidates are released. >>> >>>>> >>> >>>>> You can find the RC packages in PyPI following these links: >>> >>>>> >>> >>>>> https://py/ >>> >>>>> pi.org >>> %2Fproject%2Fapache-airflow-providers-edge3%2F1.4.1rc2%2F&data >>> >>>>> =05%7C02%7CJens.Scheffler%40de.bosch.com >>> %7Cf6343926cc624857a11208de1 >>> >>>>> >>> b819e4d%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638978440681628 >>> >>>>> >>> 771%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAw >>> >>>>> >>> MCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sd >>> >>>>> ata=HWxblv2u1%2B6Pr3QrB5N1s9k7wnOcgLZNi7qsgW3t9i0%3D&reserved=0 >>> >>>>> >>> >>>>> Cheers, >>> >>>>> Elad Kalif >>> >>>>> >>> >>>> >>> --------------------------------------------------------------------- >>> >>>> To unsubscribe, e-mail:[email protected] >>> >>>> For additional commands, e-mail:[email protected] >>> >>>> >>> >>>> >>> >>> --------------------------------------------------------------------- >>> >>> To unsubscribe, e-mail:[email protected] >>> >>> For additional commands, e-mail:[email protected] >>> >>> >> >>
