Great work! That sounds like a really nice improvement :) Le sam. 1 juin 2024 à 10:48, Jarek Potiuk <ja...@potiuk.com> a écrit :
> Hello everyone, > > TL;DR; I have finally got to something we planned when we switched to UV, I > have a green PR where we introduced automated management of "lower-bounds" > dependencies in Airflow and all providers (thanks to uv`s --lowest-direct > resolution mechanism). > > The PR is here: https://github.com/apache/airflow/pull/39946 . It's ready > to review and merge (green). > > Thanks to Maciek nagging me on slack and helping with initial checks - I > managed to complete it before going to Community Over Code this week. > > At the end of the email I summarized changes in dependencies that were > needed to do it (so basically all the missing lower-bounds that the tests > helped to detect and fix). > > Once it is merged, our CI will run a special > "LowestDirectDependencyResolution" test suite that will fail if the tests > run in PR in Airflow and any Provider uses a feature that requires adding > `>=` limit for any library version (lower binding). This means that we will > finally have "proper" lower bindings for both Airflow and Providers and > there will be no more cases where Airflow or any Provider fails because > someone has an old version of a library installed. > > For example if you are using bedrock (amazon provider) - our Amazon > provider had botocore > 1.3.3 but the tests found that Bedrock is only > available in 1.34 and we have to bump it. Once we merge the PR, those cases > will be detected automatically and you will have to fix them before you > merge your PRs. > > It's done in the way that in the special test suite, dependencies are > downgraded to lowest direct ones before our unit tests are run. This is > done for Airflow tests and for each provider separately, so we are able to > detect missing lower bounds very accurately - separately for core Airflow > and separately for each provider. > > When the test fails in CI, it will be very easy to reproduce it locally > with Breeze. For example if you work on google provider and it fails you > run this command: > > breeze shell --force-lowest-dependencies --test-type "Providers[google]" > > This will drop you in Breeze shell, and downgrade google provider > dependencies to lowest "direct" ones and allow you to run pytest tests > there and fix the problem by manually installing newer dependency versions > and re-running the tests. > > Then you can iterate over tests, manually downgrade and upgrade > dependencies as you see fit and eventually when you figure out the minimum > binding, you just add it to provider.yaml, run pre-commit and then > restarting the command above can be repeated. > > I've added detailed instructions on how to approach fixing "lowest > dependencies" problems, and when the tests fail in CI, you will be directed > to those instructions. I even described how to effectively use bisecting > to easily find the actual version of dependency that needs to be set in > such cases. > > ------------------------------- > > The list of dependency fixes: > > Airflow: > > - "asgiref", > + "asgiref>=2.3.0", > - "connexion[flask]>=2.10.0,<3.0", > + "connexion[flask]>=2.14.2,<3.0", > - "cryptography>=39.0.0", > + "cryptography>=41.0.0", > - "flask-caching>=1.5.0", > + "flask-caching>=2.0.0", > - "flask-wtf>=0.15", > + "flask-wtf>=1.1.0", > - "flask>=2.2,<2.3", > + "flask>=2.2.1,<2.3", > - "httpx", > + "httpx>=0.18.0", > - "lazy-object-proxy", > + "lazy-object-proxy>=1.2.0", > - "opentelemetry-exporter-otlp", > - "packaging>=14.0", > + "opentelemetry-exporter-otlp>=1.15.0", > + "packaging>=22.0", > - "pluggy>=1.0", > - "psutil>=4.2.0", > + "pluggy>=1.5.0", > + "psutil>=5.8.0", > - "python-dateutil>=2.3", > + "python-dateutil>=2.7.0", > + "requests-toolbelt>=0.4.0", > - "setproctitle>=1.1.8", > + "setproctitle>=1.3.3", > - "tenacity>=6.2.0,!=8.2.0", > + "tenacity>=8.0.0,!=8.2.0", > > Providers: > > Amazon: > > - - boto3>=1.33.0 > - - botocore>=1.33.0 > + - boto3>=1.34.0 > + - botocore>=1.34.0 > - - watchtower>=2.0.1,<4 > + - watchtower>=3.0.0,<4 > - - asgiref > + - asgiref>=2.3.0 > - - jmespath > + - jmespath>=0.7.0 > > Amazon[aiobotocore] > - - aiobotocore[boto3]>=2.5.3 > + - aiobotocore[boto3]>=2.10.0 > > Apache Flink: > - - cryptography>=2.0.0 > + - cryptography>=41.0.0 > > Apache Hive: > - - thrift>=0.9.2 > + - thrift>=0.11.0 > + - jmespath>=0.7.0 > > Apache Kylin: > - - kylinpy>=2.6 > + - kylinpy>=2.7.0 > > Apache Spark: > - - pyspark > + - pyspark>=3.0.0 > > CNCF Kubernetes: > - - cryptography>=2.0.0 > + - cryptography>=41.0.0 > > FAB: > - - jmespath > + - jmespath>=0.7.0 > > Github: > - - PyGithub!=1.58 > + - PyGithub>=2.1.1 > > Google: > + - dill>=0.2.3 > - - google-analytics-admin > + - google-analytics-admin>=0.9.0 > - - google-cloud-bigquery<3.21.0,>=3.0.1 > + - google-cloud-bigquery<3.21.0,>=3.4.0 > - - google-cloud-run>=0.9.0 > + - google-cloud-run>=0.10.0 > - - httpx > + - httpx>=0.18.0 > - - looker-sdk>=22.2.0 > - - pandas-gbq > + - looker-sdk>=22.4.0 > + - pandas-gbq>=0.7.0 > - - PyOpenSSL > - - python-slugify>=5.0 > + - python-slugify>=7.0.0 > + - PyOpenSSL>=23.0.0 > + - tenacity>=8.1.0 > > Grpc: > - - grpcio>=1.15.0 > + - grpcio>=1.38.0 > > Microsoft Azure: > - - azure-mgmt-cosmosdb > + - azure-mgmt-cosmosdb>=3.0.0 > - - azure-storage-file-share > + - azure-storage-file-share>=12.7.0 > - - azure-synapse-spark > + - azure-synapse-spark>=0.2.0 > > Mongo: > > devel-dependencies: > - - mongomock > + - mongomock>=3.12.0 > > MySQL: > - - mysqlclient>=1.3.6 > + - mysqlclient>=1.4.0 > > Odbc: > - - pyodbc > + - pyodbc>=4.0.24 > > Pinecone: > - - pinecone-client>=3.0.0 > + - pinecone-client>=3.1.0 > > SFTP: > - - paramiko>=2.8.0 > + - paramiko>=2.9.0 > > SSH: > - - paramiko>=2.6.0 > + - paramiko>=2.9.0 > > Tableau: > - - tableauserverclient > + - tableauserverclient>=0.25 > > Vertica: > - - vertica-python>=0.5.1 > + - vertica-python>=0.6.0 > > J. >
