The provider.yaml is the source of information, but when you install application via setup.py locally after changing it, you need to generate the "generated/provider_dependencies.json" - this is the file that is used by setup.py when you install it via ".[extras]". This file will get automatically updated by pre-commit, when you commit your code:
pre-commit run generate-provider-dependencies --all-files or if you use breeze (this one supports autocompletion): breeze static-checks --type update-providers-dependencies --all-files But you can also update it manually if you wish - it is plain json file, easy to edit. Besides, in order to check if your changes result in a consistent no-conflict set of dependencies you can run this command to have an automated attempt (by pip) to upgrade to the latest set of dependencies that are consistent with all the limits: breeze ci-image build --upgrade-to-newer-dependencies This should attempt to build the CI image which will contain the latest versions of all the dependencies with all the limits from all provider.yaml (after converting to generated/provider_dependencies.json) applied. You should be able to enter such an image with an interactive shell session with `breeze` or `breeze shell` command after it succeeds. This might not succeed though in case of unresolvable conflicts with other libraries (and the work of google to upgrade some old dependencies is what is necessary to solve some of them - especially protobuf conflicts) J. On Fri, Mar 24, 2023 at 4:32 PM Kristopher Kane <[email protected]> wrote: > > While waiting on an adjacent PR > (https://github.com/apache/airflow/pull/30067) to holistically update > Google Cloud provider deps, I'm attempting to update the provider.yaml > for Google Cloud that upgrades google-cloud-dataproc from 5.0.0 to > 5.4.0 in order to test an enhancement to Dataproc operators. I took > that PR's provider.yaml in my branch in order to start. > > I cannot get a local virtualenv to get any version of > google-cloud-dataproc but the currently resolved 5.0.0 even though I > force it to '==5.4.0' in providers.yaml. > > Here is what I have done to force it to no avail: > 1) Forced providers.yaml for Google Cloud to "google-cloud-dataproc==5.4.0" > 2) Not using a constraints file to rule it out. - run with: > INSTALL_PROVIDERS_FROM_SOURCES="true" pip install -U -e > ".[devel,google,postgres]" > 3) Observe that generated/provider_dependencies.json actually has > "google-cloud-dataproc==5.4.0" > > This technique does install successfully without the constraints file > but I still only get version 5.0.0 in 'pip list'. Is there somewhere > I should be looking other than the provider's provider.yaml? > > Thanks, > > Kris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
