Awesome thanks. I must have coincidentally done it with Breeze since I did see the generated providers updated but I must have never gone back to run the pip install. I have a working environment now to run with.
Kris On Fri, Mar 24, 2023 at 1:48 PM Jarek Potiuk <[email protected]> wrote: > > The "update-provider-dependencies" is the right pre-commit name BTW, > not "generate...". > > On Fri, Mar 24, 2023 at 6:47 PM Jarek Potiuk <[email protected]> wrote: > > > > The provider.yaml is the source of information, but when you install > > application via setup.py locally after changing it, you need to > > generate the "generated/provider_dependencies.json" - this is the file > > that is used by setup.py when you install it via ".[extras]". This > > file will get automatically updated by pre-commit, when you commit > > your code: > > > > pre-commit run generate-provider-dependencies --all-files > > > > or if you use breeze (this one supports autocompletion): > > > > breeze static-checks --type update-providers-dependencies --all-files > > > > But you can also update it manually if you wish - it is plain json > > file, easy to edit. > > > > Besides, in order to check if your changes result in a consistent > > no-conflict set of dependencies you can run this command to have an > > automated attempt (by pip) to upgrade to the latest set of > > dependencies that are consistent with all the limits: > > > > breeze ci-image build --upgrade-to-newer-dependencies > > > > This should attempt to build the CI image which will contain the > > latest versions of all the dependencies with all the limits from all > > provider.yaml (after converting to > > generated/provider_dependencies.json) applied. You should be able to > > enter such an image with an interactive shell session with `breeze` or > > `breeze shell` command after it succeeds. > > This might not succeed though in case of unresolvable conflicts with > > other libraries (and the work of google to upgrade some old > > dependencies is what is necessary to solve some of them - especially > > protobuf conflicts) > > > > J. > > > > On Fri, Mar 24, 2023 at 4:32 PM Kristopher Kane <[email protected]> > > wrote: > > > > > > While waiting on an adjacent PR > > > (https://github.com/apache/airflow/pull/30067) to holistically update > > > Google Cloud provider deps, I'm attempting to update the provider.yaml > > > for Google Cloud that upgrades google-cloud-dataproc from 5.0.0 to > > > 5.4.0 in order to test an enhancement to Dataproc operators. I took > > > that PR's provider.yaml in my branch in order to start. > > > > > > I cannot get a local virtualenv to get any version of > > > google-cloud-dataproc but the currently resolved 5.0.0 even though I > > > force it to '==5.4.0' in providers.yaml. > > > > > > Here is what I have done to force it to no avail: > > > 1) Forced providers.yaml for Google Cloud to > > > "google-cloud-dataproc==5.4.0" > > > 2) Not using a constraints file to rule it out. - run with: > > > INSTALL_PROVIDERS_FROM_SOURCES="true" pip install -U -e > > > ".[devel,google,postgres]" > > > 3) Observe that generated/provider_dependencies.json actually has > > > "google-cloud-dataproc==5.4.0" > > > > > > This technique does install successfully without the constraints file > > > but I still only get version 5.0.0 in 'pip list'. Is there somewhere > > > I should be looking other than the provider's provider.yaml? > > > > > > Thanks, > > > > > > Kris > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
