Yep. Sounds cool. On Thu, Feb 11, 2021 at 4:20 PM Kaxil Naik <[email protected]> wrote:
> Yeah that sounds good too. > > On Thu, Feb 11, 2021, 15:17 Ash Berlin-Taylor <[email protected]> wrote: > >> The problem with tightly specifying all the constraints (including the >> providers) is that it means you can't do something like `pip install -U >> apache-airflow-providers-google` but have the _core_ Airflow constrained. >> (Pip may be better at upgrading less in cases like this now?) >> >> I have a proposal: two constraint files (for each python version) -- a >> "core" and a "full". >> >> The "full" is as you propose, with the providers, and their deps in the >> file. >> >> The "core" is _just_ the core requirements for Airflow without any >> providers, or any transitive deps. This will include deps non-provider >> extras though. >> >> How does that sound? >> >> -ash >> >> On Thu, 11 Feb, 2021 at 14:07, Kaxil Naik <[email protected]> wrote: >> >> Yup, that is correct. That will allow us to make sure that whenever >> Airflow was released, all the dependencies including the provider are >> snapshotted in constraints. So even if someone tries to install the same >> version a year later with constraints it should work fine without having to >> worry about the latest version of a specific provider breaking it. >> >> And then users can ofcourse install or upgrade providers after that if >> they like. >> >> Kaxil - did I understand it correctly ? If so - I think this is the best >>> we can do to keep two properties: >> >> * repeatable installation of already released version >>> * capability (and easy way of) upgrading to latest providers >> >> >> On Thu, Feb 11, 2021 at 12:42 PM Jarek Potiuk <[email protected]> wrote: >> >>> > Oh I misunderstood -- I thought you were suggesting putting the >>> transitive deps of apache-airflow-providers-google v2.0 into constraint-2.0 >>> files etc. >>> >>> Well. That too. The transitive deps already are in the constraint files >>> and that will remain, I think this is the main reason why we have >>> the constraint files. The main reason why constraint files are "snapshots >>> of all dependencies'' (currently they exclude providers) is to have a >>> repeatable installation. Let me reiterate then how I understand Kaxil's >>> proposal (which I think makes perfect sense). >>> >>> I really see the "extras" and constraints as a convenient way for users >>> to install a released version of airflow with the set of providers they >>> choose and dependencies in versions that we know are working. No more, no >>> less. Then they are free to upgrade the dependencies as they wish. >>> >>> How I see the current proposal - the constraint files will only differ >>> from the current ones by adding 'apache-airflow-providers-google==1.0.0" >>> for example. Literally (compared to the current process) it means that we >>> will just add the version of providers that were released at the time the >>> airflow X.Y.Z version was released (this is one line change in generation >>> of constraints as we have now). So the final constraint file for 2.0.1 >>> version will look like this: >>> >>> apache-airfow-providers-google=1.0.0 >>> google-cloud-automl=1.9.0 >>> .... >>> ~500 other dependencies with == >>> ... >>> >>> Those constraint files will contain all providers that were released at >>> the time of airflow X.Y.Z release and all their transitive dependencies. >>> This way if you run 'pip apache-airflow[google, amazon]==2.0.1 >>> --constraints ...../2.0.1/python3.6.txt ' - you will always get the >>> google-provider==1.0.0 installed and amazon 1.0.0 as well. >>> >>> And that preserves the only capability that constraint files + extras >>> give - an easy installation path when you want to install an older version >>> of airflow for the first time - with pretty much guarantee that it will >>> always work (this is the only problem constraint files were introduced. >>> This will be now extended to this semantic: "install airflow x.y.z with all >>> the providers and dependencies that we found were ok at the time when x.y.z >>> were released". >>> >>> Then, the users will still be free to do `pip install --upgrade >>> apache-airflow-providers-google' and specific upgrade airflow provider to >>> the latest version. Or if they are adventurous they could upgrade all >>> dependencies to latest with 'pip install apache-airflow[google] --upgrade >>> --upgrade-strategy eager' (but without guarantee it will work). >>> >>> Or if there is a new airflow released they could run: 'pip >>> apache-airflow[google, amazon]==2.0.2 --constraints >>> ...../2.0.2/python3.6.txt` - and they will get set of dependencies and >>> providers that were there at the time of 2.0.2 release (but still they are >>> free to upgrade to latest versions of providers at will). >>> >>> Kaxil - did I understand it correctly ? If so - I think this is the best >>> we can do to keep two properties: >>> >>> * repeatable installation of already released version >>> * capability (and easy way of) upgrading to latest providers >>> >>> >>> J. >>> >>> >>> >>> On Thu, Feb 11, 2021 at 1:06 PM Ash Berlin-Taylor <[email protected]> >>> wrote: >>> >>>> Oh I misunderstood -- I thought you were suggesting putting the >>>> transitive deps of apache-airflow-providers-google v2.0 in to >>>> constraint-2.0 files etc. >>>> >>>> Cool >>>> >>>> On Thu, 11 Feb, 2021 at 12:11, Jarek Potiuk <[email protected]> wrote: >>>> >>>> > This unfortunately means that people would be unable to install v1 of >>>> the google provider anymore -- forcing them to upgrade. >>>> >>>> Not really. If we specify just this: [google] -> >>>> "apache-airflow-providers-google" - any provider version could be >>>> installed. >>>> >>>> >>>> On Thu, Feb 11, 2021 at 12:07 PM Ash Berlin-Taylor <[email protected]> >>>> wrote: >>>> >>>>> On Thu, 11 Feb, 2021 at 00:34, Jarek Potiuk <[email protected]> wrote: >>>>> >>>>> *Solution proposal:* >>>>> >>>>> Every time when we release a new wave of providers, we regenerate the >>>>> constraints for all past released 2.* versions of airflow, so that the new >>>>> providers are taken into account and they can install cleanly with `pip >>>>> install apache-airflow[provider]==2.0.N --constraint == .... 2.0.N/python >>>>> ... >>>>> >>>>> Both problems can be solved rather easily. 1) requires 2.0.2 release >>>>> of Airflow, 2) can be implemented any time (happy to do it). >>>>> >>>>> Let me know what you think. >>>>> >>>>> >>>>> This unfortunately means that people would be unable to install v1 of >>>>> the google provider anymore -- forcing them to upgrade. >>>>> >>>>> I'm not sure there's a _ready_ solution to this though. >>>>> >>>>> -ash >>>>> >>>> >>>> >>>> -- >>>> +48 660 796 129 >>>> >>>> >>> >>> -- >>> +48 660 796 129 >>> >> -- +48 660 796 129
