Re: [DISCUSS] Back to (some) dependency pinning

TL;DR; I think we can close the pinning story (for now at least ): I just merged yesterday the change that introduced requirements per-version. Rather than having single requirements.txt in the main directory, we have now requirements folder and requirements-python3.7.txt and requirements-python3.

Re: [DISCUSS] Back to (some) dependency pinning

Yep. Second half 2020. We need to cleanup some inconsistencies before we move to it :). J. On Sat, Mar 28, 2020 at 8:28 AM Driesprong, Fokko wrote: > The resolving will become more strict in the future: > > http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html > > Che

Re: [DISCUSS] Back to (some) dependency pinning

The resolving will become more strict in the future: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html Cheers, Fokko Op wo 25 mrt. 2020 om 15:40 schreef Jarek Potiuk : > OK. I think that one does not need voting then. I will proceed with my PR > for that :). > > J.

Re: [DISCUSS] Back to (some) dependency pinning

OK. I think that one does not need voting then. I will proceed with my PR for that :). J. On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman wrote: > Agreed. Kind of a “best we can do” considering the current nature of > python. > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko , > wrote: > > Ye

Re: [DISCUSS] Back to (some) dependency pinning

Agreed. Kind of a “best we can do” considering the current nature of python. On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko , wrote: > Yes, I'd be in favor of not having two packages, and just pinning the > versions then. In this case, all the versions will be pinned, so if a user > wants to ins

Re: [DISCUSS] Back to (some) dependency pinning

Yes, I'd be in favor of not having two packages, and just pinning the versions then. In this case, all the versions will be pinned, so if a user wants to install a newer version of elastic, they have to do it explicitly. For Java, you have nice packages that will check if you break any public API,

Re: [DISCUSS] Back to (some) dependency pinning

And yet another update: - after seeing how it works I will remove requirement generation from pre-commit - now that it needs to be generated separately for different versions of python it's a bit too much overhead (you'd need to have more images downloaded for different python versions for pre-comm

Re: [DISCUSS] Back to (some) dependency pinning

Update - It seems that we won't need the -pinned version eventually. I realized that we need to have slightly different requirements for different python versions. I just added PR for that: https://github.com/apache/airflow/pull/7841 I also found out (during production image exercise) that we can

Re: [DISCUSS] Back to (some) dependency pinning

Yesterday we had another master breakage - this time from elasticsearch releasing MINOR version 7.6 breaking our builds (not it was MINOR version so should be compatible it was not for us). I fixed it quickly yesterday by limiting it to < 7.6 but for me - this is quite clear that trying to rel

Re: [DISCUSS] Back to (some) dependency pinning

And rebased it right now and fixed automated requirements update. On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk wrote: > Ah BTW. I just noticed that for some reason I pasted an old PR earlier in > the thread :(. > This is the one with requirements.txt I am talking about: > https://github.com/apac

Re: [DISCUSS] Back to (some) dependency pinning

Ah BTW. I just noticed that for some reason I pasted an old PR earlier in the thread :(. This is the one with requirements.txt I am talking about: https://github.com/apache/airflow/pull/7730 On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk wrote: > Nope. Not blocking. I can work with my branch just

Re: [DISCUSS] Back to (some) dependency pinning

Nope. Not blocking. I can work with my branch just requirements.txt is enough for that :) I think the problem with semver is that it is loosely followed - we had a number of breakages in the past with minor version upgrades :(. J. On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik wrote: > Thanks fo

Re: [DISCUSS] Back to (some) dependency pinning

Thanks for the detailed explanation Jarek. How about we have an upper limit for all our dependencies, example instead of "google-cloud-storage>=1.16", we have "google-cloud-storage>=1.16,<2.0" ? If a dependency breaks compatibility in minor versions, we can't do anything about it but if they foll

Re: [DISCUSS] Back to (some) dependency pinning

I think we have similar understanding. But let me just clarify because I think we think about we think about solving two different problems My proposal is not solving all problems with dependencies - quite the contrary, I want to solve just one specific "repeatability" problem - read on :).. 1.

Re: [DISCUSS] Back to (some) dependency pinning

I think the "-pinned" version might have following 1. A potential source of confusion: using "-pinned" for installation but using "non-pinned" for DAG development. 2. Most of the users would still try to install "apache-airflow" package that might have been broken for example because

Re: [DISCUSS] Back to (some) dependency pinning

Agree Kaxil that requirements.txt is crucial. The requirements.txt is already part of this change https://github.com/apache/airflow/pull/4938 (and it is essential to the whole concept). >From what I see no-one complains about requirements.txt (the nice thing is that we will have it automatically u

Re: [DISCUSS] Back to (some) dependency pinning

I think we can have a requirements.txt (freezed when the package is released, similar to yarn.lock) instead of releasing a separate apache-airflow-pinned package. Regards, Kaxil On Tue, Mar 17, 2020 at 7:38 PM Ash Berlin-Taylor wrote: > I think irrespective of what we do about releasing a pinne

Re: [DISCUSS] Back to (some) dependency pinning

I think irrespective of what we do about releasing a pinned version, using this approach so our prod image is repeatable sounds good! On 17 March 2020 19:17:59 GMT, Jarek Potiuk wrote: >Any other comments? > >I'd love to hear your thoughts. It's the one thing that maybe not keeps >me >from prod

Re: [DISCUSS] Back to (some) dependency pinning

Any other comments? I'd love to hear your thoughts. It's the one thing that maybe not keeps me from prod image, But I would love to know if I can rely on the requirements.txt being part of the source code so that I can use it when building the prod image.. J. On Mon, Mar 16, 2020 at 12:16 PM Ja

Re: [DISCUSS] Back to (some) dependency pinning

On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko wrote: > Personally I don't like to have two versions in the PyPi repo. This also > complicates the releases, since we need to test, release and verify two > versions of Airflow > Not necessarily. Release is automated (and I run both package bui

Re: [DISCUSS] Back to (some) dependency pinning

Personally I don't like to have two versions in the PyPi repo. This also complicates the releases, since we need to test, release and verify two versions of Airflow. I'm afraid that this might confuse users. Besides that, it feels a bit like we're reinventing certain mechanisms that are already in

Re: [DISCUSS] Back to (some) dependency pinning

Does anyone think this one needs voting? I think it's an incremental change (but one that helps having stable installation mechanism on installing any released version of Airflow). J. On Sun, Mar 15, 2020 at 1:17 PM Jarek Potiuk wrote: > I think I have finally found a good and simple solution

Re: [DISCUSS] Back to (some) dependency pinning

I think I have finally found a good and simple solution that I will also be able to use in the production image. It boils down to adding "-apache-airflow-pinned" package additionally to "apache-airflow". This package is the same as apache-airflow but it has pinned dependencies (so it will never fa

Re: [DISCUSS] Back to (some) dependency pinning

Just to update the plan and have some summary for those who did not read my TL;DR mail (and include probot integration). Here is a shorter summary of the proposal I have: 1. Keep the current process of *pip install -e[]* unchanged - taking into account the limits from setup.py 2. Add *req

Re: [DISCUSS] Back to (some) dependency pinning

My 2 cents on the long-term plan: Once Airflow has Dag Isolation (i.e. DAG python dependencies are completely decoupled from Airflow), we should pin the core Airflow deps, and package operators separately with version ranges (for the reasons Ash mentioned about libraries vs applications). On Sat,

Re: [DISCUSS] Back to (some) dependency pinning

After some discussions and some tests by myself and Astronomer docker, I think I have finally a complete proposal for good, consistent solution for our dependency management. It uses well-known standards, does not introduce any new tools (like poetry) and I think it serves all the use cases we hav

Re: [DISCUSS] Back to (some) dependency pinning

I understand what Ash and Jarek are saying. I actually use airflow with custom plugins to have a end-product that fully satisfy our needs and when writing new hooks and operators I don't want to see "airflow uses requirement foo=1 but you have foo=2" - but actually that sometimes also just works

Re: [DISCUSS] Back to (some) dependency pinning

Ash is totally right - that's exactly the difficulty we face. Airflow is both a library and end product and this makes the usual advice (pin if you are end-product, don't pin if you are library) not really useful. From the very beginning of my adventures with Airflow I was for pinning of everything

Re: [DISCUSS] Back to (some) dependency pinning

On Thu, Aug 1, 2019 at 1:33 PM Chen Tong wrote: > It is sometimes hard to distinguish if it is a library or an application. > Take operator as an example, a non-tech people may think it as a well-built > application while an engineer may consider it as a library and extends > functionalities on it

Re: [DISCUSS] Back to (some) dependency pinning

It is sometimes hard to distinguish if it is a library or an application. Take operator as an example, a non-tech people may think it as a well-built application while an engineer may consider it as a library and extends functionalities on it. On Thu, Aug 1, 2019 at 4:26 PM Qingping Hou wrote: >

Re: [DISCUSS] Back to (some) dependency pinning

see replies inline :) On Thu, Aug 1, 2019 at 12:42 PM Ash Berlin-Taylor wrote: > The usual advice (at least for other languages, I don't know about Python) is > that end applications should exactly pin their deps, but that libraries > should be forgiving, so that it is easier to use it alongsid

Re: [DISCUSS] Back to (some) dependency pinning

The problem with pinning everything is that it makes installing Airflow along with other python modules more fraught. The usual advice (at least for other languages, I don't know about Python) is that end applications should exactly pin their deps, but that libraries should be forgiving, so tha

Re: [DISCUSS] Back to (some) dependency pinning

Hey Jarek, sounds good, but actually I would probably go with pinning everything by default and have a "Dependency Bot" testing new releases of packages. But regarding of the big amount of computing (=costs) we already have by running our ci pipeline we cannot set up a Dependency Bot at the mome

Re: [DISCUSS] Back to (some) dependency pinning

Is there any reason why we don't just pin all dependencies to the exact version? I can see the benefit of the current relaxed dependency requirement, which is to avoid having to maintain and do frequent update for frozen dependencies. If we are already going down the route of maintaining a separat

Re: [DISCUSS] Back to (some) dependency pinning

It's better than nothing but I am concerned that this will make managing optional dependencies overly complicated. I really think that the proper fix is patch releases. On Thu, Aug 1, 2019 at 1:05 PM Jarek Potiuk wrote: > Hello Everyone, > > Just to revive the thread - we had a discussion with A

Re: [DISCUSS] Back to (some) dependency pinning

Hello Everyone, Just to revive the thread - we had a discussion with Ash today after today's small "spanner" drama, and we came with a possible solution. This is something we yet have to try but it seems that it should be possible to generate additional "pinned" extras (pinned, gcp_api-pinned etc

Re: [DISCUSS] Back to (some) dependency pinning

Hi Jarek, Thanks for bringing this up. I certainly think this is a good idea. Unfortunately I'm in a plane right now so I'm unable to read the Google doc right now. GitHub recently acquired Dependabot which even supports automatic updates of dependencies. The we at least know when something break

Re: [DISCUSS] Back to (some) dependency pinning

All for deeper release-cycle discussion. I think after 1.10.4 is out we should discuss/agree and document the release scheme we are going to use. Semver and patching seems like a good idea. We have already quite an experience in backporting to 1.10.x branch and it was surprisingly easy - small, fo

Re: [DISCUSS] Back to (some) dependency pinning

I am +1 on pinning core packages, even though this adds a bit of manual labor for maintenance. This latest werkzeug issue highlights why this is a good idea. Also +1 on changing the versioning scheme to something more akin to semver. The current scheme basically does not support patch-only release

Re: [DISCUSS] Back to (some) dependency pinning

I think the recent case with werkzeug calls for action here (also see https://issues.apache.org/jira/browse/AIRFLOW-4903 ). We again ended up with released Airflow version that cannot be installed easily because of some transient dependencies upgrade. I think this is something we should at least c

Re: [DISCUSS] Back to (some) dependency pinning

Another suggestion someone (I forget who, sorry) had was that we could maintain a full list of _fully tested and supported versions_ (i.e. the output of `pip freeze`) - that way people _can_ use other versions if they want, but we can at least say "use these versions". I'm not 100% sure how tha

[DISCUSS] Back to (some) dependency pinning

With the recent Sphinx problem - we got back our old-time enemy. In this case sphinx autoapi has been released yesterday to 1.1.0 version and it started to caused our master to fail, causing kind of emergency rush to fix as master (and all PRs bas