Another suggestion someone (I forget who, sorry) had was that we could maintain a full list of _fully tested and supported versions_ (i.e. the output of `pip freeze`) - that way people _can_ use other versions if they want, but we can at least say "use these versions".
I'm not 100% sure how that would work in practice though, but having it be some list we can update without having to do a release is crucial. -ash > On 24 Jun 2019, at 10:00, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > With the recent Sphinx problem > <https://issues.apache.org/jira/browse/AIRFLOW-4841>- we got back our > old-time enemy. In this case sphinx autoapi has been released yesterday to > 1.1.0 version and it started to caused our master to fail, causing kind of > emergency rush to fix as master (and all PRs based on it) would be broken. > > I think I have a proposal that can address similar problems without pushing > us in emergency mode. > > *Context:* > > I wanted to return back to an old discussion - how we can avoid unrelated > dependencies to cause emergencies on our side where we have to quickly > solve such dependency issues when they break our builds. > > *Change coming soon:* > > The problems will be partially addressed with last stage of AIP-10 ( > https://github.com/apache/airflow/pull/4938 - pending only Kubernetes test > fix). It effectively freezes installed dependencies as cached layer of > docker image for builds which do not touch setup.py - so in case setup.py > does not change, the dependencies will not be updated to latest ones. > > *Possibly even better long-term solution:* > > I think we should address it a bit better. We had a number of discussions > on pinning dependencies (for example here > <https://lists.apache.org/thread.html/9e775d11cce6a3473cbe31908a17d7840072125be2dff020ff59a441@%3Cdev.airflow.apache.org%3E>). > I think the conclusion there was that airflow is both "library" (for DAGs) > - where dependencies should not be pinned and end-product (where the > dependencies should be pinned). So it's a bit catch-22 situation. > > Looking at the problem with Sphinx however It came to me that maybe we can > use hybrid solution. We pin all the libraries (like Sphinx or Flask) that > are used to merely build and test the end product but we do not pin the > libraries (like google-api) which are used in the context of library > (writing the operators and DAGs). > > What do you think? Maybe that will be the best of both worlds ? Then we > would have to classify the dependencies and maybe restructure setup.py > slightly to have an obvious distinction between those two types of > dependencies. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
