You should run `pip check` to ensure no conflicts. Pip does not do this on its own.
On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Great that this discussion already happened :). Lots of useful things in > it. And yes - it means pinning in requirement.txt - this is how pip-tools > work. > > J. > > Principal Software Engineer > Phone: +48660796129 > > On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, <arthur.wied...@gmail.com> > wrote: > > > Hi Jarek, > > > > I will +1 the discussion Dan is referring to and George's advice. > > > > I just want to double check we are talking about pinning in > > requirements.txt only. > > > > This offers the ability to > > pip install -r requirements.txt > > pip install --no-deps airflow > > For a guaranteed install which works. > > > > Several different requirement files can be provided for specific use > cases, > > like a stable dev one for instance for people wanting to work on > operators > > and non-core functions. > > > > However, I think we should proactively test in CI against unpinned > > dependencies (though it might be a separate case in the matrix) , so that > > we get advance warning if possible that things will break. > > CI downtime is not a bad thing here, it actually caught a problem :) > > > > We should unpin as possible in setup.py to only maintain minimum required > > compatibility. The process of pinning in setup.py is extremely > detrimental > > when you have a large number of python libraries installed with different > > pinned versions. > > > > Best, > > Arthur > > > > On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavy...@twitter.com.invalid > > > > wrote: > > > > > Relevant discussion about this: > > > > > > > > > https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174 > > > > > > On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <jarek.pot...@polidea.com > > > > > wrote: > > > > > > > TL;DR; A change is coming in the way how dependencies/requirements > are > > > > specified for Apache Airflow - they will be fixed rather than > flexible > > > (== > > > > rather than >=). > > > > > > > > This is follow up after Slack discussion we had with Ash and Kaxil - > > > > summarising what we propose we'll do. > > > > > > > > *Problem:* > > > > During last few weeks we experienced quite a few downtimes of > TravisCI > > > > builds (for all PRs/branches including master) as some of the > > transitive > > > > dependencies were automatically upgraded. This because in a number of > > > > dependencies we have >= rather than == dependencies. > > > > > > > > Whenever there is a new release of such dependency, it might cause > > chain > > > > reaction with upgrade of transitive dependencies which might get into > > > > conflict. > > > > > > > > An example was Flask-AppBuilder vs flask-login transitive dependency > > with > > > > click. They started to conflict once AppBuilder has released version > > > > 1.12.0. > > > > > > > > *Diagnosis:* > > > > Transitive dependencies with "flexible" versions (where >= is used > > > instead > > > > of ==) is a reason for "dependency hell". We will sooner or later hit > > > other > > > > cases where not fixed dependencies cause similar problems with other > > > > transitive dependencies. We need to fix-pin them. This causes > problems > > > for > > > > both - released versions (cause they stop to work!) and for > development > > > > (cause they break master builds in TravisCI and prevent people from > > > > installing development environment from the scratch. > > > > > > > > *Solution:* > > > > > > > > - Following the old-but-good post > > > > https://nvie.com/posts/pin-your-packages/ we are going to fix the > > > > pinned > > > > dependencies to specific versions (so basically all dependencies > are > > > > "fixed"). > > > > - We will introduce mechanism to be able to upgrade dependencies > > with > > > > pip-tools (https://github.com/jazzband/pip-tools). We might also > > > take a > > > > look at pipenv: https://pipenv.readthedocs.io/en/latest/ > > > > - People who would like to upgrade some dependencies for their PRs > > > will > > > > still be able to do it - but such upgrades will be in their PR > thus > > > they > > > > will go through TravisCI tests and they will also have to be > > specified > > > > with > > > > pinned fixed versions (==). This should be part of review process > to > > > > make > > > > sure new/changed requirements are pinned. > > > > - In release process there will be a point where an upgrade will > be > > > > attempted for all requirements (using pip-tools) so that we are > not > > > > stuck > > > > with older releases. This will be in controlled PR environment > where > > > > there > > > > will be time to fix all dependencies without impacting others and > > > likely > > > > enough time to "vet" such changes (this can be done for alpha/beta > > > > releases > > > > for example). > > > > - As a side effect dependencies specification will become far > > simpler > > > > and straightforward. > > > > > > > > Happy to hear community comments to the proposal. I am happy to take > a > > > lead > > > > on that, open JIRA issue and implement if this is something community > > is > > > > happy with. > > > > > > > > J. > > > > > > > > -- > > > > > > > > *Jarek Potiuk, Principal Software Engineer* > > > > Mobile: +48 660 796 129 > > > > > > > > > >
