Relevant discussion about this: https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > TL;DR; A change is coming in the way how dependencies/requirements are > specified for Apache Airflow - they will be fixed rather than flexible (== > rather than >=). > > This is follow up after Slack discussion we had with Ash and Kaxil - > summarising what we propose we'll do. > > *Problem:* > During last few weeks we experienced quite a few downtimes of TravisCI > builds (for all PRs/branches including master) as some of the transitive > dependencies were automatically upgraded. This because in a number of > dependencies we have >= rather than == dependencies. > > Whenever there is a new release of such dependency, it might cause chain > reaction with upgrade of transitive dependencies which might get into > conflict. > > An example was Flask-AppBuilder vs flask-login transitive dependency with > click. They started to conflict once AppBuilder has released version > 1.12.0. > > *Diagnosis:* > Transitive dependencies with "flexible" versions (where >= is used instead > of ==) is a reason for "dependency hell". We will sooner or later hit other > cases where not fixed dependencies cause similar problems with other > transitive dependencies. We need to fix-pin them. This causes problems for > both - released versions (cause they stop to work!) and for development > (cause they break master builds in TravisCI and prevent people from > installing development environment from the scratch. > > *Solution:* > > - Following the old-but-good post > https://nvie.com/posts/pin-your-packages/ we are going to fix the > pinned > dependencies to specific versions (so basically all dependencies are > "fixed"). > - We will introduce mechanism to be able to upgrade dependencies with > pip-tools (https://github.com/jazzband/pip-tools). We might also take a > look at pipenv: https://pipenv.readthedocs.io/en/latest/ > - People who would like to upgrade some dependencies for their PRs will > still be able to do it - but such upgrades will be in their PR thus they > will go through TravisCI tests and they will also have to be specified > with > pinned fixed versions (==). This should be part of review process to > make > sure new/changed requirements are pinned. > - In release process there will be a point where an upgrade will be > attempted for all requirements (using pip-tools) so that we are not > stuck > with older releases. This will be in controlled PR environment where > there > will be time to fix all dependencies without impacting others and likely > enough time to "vet" such changes (this can be done for alpha/beta > releases > for example). > - As a side effect dependencies specification will become far simpler > and straightforward. > > Happy to hear community comments to the proposal. I am happy to take a lead > on that, open JIRA issue and implement if this is something community is > happy with. > > J. > > -- > > *Jarek Potiuk, Principal Software Engineer* > Mobile: +48 660 796 129 >
