TL;DR; A change is coming in the way how dependencies/requirements are specified for Apache Airflow - they will be fixed rather than flexible (== rather than >=).
This is follow up after Slack discussion we had with Ash and Kaxil - summarising what we propose we'll do. *Problem:* During last few weeks we experienced quite a few downtimes of TravisCI builds (for all PRs/branches including master) as some of the transitive dependencies were automatically upgraded. This because in a number of dependencies we have >= rather than == dependencies. Whenever there is a new release of such dependency, it might cause chain reaction with upgrade of transitive dependencies which might get into conflict. An example was Flask-AppBuilder vs flask-login transitive dependency with click. They started to conflict once AppBuilder has released version 1.12.0. *Diagnosis:* Transitive dependencies with "flexible" versions (where >= is used instead of ==) is a reason for "dependency hell". We will sooner or later hit other cases where not fixed dependencies cause similar problems with other transitive dependencies. We need to fix-pin them. This causes problems for both - released versions (cause they stop to work!) and for development (cause they break master builds in TravisCI and prevent people from installing development environment from the scratch. *Solution:* - Following the old-but-good post https://nvie.com/posts/pin-your-packages/ we are going to fix the pinned dependencies to specific versions (so basically all dependencies are "fixed"). - We will introduce mechanism to be able to upgrade dependencies with pip-tools (https://github.com/jazzband/pip-tools). We might also take a look at pipenv: https://pipenv.readthedocs.io/en/latest/ - People who would like to upgrade some dependencies for their PRs will still be able to do it - but such upgrades will be in their PR thus they will go through TravisCI tests and they will also have to be specified with pinned fixed versions (==). This should be part of review process to make sure new/changed requirements are pinned. - In release process there will be a point where an upgrade will be attempted for all requirements (using pip-tools) so that we are not stuck with older releases. This will be in controlled PR environment where there will be time to fix all dependencies without impacting others and likely enough time to "vet" such changes (this can be done for alpha/beta releases for example). - As a side effect dependencies specification will become far simpler and straightforward. Happy to hear community comments to the proposal. I am happy to take a lead on that, open JIRA issue and implement if this is something community is happy with. J. -- *Jarek Potiuk, Principal Software Engineer* Mobile: +48 660 796 129
