On 15 April 2015 at 09:35, Joe Gordon <joe.gord...@gmail.com> wrote: > > > On Sun, Apr 12, 2015 at 6:09 PM, Robert Collins <robe...@robertcollins.net> > wrote: >> >> On 13 April 2015 at 12:53, Monty Taylor <mord...@inaugust.com> wrote: >> >> > If we pin the stable branches with hard pins of direct and indirect >> > dependencies, we can have our stable branch artifacts be installable. >> > Thats awesome. IF there is a bugfix release or a security update to a >> > dependent library - someone can propose it. Otherwise, the stable >> > release should not be moving. >> >> Can we do that in stable branches? We've still got the problem of >> bumping dependencies across multiple packages. > > What do you mean bumping dependencies across mulitple packages?
nova depends on oslo.messaging. Both nova and oslo.messaging have $foo in install_requires. A hard pin of $foo==X will then be applied by both nova and oslo.messaging. AIUI we've moved to installing deps from PyPI (which is great), but lets analyze both cases. a) we install oslo.messaging from PyPI. - we go to change our version of foo to X+1 in nova - the gate job that overrides install_requires (by editing requirements.txt from memory, which feeds into install_requires via our reflection logic) will then fail when oslo.messaging is evaluated, because X+1 != X and oslo.messaging requires ==X. b) we install oslo.messaging from git - we go to change our version of foo to X+1 in nova - we edit oslo.messaging's install_requires as well as nova's - the commit succeeds, but oslo.messaging git and nova git are no longer coinstallable, because oslo.messaging's foo==X has not been changed in git > We cannot do this today with 'pip install -r requirements.txt' but we can > with 'pip install -r --no-deps requirements.txt' if requirements includes > all transitive dependencies. And then we have to figure out transitive > dependencies for all projects etc. pip install -r a-requirements-file-from-pip-freeze should always work, because all transitive dependencies are included, and the versions selected are mutually co-installable (assuming the environment that was pip freeze's was safely constructed [there is one bug https://github.com/pypa/pip/issues/2687 I believe that could be triggered as we use multiple pip install runs to install stuff]). The issues we have with our current requirements.txt and the different ways it might be used are the source of us having had to try --no-deps before: we have global requirements broad because it takes time to propogate a change across repos, and we can't bump lower versions when that breaks not-yet-ready repos (because we use global requirements to override in the gate). The issue inside pip that triggers this is: - when a requirement is added, we initially store the constraint for it - then we evaluate it against the indices in use, finding a version that matches the constraints and selecting that. - if a later source of requirements (e.g. the install_requires from the wheel or sdist of something we selected) has the same requirement, and the selected version does not meet the constraints this new requirement expressed, pip errors. What needs to happen is that we need to be able to backtrack the selection state back to where we selected the version that didn't work, and re-evaluate without that one. This is of course theoretically NPC because as we discover each version the conflict might change each time as well... and the need to actually run sdist egg_info to find out the requirements from each version adds even more pain. In principle though, a solid cache layer + a SAT tool should eat it up for breakfast - but we need to rearrange a chunk of internal code to be able to express it this way. OTOH I'm in the middle of doing that because I need much of the same plumbing to be able to do setup_requires in the way the setuptools and pip folk want. So... its coming. ***but*** none of that matters, because the output from freeze - transitive locked versions - will Just Work today. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev