Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread David Cournapeau
On Sat, May 16, 2015 at 11:36 PM, Justin Cappos jcap...@nyu.edu wrote: I am no expert, but I don't understand why backtracking algorithms would to be faster than SAT, since they both potentially need to walk over the full set of possible solutions. It is hard to reason about the cost because

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread Daniel Holth
On May 16, 2015 11:22 AM, David Cournapeau courn...@gmail.com wrote: On Sat, May 16, 2015 at 11:36 PM, Justin Cappos jcap...@nyu.edu wrote: I am no expert, but I don't understand why backtracking algorithms would to be faster than SAT, since they both potentially need to walk over the full

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread Nick Coghlan
On 17 May 2015 at 00:36, Justin Cappos jcap...@nyu.edu wrote: This only considers computation cost though. Other factors can become more expensive than computation. For example, SAT solvers need all the rules to consider. So a SAT solution needs to effectively download the full dependency

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread Justin Cappos
I am no expert, but I don't understand why backtracking algorithms would to be faster than SAT, since they both potentially need to walk over the full set of possible solutions. It is hard to reason about the cost because the worst case is in theory growing exponentially in both cases. This

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread David Cournapeau
On Sun, May 17, 2015 at 12:40 AM, Daniel Holth dho...@gmail.com wrote: On May 16, 2015 11:22 AM, David Cournapeau courn...@gmail.com wrote: On Sat, May 16, 2015 at 11:36 PM, Justin Cappos jcap...@nyu.edu wrote: I am no expert, but I don't understand why backtracking algorithms

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread Donald Stufft
On May 16, 2015, at 1:24 PM, Nick Coghlan ncogh...@gmail.com wrote: On 17 May 2015 at 00:36, Justin Cappos jcap...@nyu.edu wrote: This only considers computation cost though. Other factors can become more expensive than computation. For example, SAT solvers need all the rules to

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-16 Thread Nick Coghlan
On 16 May 2015 at 11:52, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 13:45, Donald Stufft don...@stufft.io wrote: On May 15, 2015, at 9:22 PM, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote: Why not start

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Marcus Smith
Why not start with pip at least being a simple fail-on-conflict resolver (vs the 1st found wins resolver it is now)... You'd backtrack for the sake of re-walking when new constraints are found, but not for the purpose of solving conflicts. I know you're motivated to solve Openstack build issues,

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote: Why not start with pip at least being a simple fail-on-conflict resolver (vs the 1st found wins resolver it is now)... You'd backtrack for the sake of re-walking when new constraints are found, but not for the purpose of solving

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Donald Stufft
On May 15, 2015, at 9:22 PM, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote: Why not start with pip at least being a simple fail-on-conflict resolver (vs the 1st found wins resolver it is now)... You'd backtrack for the sake

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 13:45, Donald Stufft don...@stufft.io wrote: On May 15, 2015, at 9:22 PM, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote: Why not start with pip at least being a simple fail-on-conflict resolver (vs the 1st

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread David Cournapeau
On Sat, May 16, 2015 at 10:52 AM, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 13:45, Donald Stufft don...@stufft.io wrote: On May 15, 2015, at 9:22 PM, Robert Collins robe...@robertcollins.net wrote: On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote:

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Noah Kantrowitz
On May 15, 2015, at 9:19 PM, Donald Stufft don...@stufft.io wrote: On May 15, 2015, at 2:57 PM, Robert Collins robe...@robertcollins.net wrote: So, I am working on pip issue 988: pip doesn't resolve packages at all. This is O(packages^alternatives_per_package): if you are resolving 10

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 07:18, Justin Cappos jcap...@nyu.edu wrote: One thing to consider is that if conflicts do not exist (or are very rare), the number of possible combinations is a moot point. A greedy algorithm for installation (which just chooses the most favored package to resolve each

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Jim Fulton
On Fri, May 15, 2015 at 2:57 PM, Robert Collins robe...@robertcollins.net wrote: So, I am working on pip issue 988: pip doesn't resolve packages at all. This is O(packages^alternatives_per_package): if you are resolving 10 packages with 10 versions each, there are approximately 10^10 or 10G

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 08:27, Jim Fulton j...@zope.com wrote: If folk consider PyPI a sort of historical archive then perhaps we could have a feature to select 'supported' versions by the author, and allow a query parameter to ask for all the versions. You could simply limit the number of

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Justin Cappos
Example: say I have an ecosystem of 10 packages. A-J. And they do a release every 6 months that is guaranteed to work together, but every time some issue occurs which ends up clamping the group together- e.g. an external release breaks API and so A1s deps are disjoint with A2s, and then the

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Donald Stufft
On May 15, 2015, at 2:57 PM, Robert Collins robe...@robertcollins.net wrote: So, I am working on pip issue 988: pip doesn't resolve packages at all. This is O(packages^alternatives_per_package): if you are resolving 10 packages with 10 versions each, there are approximately 10^10 or 10G

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Justin Cappos
One thing to consider is that if conflicts do not exist (or are very rare), the number of possible combinations is a moot point. A greedy algorithm for installation (which just chooses the most favored package to resolve each dependency) will run in linear time with the number of packages it

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 06:57, Robert Collins robe...@robertcollins.net wrote: So, I am working on pip issue 988: pip doesn't resolve packages at all. This is O(packages^alternatives_per_package): if you are resolving 10 ... Scaling the number of packages is relatively easy; scaling the number of

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Ben Finney
Donald Stufft don...@stufft.io writes: On May 15, 2015, at 2:57 PM, Robert Collins robe...@robertcollins.net wrote: If folk consider PyPI a sort of historical archive then perhaps we could have a feature to select 'supported' versions by the author, and allow a query parameter to

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 07:19, Donald Stufft don...@stufft.io wrote: There have been a handful of projects which would only keep the latest N versions uploaded to PyPI. I know this primarily because it has caused people a decent amount of pain over time. It’s common for deployments people have to

Re: [Distutils] PyPI is a sick sick hoarder

2015-05-15 Thread Robert Collins
On 16 May 2015 at 08:46, Justin Cappos jcap...@nyu.edu wrote: Example: say I have an ecosystem of 10 packages. A-J. And they do a release every 6 months that is guaranteed to work together, but every time some issue occurs which ends up clamping the group together- e.g. an external release