(Note: I'm still traveling, so I'm not fully caught up on all the threads, this one just caught my eye)
On 30 October 2015 at 00:05, Donald Stufft <[email protected]> wrote: > On October 29, 2015 at 6:59:00 PM, Robert Collins ([email protected]) > wrote: >> On 30 October 2015 at 11:50, Robert Collins wrote: >> ... >> > However, the big step you're proposing that I think is fundamentally >> > unsound is that of requiring a wheel be built before dependencies can >> > be queried. Wheels are *expensive* to build in enough cases that I >> > really believe we have a choice between being able to fix issue 988 or >> > utilising wheels as the interface to get at install metadata. Remember >> > too that the numpy ABI discussion is going to cause install >> > dependencies to depend on the version of numpy present at build-time, >> > so the resolver is going to have to re-evaluate the dependencies for >> > those distributions when it selects a new numpy version: so its not >> > even a one-time-per-package cost, its a >> > one-time-per-arc-of-the-N!-graph-paths cost. >> >> On further thought there is one mitigating factor here - we'd have to >> be building numpy fully anyway for the cases where numpy is a >> build-dependency for things further up the stack, so even in my most >> paranoid world we will *anyway* have to pay that cost - but we can >> avoid the cost of multiple full rebuilds of C extensions built on top >> of numpy. I think avoiding that cost is still worth it, its just a >> little more into the grey area rather than being absolutely obviously >> worth it. >> > > We’d only need to pay that cost on the assumption that we don’t have a Wheel > cached already right? Either in the machine local cache > (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since > opting into the new build system mandates the ability to build wheels, for > these items we can always assume we can build a wheel. > > I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not > dead set on requiring a wheel build during resolution. I did that because we > currently have a bit of a race condition since we use ``setup.py egg_info`` > to query the dependencies and then we run ``setup.py bdist_wheel`` to build > the thing and the dependencies are not guaranteed to be the same between the > invocation of these two commands. If we moved to always building Wheels then > we eliminate the race condition and we make the required interface smaller. I > wouldn’t be opposed to including something like ``setup.py dist-info`` in the > interface if we included an assertion stating that, given no state on the > machine changes (no additional packages installed, not invoking from a > different Python, etc), that the metadata produced by that command must be > equal to the metadata produced in the wheel, and we can put an assertion in > pip to ensure it. I think this is the key. If the *core* build-and-dependency-resolution is defined in terms of: * building and caching wheels (so they get built at most once per venv, and potentially per machine) * inspecting their metadata for dependencies then "dist-info" can be introduced later as an optimisation that *just* generates the wheel metadata, without actually doing the full binary build. For wheels we're downloading, we won't need to build them, and for wheels we're installing, we'll need to build them anyway. That means we can move defining the interface for a dist-info subcommand off the critical path for decoupling the build system, and instead use the existing directory based metadata format defined for wheel files. However, I also think there's one refinement we can make that lets us drop the need for a copy-and-paste "setup.py", *without* needing to define a programmatic build system API: let setup.cfg define a module name to invoke with "python -m <module>" instead of running "setup.py". That way, instead of the static setup.py needing to be copied and pasted into each project, there's just another line in the config file like: [bootstrap] setup_requires = my_build_system setup_cli_module = my_build_system.setup_cli If setup_cli_module isn't specified, then pip will expect to find a setup.py in the project directory. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
