There was some really good feedback in the binary dependency thread, but it ended up going off on a few different tangents. Rather than expecting people to read the whole thing, I figured I'd try to come up with a summary of where it has gone so far, and where we might want to take it from here.
Given the diversity of topics, this should arguably be multiple new threads, but that approach has its own problems, since some of these topics are at least somewhat interrelated. In several cases, some of the discussion could possible move to the tracker issues I created :) == Regarding documentation == One of the things I got from the thread is that we don't currently have a clear overview published anywhere of *why* people use binary extensions. The distutils docs and both dated and focus solely on the mechanics, without discussing either use cases, or the benefits and limitations of using them. We also don't have a good location to describe the approach of statically linking or bundling additional libraries into "platlib" to avoid issues with external dependencies and incompatible ABI changes when distributing wheel files. This seems like something that would be suitably covered as an "Advanced Topic" in the packaging user's guide, so I filed an issue with some more specific ideas: https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-section-that-covers-binary == Regarding conda == In terms of providing an answer to the question "Where does conda fit in the scheme of packaging tools?", my conclusion from the thread is that once a couple of security related issues are fixed (think PyPI before the rubygems.org compromise for the current state of conda's security model), and once the Python 3.3 compatibility issue is addressed on Windows, it would be reasonable to recommend it as one of the current options for getting hold of pre-built versions of the scientific Python stack. I think this is important enough to warrant a "NumPy and the Scientific Python stack" section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific == Regarding alternative index servers == Something I believe conda supports is the idea of installer configuration settings that are specific to a particular virtual environment (I can't find specific docs confirming that capability, but I certainly got that impression somewhere along the line). At the moment, it isn't straightforward to tell pip "when in this virtual environment, use these additional settings", but I believe such a feature could prove useful in dealing with some of the thornier binary compatibility problems. In particular, it would be good to be able to lock an environment to a particular custom index server that it will use instead of defaulting to PyPI. I've posted this idea to the pip issue tracker: https://github.com/pypa/pip/issues/1362 == Regarding NumPy, build variants and API/ABI consistency == My current reading of the NumPy situation is that it boils down to needing two additional features: - a richer platform tagging mechanism (which we need for *nix systems anyway) - a way to ensure internal consistency of the installed *builds* in an environment, not just the abstract dependencies I've opened a wheel format definition issue for the first problem: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for I've posted a metadata 2.0 standard extension proposal for the latter: https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-based-consistency-checking (the alternative index server idea is also relevant, since PyPI wouldn't support hosting multiple variants with the same wheel level compatibility tags) == Regarding custom installation directories == This technically came up in the cobblerd thread (regarding installing scripts to /usr/sbin instead of /usr/bin), but I believe it may also be relevant to the problem of shipping external libraries inside wheels, static data files for applications, etc. It's a little underspecified in PEP 427, but the way the wheel format currently handles installation to paths other than purelib and platlib (or to install to both of those as part of the same wheel) is to use the sysconfig scheme names as subdirectories within the wheel's .data directory. This approach is great for making it easy to build well-behaved cross platform wheels that play nice with virtual environments, but allowing a "just put it here" escape clause could potentially be a useful approach for platform specific wheels (especially on *nix systems that use the Filesystem Hierarchy Standard). I've posted this idea to the metadata format issue tracker: https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig