On 19 May 2015 at 00:25, Chris Barker <chris.bar...@noaa.gov> wrote: > Pretty much, yes. conda provides a way to package up and manage arbitrary > stuff -- in this case, that would be non-python dependencies -- i.e. shared > libs. > > So you can say that my_python_package depends on this_c_lib, and as long as > you, or someone else has made a conda package for this_c_lib, then all is > well. > > But python, setuptools, pip, wheel, etc. don't have a way to handle that > shared lib as a dependency -- no standard way where to put it, no way to > package it as a wheel, etc. > > So the way to deal with this with wheels is to statically link everything. > But that's not how conda pa cakges are built, so no way to leverage conda > here.
Thanks for the explanation. So, in effect, conda-as-a-platform defines a (somewhat) incompatible platform for running Python, which can use wheels just as python.org Python can, but which uses conda-as-an-installer as its package manager (much like RPM or apt on Unix). The downside of this is that wheels built for conda (assuming that it's OK to link with shared libs) are not compatible with python.org builds (as those shared libs aren't available) and that difference isn't reflected in the wheel ABI tags (and it's not particularly clearly understood by the community, it seems). So publishing conda-based wheels on PyPI would be a bad idea, because they wouldn't work with python.org python (more precisely, only things that depend on shared libs are affected, but the point remains). > We need to remember what leveraging conda would buy us: > > conda doesn't actually make it any easier to build anything -- you need a > platform-specific build script to build a conda package. > > conda does provide a way to manage non-python dependencies -- but that > doesn't buy you anything unless you are using conda to manage your system > anyway. > > conda DOES provide a community of people figuring out how to build complex > packages, and building them, and putting them up for public dissemination. > > So the thing that leveraging conda can do is reduce the need for a lot of > duplicated effort. And that effort is almost entirely about those third part > libs -- after all, a compiled extension that has no dependencies is easy to > build and put on PyPi. (OK, there is still a bit of duplicated effort in > making the builds themselves on multiple platforms -- but with CI systems, > that's not huge) Agreed - that benefit would be significant (and not just in terms of Python packaging - I'd personally find it useful in a lot of non-Python projects), but it does rely on the information/scripts being accessible in non-conda contexts. For example, for python.org wheels, some way of modifying the build to create static libraries rather than shared ones. I'm not sure to what extent the conda folks would want to deal with handling the issues outside their own environment, though (after all, working within a closed environment is part of what makes the problem tractable for them). The whole area of collecting build processes for common libraries is an area that's always been badly managed on Windows, probably because no-one has ever looked at it in a wider context than their own project, and also because every library has its own "building on Windows" solution (configure, custom MSVC solution files, CMake, etc). And also because C/C++ has no concept anything like PyPI. But while this is hugely interesting to me, it's not clear how well it fits with Python packaging and distutils-sig (except as an unresolved C/C++ issue that we have to contend with). Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
