Re: [Distutils] Making pip and PyPI work with conda packages
On Wed, May 20, 2015 at 5:43 PM, Nick Coghlan ncogh...@gmail.com wrote: One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes. That would be great, yes! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 20 May 2015 at 15:38, David Cournapeau courn...@gmail.com wrote:
On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan ncogh...@gmail.com wrote:
On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote:
On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com
wrote:
Indirection via pip injects the usage of setuptools even for plain
distutils projects, and generates
https://www.python.org/dev/peps/pep-0376/
compliant metadata by default.
Note that some packages will push hard against injecting setuptools, at
least until it does not offer a way to prevent from installing as an egg
directory. Most of the core scientific packages avoid setuptools because
of
this.
pip changes the default behaviour of setuptools to be more in line
with the behaviour of plain distutils (e.g. forcing
--single-version-externally-managed).
Yes, but that cannot be the only way to install the package.
This is why I would like to see a way forward for
https://bitbucket.org/pypa/setuptools/issue/371/setuptools-and-state-of-pep-376,
to start pushing packages using setuptools in their setup.py in the
scientific stack. Without this, it will be hard to move forward politically
speaking. Lots of key contributors have a strong aversion to setuptools way
of doing things (partly historical reasons, but it is hard to change minds).
As described in my last comment on that issue, I think the key is to
offer a patch that generates PEP 376 metadata when
--single-version-externally-managed is passed to setuptools.
The PEP 376 metadata layout doesn't cover what should happen when
using the parallel installation support in setuptools, so it doesn't
make sense to try to abide by it in those cases.
If every system (pip, setup.py install, conda, enstaller) were to at least
write the {dist/egg}-info directories correctly, it would be already a
significant step toward interoperation.
Agreed.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
The conda package specification is published at
http://conda.pydata.org/docs/spec.html
The file format is nice and simple. A conda package is a bzipped tar
archive (.tar.bz2) which contains metadata under the info/ directory,
and a collection of files which are installed directly into an install
prefix. The format is identical across platforms and operating
systems. It is important to note that during the install process, all
files are basically just extracted into the install prefix, with the
exception of the ones in info/.
(Compare to the Debian package format's embedded metadata and content archives.)
It has concise metadata in info/index.json
{
arch: x86_64,
build: py27_138_g4f40f08,
build_number: 138,
depends: [
jinja2,
jsonpointer,
jsonschema,
mistune,
pandoc,
pygments,
python 2.7*,
pyzmq,
terminado,
tornado
],
license: MIT License,
name: ipython-we,
platform: linux,
subdir: linux-64,
version: 3.1.0
}
The package includes its build recipe in info/recipe
This particular package has setuptools metadata in
lib/python2.7/site-packages/ipython-3.1.0-py2.7.egg-info
On the index, conda packages are organized by placing packages for a
platform+architecture in their own (sub)directory, not by putting all
that information in the filename. According to the docs it doesn't
interpret the platform metadata.
When conda installs a package, it gets unpacked into a common
directory and then linked into each environment, so that it can be
installed to lots of environments without taking up much extra disk
space. Packages can have link and unlink scripts to provide custom
behavior (perhaps fixing up some paths for files that can't just be
linked) when this happens.
It occurs to me that the setuptools packaging in general is more like
a shared library format .so or .dll, aka libraries searched for along
a path, than an OS level package manager.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 20 May 2015 at 15:53, David Mertz dme...@continuum.io wrote: It's not *only* the 'setup.py install', but it's not *that* much mystery either. wxPython I can't seem to find, not sure what I'm missing. Yeah, I had been under the impression that there was a lot of knowledge on how to build the dependencies (things like libyaml or whatever) in there. Looks like that's not the case... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Wed, May 20, 2015 at 6:30 AM, Daniel Holth dho...@gmail.com wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to. And since a conda package is just a tar archive, you can presumably build them in other ways than a conda build recipe. By the way -- libxml is one example of one without a recipe... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to. The other day, I upgraded the version of conda-recipes/arrow to v0.5.4, and added ofxparse. I should probably create some sort of recurring cron task to show how far behind stable the version number in the meta.yaml is. (see: conda skeleton --version-compare issue/PR (GH:conda/conda-build)) And since a conda package is just a tar archive, you can presumably build them in other ways than a conda build recipe. By the way -- libxml is one example of one without a recipe... Hours of compilation time. * https://www.google.com/#q=inurl:libxml2+conda+meta.yaml (3 results) * https://pypi.python.org/pypi?%3Aaction=searchterm=buildout+libxml * https://pypi.python.org/pypi/z3c.recipe.staticlxml/0.10 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 21 May 2015 at 05:05, Wes Turner wes.tur...@gmail.com wrote: On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to. The other day, I upgraded the version of conda-recipes/arrow to v0.5.4, and added ofxparse. I should probably create some sort of recurring cron task to show how far behind stable the version number in the meta.yaml is. (see: conda skeleton --version-compare issue/PR (GH:conda/conda-build)) https://release-monitoring.org/ is a public service for doing that (more info on supported upstream backends at https://release-monitoring.org/about, more info on the federated messaging protocol used to publish alerts at http://www.fedmsg.com/en/latest/) Anitya (the project powering release-monitoring.org) was built as the monitoring part of Fedora's upstream release notification pipeline: https://fedoraproject.org/wiki/Upstream_release_monitoring One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 20 May 2015 at 23:30, Daniel Holth dho...@gmail.com wrote: It occurs to me that the setuptools packaging in general is more like a shared library format .so or .dll, aka libraries searched for along a path, than an OS level package manager. Yep, that was what PJE was after for Chandler, so that's what he built. It was just useful enough for other folks that it was adopted well beyond that original use case. The key benefit it offered at the time was that pkg_resources could use sys.path + the assumption of directory or zip archive based installation to implement searching for metadata, which avoided the need to come up with an alternative cross-platform approach to metadata storage and retrieval. A related potentially interesting project I've never had time to pursue is an idea for a virtualenv friendly installation layout that doesn't quite lead to the same kind of version proliferation as setuptools did (or as something like NixOS does), while remaining compatible with sys.path based metadata discovery. The essential concept would be to assume semantic versioning for shared installations, and install the shared packages into directories named as package/MAJOR_VERSION/what-would-otherwise-go-into-site-packages. In each virtualenv that opts in to using the shared package rather than its own bundled copy, you'd then install a *.pth file that added package/MAJOR_VERSION to sys.path in that environment. This would be similar in principle to the way Nix user profiles work (http://nixos.org/releases/nix/nix-0.12/manual/#sec-profiles), but adapted to be compatible with Python's existing *.pth file mechanism. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 20, 2015 7:43 PM, Nick Coghlan ncogh...@gmail.com wrote: On 21 May 2015 at 05:05, Wes Turner wes.tur...@gmail.com wrote: On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to. The other day, I upgraded the version of conda-recipes/arrow to v0.5.4, and added ofxparse. I should probably create some sort of recurring cron task to show how far behind stable the version number in the meta.yaml is. (see: conda skeleton --version-compare issue/PR (GH:conda/conda-build)) https://release-monitoring.org/ is a public service for doing that (more info on supported upstream backends at https://release-monitoring.org/about, more info on the federated messaging protocol used to publish alerts at http://www.fedmsg.com/en/latest/) Anitya (the project powering release-monitoring.org) was built as the monitoring part of Fedora's upstream release notification pipeline: https://fedoraproject.org/wiki/Upstream_release_monitoring Thanks! One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes. http://conda.pydata.org/docs/bdist_conda.html bdist_conda? Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 21 May 2015 at 10:52, Wes Turner wes.tur...@gmail.com wrote: On May 20, 2015 7:43 PM, Nick Coghlan ncogh...@gmail.com wrote: One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes. http://conda.pydata.org/docs/bdist_conda.html bdist_conda? conda has the benefit of *not* renaming Python packages in convoluted ways that interfere with automated identification of dependencies :) Both conda and Linux distros run into the it's difficult/impossible to describe external binary dependencies in a cross-platform way problem, though. While https://www.biicode.com/ is interesting in the context of CMake based projects, that still excludes a lot of software. (RYPPL is another I'd heard of, but it's GitHub repo hasn't seen much activity since 2013, and ryppl.org appears to be entirely dead) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 4:11 PM, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: conda -- a fully open source package management system Anaconda -- a python and other stuff distribution produced by Continuum. How Continuum does or doesn't publish the recipes it used to build Anaconda doesn't really have anything to do with conda-the-technology. True. Also though, in answer to the a question here, I asked a Continuum colleague on the conda team. It seems that Anaconda was built using a proprietary system before conda-build and conda-recipes was opened, so not all recipes have made it over to the Free side of the fence yet. But y'know, gh:conda-build *is* a public repository, anyone could add more. I will note that most recipes seem to consist of either 'python setup.py install' or './configure; make; make install'. sure -- but those aren't the ones we want ;-) Understood. see if you can find the wxPython one, while you are at it :-) -- though I suspect that was built from the official executable, rather than re-built from scratch. In the case of pyyaml, this is actually what's behind the wall #!/bin/bash patch -p0 EOF --- setup.cfg~ 2011-05-29 22:31:18.0 -0500 +++ setup.cfg 2012-07-10 20:33:50.0 -0500 @@ -4,10 +4,10 @@ [build_ext] # List of directories to search for 'yaml.h' (separated by ':'). -#include_dirs=/usr/local/include:../../include +include_dirs=$PREFIX/include # List of directories to search for 'libyaml.a' (separated by ':'). -#library_dirs=/usr/local/lib:../../lib +library_dirs=$PREFIX/lib # An alternative compiler to build the extention. #compiler=mingw32 EOF $PYTHON setup.py install It's not *only* the 'setup.py install', but it's not *that* much mystery either. wxPython I can't seem to find, not sure what I'm missing. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 00:25, Chris Barker chris.bar...@noaa.gov wrote: Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as long as you, or someone else has made a conda package for this_c_lib, then all is well. But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. Thanks for the explanation. So, in effect, conda-as-a-platform defines a (somewhat) incompatible platform for running Python, which can use wheels just as python.org Python can, but which uses conda-as-an-installer as its package manager (much like RPM or apt on Unix). The downside of this is that wheels built for conda (assuming that it's OK to link with shared libs) are not compatible with python.org builds (as those shared libs aren't available) and that difference isn't reflected in the wheel ABI tags (and it's not particularly clearly understood by the community, it seems). So publishing conda-based wheels on PyPI would be a bad idea, because they wouldn't work with python.org python (more precisely, only things that depend on shared libs are affected, but the point remains). We need to remember what leveraging conda would buy us: conda doesn't actually make it any easier to build anything -- you need a platform-specific build script to build a conda package. conda does provide a way to manage non-python dependencies -- but that doesn't buy you anything unless you are using conda to manage your system anyway. conda DOES provide a community of people figuring out how to build complex packages, and building them, and putting them up for public dissemination. So the thing that leveraging conda can do is reduce the need for a lot of duplicated effort. And that effort is almost entirely about those third part libs -- after all, a compiled extension that has no dependencies is easy to build and put on PyPi. (OK, there is still a bit of duplicated effort in making the builds themselves on multiple platforms -- but with CI systems, that's not huge) Agreed - that benefit would be significant (and not just in terms of Python packaging - I'd personally find it useful in a lot of non-Python projects), but it does rely on the information/scripts being accessible in non-conda contexts. For example, for python.org wheels, some way of modifying the build to create static libraries rather than shared ones. I'm not sure to what extent the conda folks would want to deal with handling the issues outside their own environment, though (after all, working within a closed environment is part of what makes the problem tractable for them). The whole area of collecting build processes for common libraries is an area that's always been badly managed on Windows, probably because no-one has ever looked at it in a wider context than their own project, and also because every library has its own building on Windows solution (configure, custom MSVC solution files, CMake, etc). And also because C/C++ has no concept anything like PyPI. But while this is hugely interesting to me, it's not clear how well it fits with Python packaging and distutils-sig (except as an unresolved C/C++ issue that we have to contend with). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 10:55, Paul Moore p.f.mo...@gmail.com wrote: But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. Thanks for the explanation. So, in effect, conda-as-a-platform defines a (somewhat) incompatible platform for running Python, which can use wheels just as python.org Python can, but which uses conda-as-an-installer as its package manager (much like RPM or apt on Unix). The downside of this is that wheels built for conda (assuming that it's OK to link with shared libs) are not compatible with python.org builds (as those shared libs aren't available) and that difference isn't reflected in the wheel ABI tags (and it's not particularly clearly understood by the community, it seems). So publishing conda-based wheels on PyPI would be a bad idea, because they wouldn't work with python.org python (more precisely, only things that depend on shared libs are affected, but the point remains). I've been peripherally following this thread so I may be missing the point but it seems to me that Python already has a mature and flexible way of locating and loading shared libs through the module/import system. Surely the best way to manage non-Python shared libs is by exposing them as extension modules which can be packaged up on PyPI. Then you have dependency resolution for pip, you don't need to worry about the OS-specific shared library loading details and ABI information can be stored as metadata in the module. It would even be possible to load multiple versions or ABIs of the same library as differently named Python modules IIUC. As a case in point numpy packages up a load of C code and wraps a BLAS/Lapack library. Many other extension modules are written which can all take advantage of the non-Python shared libraries that embody numpy via its C API. Is there some reason that this is not considered a good solution? -- Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote: On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this. pip changes the default behaviour of setuptools to be more in line with the behaviour of plain distutils (e.g. forcing --single-version-externally-managed). This means that pip install X consistently gives setuptools levels of capabilities in terms of dependency management, without defaulting to installing things as egg archives or directories (modulo the rough edges around setup-requires). As Donald notes, pip install also abstracts away any future invocation of a metadata based pluggable build system, while ./setup.py install assumes the distutils-based status quo will remain in place forever. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan ncogh...@gmail.com wrote:
On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote:
On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com
wrote:
Indirection via pip injects the usage of setuptools even for plain
distutils projects, and generates
https://www.python.org/dev/peps/pep-0376/
compliant metadata by default.
Note that some packages will push hard against injecting setuptools, at
least until it does not offer a way to prevent from installing as an egg
directory. Most of the core scientific packages avoid setuptools because
of
this.
pip changes the default behaviour of setuptools to be more in line
with the behaviour of plain distutils (e.g. forcing
--single-version-externally-managed).
Yes, but that cannot be the only way to install the package.
This is why I would like to see a way forward for
https://bitbucket.org/pypa/setuptools/issue/371/setuptools-and-state-of-pep-376,
to start pushing packages using setuptools in their setup.py in the
scientific stack. Without this, it will be hard to move forward politically
speaking. Lots of key contributors have a strong aversion to setuptools way
of doing things (partly historical reasons, but it is hard to change minds).
If every system (pip, setup.py install, conda, enstaller) were to at
least write the {dist/egg}-info directories correctly, it would be already
a significant step toward interoperation.
David
This means that pip install X consistently gives setuptools levels
of capabilities in terms of dependency management, without defaulting
to installing things as egg archives or directories (modulo the rough
edges around setup-requires).
As Donald notes, pip install also abstracts away any future
invocation of a metadata based pluggable build system, while
./setup.py install assumes the distutils-based status quo will
remain in place forever.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 19, 2015 4:55 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 00:25, Chris Barker chris.bar...@noaa.gov wrote: Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as long as you, or someone else has made a conda package for this_c_lib, then all is well. But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. Thanks for the explanation. So, in effect, conda-as-a-platform defines a (somewhat) incompatible platform for running Python, which can use wheels just as python.org Python can, but which uses conda-as-an-installer as its package manager (much like RPM or apt on Unix). A repeatable, reproducible environment (GH:conda/conda-env) that does not require a C compiler to be installed for deployment. The downside of this is that wheels built for conda (assuming that it's OK to link with shared libs) are not compatible with python.org builds (as those shared libs aren't available) and that difference isn't reflected in the wheel ABI tags (and it's not particularly clearly understood by the community, it seems). So publishing conda-based wheels on PyPI would be a bad idea, because they wouldn't work with python.org python (more precisely, only things that depend on shared libs are affected, but the point remains). system-site-packages may very well be for a different version of the python interpreter and stdlib. We need to remember what leveraging conda would buy us: conda doesn't actually make it any easier to build anything -- you need a platform-specific build script to build a conda package. meta.yaml (w/ optional preprocessing # [selectors]) http://conda.pydata.org/docs/build.html calls build.sh or build.bat by default, at build time. conda does provide a way to manage non-python dependencies -- but that doesn't buy you anything unless you are using conda to manage your system anyway. NodeJS, R, Perl (GH:conda/conda-recipes) conda DOES provide a community of people figuring out how to build complex packages, and building them, and putting them up for public dissemination. So the thing that leveraging conda can do is reduce the need for a lot of duplicated effort. And that effort is almost entirely about those third part libs -- after all, a compiled extension that has no dependencies is easy to build and put on PyPi. (OK, there is still a bit of duplicated effort in making the builds themselves on multiple platforms -- but with CI systems, that's not huge) PPAs would be great Agreed - that benefit would be significant (and not just in terms of Python packaging - I'd personally find it useful in a lot of non-Python projects), but it does rely on the information/scripts being accessible in non-conda contexts. For example, for python.org wheels, some way of modifying the build to create static libraries rather than shared ones. I'm not sure to what extent the conda folks would want to deal with handling the issues outside their own environment, though (after all, working within a closed environment is part of what makes the problem tractable for them). The whole area of collecting build processes for common libraries is an area that's always been badly managed on Windows, probably because no-one has ever looked at it in a wider context than their own project, and also because every library has its own building on Windows solution (configure, custom MSVC solution files, CMake, etc). And also because C/C++ has no concept anything like PyPI. But while this is hugely interesting to me, it's not clear how well it fits with Python packaging and distutils-sig (except as an unresolved C/C++ issue that we have to contend with). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
Maybe I wasn't very clear -- I was addressing what conda might provide in the context of using conda packages with pip/pipy. A conda environment provides a great deal more, yes. system-site-packages may very well be for a different version of the python interpreter and stdlib. Isn't that handled by the wheel meta-data? At least in practice -- Anaconda is delivering python.org compatible pythons. meta.yaml (w/ optional preprocessing # [selectors]) http://conda.pydata.org/docs/build.html calls build.sh or build.bat by default, at build time. Exactly -- you need to write those build scripts yourself. Conda does set up the environment for you (lib paths,etc) so it's often as easy as ./configure make make install, but if it's not (particularly on windows!) you've got to figure it out. Which brings in the community aspect -- a lot of stuff has been figured out. PPAs would be great Personal package archives? Can't we do that now with pip+wheel? Indeed, isn't that 1/2 of what binary wheels are for? And the only thing they are for on Linux? -Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 4:27 AM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: Surely the best way to manage non-Python shared libs is by exposing them as extension modules which can be packaged up on PyPI. Then you have dependency resolution for pip, you don't need to worry about the OS-specific shared library loading details and ABI information can be stored as metadata in the module. It would even be possible to load multiple versions or ABIs of the same library as differently named Python modules IIUC. yes, that's what I proposed earlier in this thread. I put proposed in quotes because it was buried in the discussion, and not the least bit fleshed out, but that was the point. As a case in point numpy packages up a load of C code and wraps a BLAS/Lapack library. Many other extension modules are written which can all take advantage of the non-Python shared libraries that embody numpy via its C API. Though to be fair -- managing that has been a challenge -- numpy may be built with different versions of BLAS, and there is no way to really know what you might be getting... but I think this is solved with the curated packages approach. Is there some reason that this is not considered a good solution? Well, I've had some issues with getting the linking worked out right so that modules could use each-other's libraries, at least on Windows -- but that can probably be worked out. The missing piece in terms of the PyPA infrastructure is that that is no way to specify a binary dependency in the meta data: we can specify that this python package depends on some other python package, but not that this particular binary wheel depends on some other binary wheel. For example, the same python package (say something like PIL) might use the system libs when built on Linux, some of the system libs when built for OS-X, and need all the third party libs when built for Windows. So the OS-X wheel has different dependencies than the Windows wheel, which has different dependencies than, say, a conda package of the same lib. Then there are wheels that might be built to use the homebrew libs on OS-X -- it gets messy! But that can be hacked around, so that we could give it a try and see how it works. The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 9:15 AM, David Cournapeau courn...@gmail.com wrote: Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. well - it does get us pretty far Because some libraries simply don't work as static libraries, or are too big (MKL comes to mind). Also, we have been historically using static libs for our eggs at Enthought on windows, and it has been a nightmare to support. It just does not scale when you have 100s of packages. there is also the issue of semi-developers -- I want people to be able to easily build my package, that depends on a bunch of libs that I really want to be the same. I suppose I could deliver the static libs themselves, along with the headers, etc, but that does get ugly. But really, once wheels support arbitrary file locations, this becomes fairly easy at the packaging level: the remaining issue is one of ABI / standards, but that's mostly a non technical issue. yup -- social issues are the big one. Gholke has 100s of packages using wheels, doesn't he ship the dlls with the packages, even when not using static libs? so that multiple packages may have the same dll? which is almost the same as a static lib. and we ourselves at Enthought have close to 500 packages for windows, all packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++ programs, etc... It is more about agreeing about a common way of doing things than a real technical limitation. good to know -- I suspected as much but haven't tried it yet. If someone were to create and publish a Python compatible static .lib file for the various hard-to-build dependencies, extensions could specify that you link with it in setup.py, and all the person building the wheel has to do is download the needed libraries for the build. OK, but from a social perspective, this is unlikely to happen (it hasn't yet!), without some official support on PyPi, with pip, or ??? So even if static is the way to go -- there is an infrastructure need. If there's a technical reason why dynamic linking at runtime is better than static linking (sufficiently better that it justifies all the effort needed to resolve the issues involved), I'm not sure the issues are that much greater -- we could build binary wheels that hold dynamic libs, and put them up on PyPi -- then other package maintainers would need to actually use those -- almost the same thing as getting a variety of package maintainers to use this mythical repository of static libs. (and by the way, gcc makes it remarkably had to force a static build) Another issue I've run into is nested static libs -- you have to change yoru setup.py in special ways for that to work: libnetcdf depends on libhdf5, depends on libcurl and libz and... getting all that statically linked into my extension is tricky. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Wed, May 20, 2015 at 12:46 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together... Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. Because some libraries simply don't work as static libraries, or are too big (MKL comes to mind). Also, we have been historically using static libs for our eggs at Enthought on windows, and it has been a nightmare to support. It just does not scale when you have 100s of packages. But really, once wheels support arbitrary file locations, this becomes fairly easy at the packaging level: the remaining issue is one of ABI / standards, but that's mostly a non technical issue. Gholke has 100s of packages using wheels, and we ourselves at Enthought have close to 500 packages for windows, all packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++ programs, etc... It is more about agreeing about a common way of doing things than a real technical limitation. David If someone were to create and publish a Python compatible static .lib file for the various hard-to-build dependencies, extensions could specify that you link with it in setup.py, and all the person building the wheel has to do is download the needed libraries for the build. If there's a technical reason why dynamic linking at runtime is better than static linking (sufficiently better that it justifies all the effort needed to resolve the issues involved), then I've yet to see a good explanation of it. The only things I've seen are disk space, or maintenance (where this usually means it's easier to release a new DLL with a security fix than get all the various statically linked packages updated - that's a valid point, but given how hard it is to get a working dynamic linking solution in this environment, I have to wonder whether that argument still holds). All of this applies to Windows only, of course - dynamic linking and system management of shared libraries is very much a per-platform issue, and I don't pretend to know the trade-offs on OSX or Linux. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: So, for example the process for building the pyyaml package available via conda is private? well, I haven't been able to find them... I don't know if continuum keeps them private on purpose or, just haven't happened to publish them. That seems like a rather striking downside to conda that I wasn't aware of. We need to be careful here about what is: conda -- a fully open source package management system Anaconda -- a python and other stuff distribution produced by Continuum. How Continuum does or doesn't publish the recipes it used to build Anaconda doesn't really have anything to do with conda-the-technology. On Tue, May 19, 2015 at 3:23 PM, David Mertz dme...@continuum.io wrote: It is certainly not our intention at Continuum to keep build recipes private. I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes. That would be great! I will note that most recipes seem to consist of either 'python setup.py install' or './configure; make; make install'. sure -- but those aren't the ones we want ;-) So there is quite likely actually little significant work that has failed to have been published. But I'm not sure of pyyaml off the top of my head, and how that is built. see if you can find the wxPython one, while you are at it :-) -- though I suspect that was built from the official executable, rather than re-built from scratch. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 20:26, Chris Barker chris.bar...@noaa.gov wrote: This entire conversation is about when the build dependencies are NOT simple :-). And while it may be project specific, commonly used libs are not project specific, and they are where the time and pain are. So some shared infrastructure would be nice. Fair point. And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way for folks to collaborate on this kind of ugly package building effort. Yeah, it's all about getting people interested. I wish the github project model would work (it did for msys2) but just wishing doesn't help much. Let me ask a different question, then. If I wanted to get hold of (say) libraries for libyaml, libxml2, and maybe a few others (libxpm?) is the conda work of any use for me? I don't want to build conda packages, or depend on them, I just want to grab Python-compatible libraries that I can link into my extension wheel build, or into my application that embeds Python. Ideally, I'd want static libs, but I understand that conda doesn't go that route (at the moment, at least). Even if it's just for me to better understand how conda works, is there an easy to follow guide I could read to see how the conda build of libyaml works? (I assume there must *be* a conda build of libyaml, as conda includes pyyaml which uses libyaml...) I suspect the answer is no, conda's not designed to support that use case. Which is fine - but a shame. The github project with build scripts approach could potentially allow *more* users of the builds and libraries, And that, in a nutshell is the problem (on Windows, at least) - the community of people developing builds of common Unix tools and libraries is completely fragmented - msys2, conda, mingw, ... Anyway, I feel like we're now going round in circles. It's a hard (social) issue, and I don't feel like I have any answers, really. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
It is certainly not our intention at Continuum to keep build recipes private. I have just come on board at the company, but I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes. On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 22:29, Chris Barker chris.bar...@noaa.gov wrote: As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. So, for example the process for building the pyyaml package available via conda is private? (I want to say proprietary, but there's a lot of implications in that term that I don't intend...) That seems like a rather striking downside to conda that I wasn't aware of. Hopefully, I'm misunderstanding something :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 1:27 PM, Paul Moore p.f.mo...@gmail.com wrote: And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way for folks to collaborate on this kind of ugly package building effort. Yeah, it's all about getting people interested. I wish the github project model would work (it did for msys2) but just wishing doesn't help much. well, to be fair, critical mass is key -- and I did not get far in generating that critical mass myself. If a couple dedicated people were to start such a project, and get it far enough that is was really easy for new folks to jump in and grab what they need -- it might keep rolling. Let me ask a different question, then. If I wanted to get hold of (say) libraries for libyaml, libxml2, and maybe a few others (libxpm?) is the conda work of any use for me? I don't want to build conda packages, or depend on them, I just want to grab Python-compatible libraries that I can link into my extension wheel build, or into my application that embeds Python. Ideally, I'd want static libs, but I understand that conda doesn't go that route (at the moment, at least). Even if it's just for me to better understand how conda works, is there an easy to follow guide I could read to see how the conda build of libyaml works? (I assume there must *be* a conda build of libyaml, as conda includes pyyaml which uses libyaml...) Here's the trick -- as far as I know, a conda binary package itself does not include the recipe used to build it. So the question is: has someone published the conda recipe ( conda uses a build dir with a bunch of stuff in it, yaml, shell scripts, what have you ) for that package? if so, then yes, it's easy to go in a look at it and see how it's done. As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. But they do host: https://github.com/conda/conda-recipes though it's not in the least bit complete or particularly maintained. (I don't see libxml in there). There is also a growning community of folks developing and maintaining conda recipes for all sorts of stuff -- much of it on gitHub: https://github.com/ioos/conda-recipes is one example. (those are all getting built and pushed to binstar too). I suspect the answer is no, conda's not designed to support that use case. Which is fine - but a shame. it would be nice if a conda package got a copy of its build recipe embedded in it. The github project with build scripts approach could potentially allow *more* users of the builds and libraries, And that, in a nutshell is the problem (on Windows, at least) - the community of people developing builds of common Unix tools and libraries is completely fragmented - msys2, conda, mingw, yes, it sure is. Anyway, I feel like we're now going round in circles. It's a hard (social) issue, and I don't feel like I have any answers, really. me neither -- come on -- someone give us the answer! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 22:29, Chris Barker chris.bar...@noaa.gov wrote: As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. So, for example the process for building the pyyaml package available via conda is private? (I want to say proprietary, but there's a lot of implications in that term that I don't intend...) That seems like a rather striking downside to conda that I wasn't aware of. Hopefully, I'm misunderstanding something :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
I will note that most recipes seem to consist of either 'python setup.py install' or './configure; make; make install'. So there is quite likely actually little significant work that has failed to have been published. But I'm not sure of pyyaml off the top of my head, and how that is built. On Tue, May 19, 2015 at 3:23 PM, David Mertz dme...@continuum.io wrote: It is certainly not our intention at Continuum to keep build recipes private. I have just come on board at the company, but I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes. On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 22:29, Chris Barker chris.bar...@noaa.gov wrote: As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. So, for example the process for building the pyyaml package available via conda is private? (I want to say proprietary, but there's a lot of implications in that term that I don't intend...) That seems like a rather striking downside to conda that I wasn't aware of. Hopefully, I'm misunderstanding something :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together... Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. If someone were to create and publish a Python compatible static .lib file for the various hard-to-build dependencies, extensions could specify that you link with it in setup.py, and all the person building the wheel has to do is download the needed libraries for the build. If there's a technical reason why dynamic linking at runtime is better than static linking (sufficiently better that it justifies all the effort needed to resolve the issues involved), then I've yet to see a good explanation of it. The only things I've seen are disk space, or maintenance (where this usually means it's easier to release a new DLL with a security fix than get all the various statically linked packages updated - that's a valid point, but given how hard it is to get a working dynamic linking solution in this environment, I have to wonder whether that argument still holds). All of this applies to Windows only, of course - dynamic linking and system management of shared libraries is very much a per-platform issue, and I don't pretend to know the trade-offs on OSX or Linux. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 19 May 2015 at 17:15, David Cournapeau courn...@gmail.com wrote: On Wed, May 20, 2015 at 12:46 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together... Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. Because some libraries simply don't work as static libraries, or are too big (MKL comes to mind). Also, we have been historically using static libs for our eggs at Enthought on windows, and it has been a nightmare to support. It just does not scale when you have 100s of packages. But really, once wheels support arbitrary file locations, this becomes fairly easy at the packaging level: the remaining issue is one of ABI / standards, but that's mostly a non technical issue. Gholke has 100s of packages using wheels, and we ourselves at Enthought have close to 500 packages for windows, all packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++ programs, etc... It is more about agreeing about a common way of doing things than a real technical limitation. Thanks. I'm not going to try to argue against the voice of experience. If you were saying dynamic libs were impossible with wheels (which has been a definite impression I've got elsewhere in this thread) I might try to argue, but if you're saying that the technical issues are solvable, then that's great. We need to establish the details of the common way of doing things that needs to be agreed to move forward. Maybe an interoperability PEP would be useful here, with someone simply proposing a solution to give the debate a concrete starting point? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Tue, May 19, 2015 at 11:10 AM, Paul Moore p.f.mo...@gmail.com wrote: to be the same. I suppose I could deliver the static libs themselves, along with the headers, etc, but that does get ugly. Hmm, that seems to me to be something of a non-goal. If you publish wheels, 99.999% of people should never need to build your software. I'm not so sure -- with a nice mature package, sure. But with something under active development, having a low barrier to entry to running the latest code is really helpful. I suppose I could use a CI system to push a new version of the binary wheel with every update - maybe that is the way to go. I'm 100% in favour of repeatable, automated builds - absolutely But we don't really need a standard infrastructure for them, you write a setup_build_environment.py script that ships with your project, run it once, and then run python setup.py bdist_wheel. If your build dependencies are simple, setup_build_environment.py could be short (or even non-existent!) or if not it could be many hundreds of lines of code. But it's specific to your project. This entire conversation is about when the build dependencies are NOT simple :-). And while it may be project specific, commonly used libs are not project specific, and they are where the time and pain are. So some shared infrastructure would be nice. And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way for folks to collaborate on this kind of ugly package building effort. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
A member of the conda dev team could answer this better than I, but I've used enough to _think_ I understand the basics: On Mon, May 18, 2015 at 3:30 AM, Paul Moore p.f.mo...@gmail.com wrote: One way forward in terms of building wheels is to use any build process you like to do an isolated build (I think it's --root that distutils uses for this sort of thing) and then use distlib to build a wheel from the resulting directory structure (or do it by hand, it's not much more than a bit of directory rearrangement and zipping things up). That process can be automated any way you like - although ideally via something general, so projects don't have to reinvent the wheel every time. sure -- you can put virtually anything in a conda build script. what conda build does is more or less: * setup an isolated environment with some handy environment variables for things like the python interpreter, etc. * run your build script * package up whatever got built. If processes like conda then used wheels as their input for building packages, the wheels could *also* be published I'm not sure it's any easier to build a wheel, then make a conda package out of it, than to build a conda package, and then make a wheel out of it. Or have your build scrit build a wheel, and then independently build a conda package. In any case, the resulting wheel would depend on an environment like the one set up by conda build -- and that is an environment with all the dependencies installed -- which is where this gets ugly. [remember, making the wheel itself it the easy part] not least, does the way conda uses shared libraries make going via wheels impossible (or at least make the wheels unusable without conda's support for installing non-Python shared libraries)? Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as long as you, or someone else has made a conda package for this_c_lib, then all is well. But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. We need to remember what leveraging conda would buy us: conda doesn't actually make it any easier to build anything -- you need a platform-specific build script to build a conda package. conda does provide a way to manage non-python dependencies -- but that doesn't buy you anything unless you are using conda to manage your system anyway. conda DOES provide a community of people figuring out how to build complex packages, and building them, and putting them up for public dissemination. So the thing that leveraging conda can do is reduce the need for a lot of duplicated effort. And that effort is almost entirely about those third part libs -- after all, a compiled extension that has no dependencies is easy to build and put on PyPi. (OK, there is still a bit of duplicated effort in making the builds themselves on multiple platforms -- but with CI systems, that's not huge) An example: I have a complex package that not depends on all sorts of hard-to-build python packages, but also has its own C++ code that depends on the netcdf4 library. Which in turn, depends on the hdf5 lib, which depends on libcurl, and zlib, and (I think one or two others). Making binary wheels of this requires me to figure out how to build all those deps on at least two platforms (Windows being the nightmare, but OS-X is not trivial, too, if I want it to match the python.org build, and support older OS versions than I am running) Then I could have a nice binary wheel that my users can pip install and away they go. But: 1) They also need the Py_netCDF4 package, which may or may not be easy to find. If not -- they need to go through all that build hell. Then they have a package that is using a bunch of the same shared libs as mine -- and hopefully no version conflicts... 2) my package is under development -- what I really want is for it to be easy for my users to build from source, so they can keep it all up to date from the repo. Now they need to get a development version of all those libs up and running on their machines -- a heavy lift for a lot of people. So now - use Anaconda is a pretty good solution -- it provides all the libs I need, and someone else has figured out how to build them on the platforms I care about. But it would be nice if we could find a way for the standard python toolchain could support this. NOTE: as someone suggested on this list was to provide (outside of PyPi + pip), a set of static libs all built and configured, so that I could say: install these libs from some_other_place, then you can build and run my code --
Re: [Distutils] Making pip and PyPI work with conda packages
On 18 May 2015 at 05:32, David Cournapeau courn...@gmail.com wrote: Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this. One way forward in terms of building wheels is to use any build process you like to do an isolated build (I think it's --root that distutils uses for this sort of thing) and then use distlib to build a wheel from the resulting directory structure (or do it by hand, it's not much more than a bit of directory rearrangement and zipping things up). That process can be automated any way you like - although ideally via something general, so projects don't have to reinvent the wheel every time. If processes like conda then used wheels as their input for building packages, the wheels could *also* be published (either on PyPI or on a 3rd party site such as binstar) to provide support for people not using conda for their package maintenance. There may be technical issues with this process - not least, does the way conda uses shared libraries make going via wheels impossible (or at least make the wheels unusable without conda's support for installing non-Python shared libraries)? But it does remove a lot of the duplication of effort and competition that currently seems to be involved in the conda vs wheel situation. In this scenario, there are two binary formats - wheel and conda (the format). There are a number of packaging tools, notably pip and conda (the tool). Other tools and/or formats can be built on the wheel binary format to address specific communities' needs, much as conda does for the Scientific community. Here, wheel and pip are the bottom layer, the Python specific binary format and package manager. They are appropriate for people with generalised needs, and for communities with no need for a more advanced/specialised tool. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. Rob ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sun, May 17, 2015 at 5:12 PM, Robert Collins robe...@robertcollins.net wrote: But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. +1 -- and rip out setuptools installing of dependencies, while we're at it :-) (OK, I know e can't do that...) But is the up shot that using pip to install won't buy anythign over setuptools right now? (except for packages that aren't already using setuptools...) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this. David However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 17, 2015, at 8:12 PM, Robert Collins robe...@robertcollins.net wrote: On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov mailto:chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. So a benefit of using pip instead of setuptools is that as we move to a pluggable build system pip can act as a unified fronted to multiple build systems, instead of every system having to implement each pluggable build system themselves. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? And would this be the right incantation to put in a build script: pip install --no-deps ./ (if you are in the package's main dir -- next to setup.py) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 at 19:05, Nick Coghlan ncogh...@gmail.com wrote: On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. So you're generating a record file and then copying it into place in the rpm's? I'm not sure if thats an external-stable-thing yet, perhaps it is. The distutils data directory with the METADATA etc is what pkg_resources needs (and is preserve on Debian and derived systems already). I see no reason why we can't make this a part of the contract, but we should at least get it in a PEP first, no? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Making pip and PyPI work with conda packages
I've just started monitoring this SIG to get a sense of the issues and status of things. I've also just started working for Continuum Analytics. Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons). However, we would like for users of other distros still to be able to benefit from our creation of binary packages for many platforms in the conda format. As has been discussed in recent threads on dependency solving, the way conda provides metadata apart from entire packages makes much of that work easier. But even aside from that, there are simply a large number of well-tested packages (not only for Python, it is true, so that's possibly a wrinkle in the task) we have generated in conda format. It is true that right now, a user can in principle type: % pip install conda % conda install some_conda_package But that creates two separate systems for tracking what's installed and what dependencies are resolved; and many users will not want to convert completely to conda after that step. What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package Whether that second command ultimately downloads code from pyip.python.org or from repo.continuum.io is probably less important for a user experience perspective. Continuum is very happy to upload all of our conda packages to PyPI if this would improve this user experience. Obviously, the idea here would be that the user would be able to type 'pip list' and friends afterward, and have knowledge of what was installed, even as conda packages. I'm hoping members of the SIG can help me understand both the technical and social obstacles that need to be overcome before this can happen. Yours, David... -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 16 May 2015 at 20:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package Whether that second command ultimately downloads code from pyip.python.org or from repo.continuum.io is probably less important for a user experience perspective. Continuum is very happy to upload all of our conda packages to PyPI if this would improve this user experience. Obviously, the idea here would be that the user would be able to type 'pip list' and friends afterward, and have knowledge of what was installed, even as conda packages. I'm hoping members of the SIG can help me understand both the technical and social obstacles that need to be overcome before this can happen. My immediate thought is, what obstacles stand in the way of a conda to wheel conversion utility? With such a utility, a wholesale conversion of conda packages to wheels, along with hosting those wheels somewhere (binstar? PyPI isn't immediately possible as only package owners can upload files), would essentially give this capability. There presumably are issues with this approach (maybe technical, more likely social) but it seems to me that understanding *why* this approach doesn't work would be a good first step towards identifying an actual solution. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sat, May 16, 2015 at 12:04 PM, David Mertz dme...@continuum.io wrote: Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons). Hmm -- this strikes me as very, very , tricky -- and of course, tied in to the other thread I've been spending a bunch of time on... However, we would like for users of other distros still to be able to benefit from our creation of binary packages for many platforms in the conda format. Frankly, if you want your efforts at building binaries to get used outside of Anaconda, then you shoudl be building wheels in the first place. While conda does more than pip + wheel can do -- I suppose you _could_ use wheels for the things it can support.. But on to the technical issues: conda python packages depend on other conda packages, and some of those packages are not python packages at all. The common use case here are non-python dynamic libs -- exactly the use case I've been going on in the other thread about... And conda installs those dynamic libs in a conda environment -- outside of the python environment. So you can't really use a conda package without a conda enviroment, and an installer that understands that environment (I think conda install does some lib path re-naming, yes?), i.e. conda itself. So I think that's kind of a dead end. So what about the idea of a conda-package-to-wheel converter? conda packages an wheels have a bit in common -- IIUC, they are both basically a zip of all the files you need installed. But again the problem is those dependencies on third party dynamic libs. So far that to work -- pip+wheel would have to grow a way to deal with installing, managing and using dynamic libs. See the other thread for the nightmare there... And while I'd love to see this happen, perhaps an easier route would be for conda_build to grow a static flag that will statically link stuff and get to somethign already supported by pip, wheel, and pypi. -Chris It is true that right now, a user can in principle type: % pip install conda % conda install some_conda_package But that creates two separate systems for tracking what's installed and what dependencies are resolved; Indeed -- which is why some folks are working on making it easier to use conda for everythingconverting a wheel to a conda package is probably easier than the other way around.. Funny -- just moments ago I wrote that it didn't seem that anyone other than me was interested in extending pip_wheel to support this kind of thing -- I guess I was wrong! Great to see you and continuum thinking about this. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 16, 2015, at 3:04 PM, David Mertz dme...@continuum.io wrote: I've just started monitoring this SIG to get a sense of the issues and status of things. I've also just started working for Continuum Analytics. Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons). However, we would like for users of other distros still to be able to benefit from our creation of binary packages for many platforms in the conda format. As has been discussed in recent threads on dependency solving, the way conda provides metadata apart from entire packages makes much of that work easier. But even aside from that, there are simply a large number of well-tested packages (not only for Python, it is true, so that's possibly a wrinkle in the task) we have generated in conda format. It is true that right now, a user can in principle type: % pip install conda % conda install some_conda_package But that creates two separate systems for tracking what's installed and what dependencies are resolved; and many users will not want to convert completely to conda after that step. What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package Whether that second command ultimately downloads code from pyip.python.org http://pyip.python.org/ or from repo.continuum.io http://repo.continuum.io/ is probably less important for a user experience perspective. Continuum is very happy to upload all of our conda packages to PyPI if this would improve this user experience. Obviously, the idea here would be that the user would be able to type 'pip list' and friends afterward, and have knowledge of what was installed, even as conda packages. I'm hoping members of the SIG can help me understand both the technical and social obstacles that need to be overcome before this can happen. As Paul mentioned, I’m not sure I see a major benefit to being able to ``pip install`` a conda package that doesn’t come with a lot of footguns, since any conda package either won’t be able to depend on things like Python or random C libraries or we’re going to have to just ignore those dependencies or what have you. I think a far more workable solution is one that translates a conda package to a Wheel. Practically speaking the only real benefit that conda packages has over pip is the one benefit that simply teaching pip to install conda packages won’t provide - Namely that it supports things which aren’t Python packages. However I don’t think it’s likely that we’re going to be able to install R or erlang or whatever into a virtual environment (for instance), but maybe I’m wrong. There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, it just hasn’t gotten them yet because either we have to convince people to publish our new formats (e.g. we can’t go out and create a wheel repo of common packages) or because we haven’t gotten to it yet because dealing with the crushing legacy of PyPI’s ~400k packages is significant slow down factor. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sat, May 16, 2015 at 4:16 PM, Donald Stufft don...@stufft.io wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, Huh? I'm confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-) but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment…. I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages. well, I got a bit distraced by Erlang and R -- i.e. things that have nothing to do with python packages. libxml, on the other hand, is a lib that one might want to use with a python package -- so a bit more apropos here. But my confusion was about: things that conda has that pip is planning on getting -- what are those things? Any of the stuff that conda has that really useful like handling shared libs, pip is NOT getting -- yes? A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library. exactly -- if you're going to install a shared lib, you need somewhere to put it -- and that's what a conda environment provides. Trying not to go around in circles, but python _could_ provide a standard place in which to put shared libs -- and then pip _could_ provide a way to manage them. That would require dealing with that whole binary API problem, so we probably won't do it. I'm not sure what the point of contention is here: I think it would be useful to have a way to manage shared libs solely for python packages to use -- and it would be useful to that way to be part of the standard python ecosytem. Others may not think it would be useful enough to be worth the pain in the neck it would be. And that's what the nifty conda packages continuum (and others) have built could provide -- those shared libs that are built in a compatible way with a python binary. After all, pure python packages are no problem, compiled python packages without any dependencies are little problem. The hard part is those darn third party libs. conda also provides a way to mange all sorts of other stuff that has nothing to do with python, but I'm guessing that's not what continuum would like to contribute to pypi -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 16, 2015, at 8:50 PM, Chris Barker chris.bar...@noaa.gov wrote: On Sat, May 16, 2015 at 4:16 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, Huh? I'm confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-) but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment…. I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages. well, I got a bit distraced by Erlang and R -- i.e. things that have nothing to do with python packages. libxml, on the other hand, is a lib that one might want to use with a python package -- so a bit more apropos here. But my confusion was about: things that conda has that pip is planning on getting -- what are those things? Any of the stuff that conda has that really useful like handling shared libs, pip is NOT getting -- yes? The ability to resolve dependencies with static metadata is the major one that comes to my mind that’s specific to pip. The ability to have better build systems besides distutils/setuptools is a more ecosystem level one but that’s something we’ll get too. As far as shared libs… beyond what’s already possible (sticking a shared lib inside of a python project and having libraries load that .dll explicitly) it’s not currently on the road map and may never be. I hesitate to say never because it’s obviously a problem that needs solved and if the Python ecosystem solves it (specific to shared libraries, not whole runtimes or other languages or what have you) then that would be a useful thing. I think we have lower hanging fruit that we need to deal with before something like that is even possibly to be on the radar though (if we ever put it on the radar). A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library. exactly -- if you're going to install a shared lib, you need somewhere to put it -- and that's what a conda environment provides. Trying not to go around in circles, but python _could_ provide a standard place in which to put shared libs -- and then pip _could_ provide a way to manage them. That would require dealing with that whole binary API problem, so we probably won't do it. I'm not sure what the point of contention is here: I think it would be useful to have a way to manage shared libs solely for python packages to use -- and it would be useful to that way to be part of the standard python ecosytem. Others may not think it would be useful enough to be worth the pain in the neck it would be. And that's what the nifty conda packages continuum (and others) have built could provide -- those shared libs that are built in a compatible way with a python binary. After all, pure python packages are no problem, compiled python packages without any dependencies are little problem. The hard part is those darn third party libs. conda also provides a way to mange all sorts of other stuff that has nothing to do with python, but I'm guessing that's not what continuum would like to contribute to pypi…. I guess I’m confused what the benefit of making pip able to install a conda package would be. If Python adds someplace for shared libs to go then we could just add shared lib support to Wheels, it’s just another file type so that’s not a big deal. The hardest part is dealing with ABI compatibility. However, given the current state of things, what’s the benefit of being able to do ``pip install conda-lxml``? Either it’s going to flat out break or you’re going to have to do ``conda install libxml2`` first, and if you’re doing ``conda install libxml2`` first then why not just do ``conda install lxml``? I view conda the same way I view apt-get, yum, Chocolatey, etc. It provides an environment and you can install a Python package into that environment, but that pip shouldn’t know how to install a .deb or a .rpm or a conda package because those packages rely on specifics to that environment and Python packages can’t. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist
Re: [Distutils] Making pip and PyPI work with conda packages
On May 16, 2015, at 7:09 PM, Chris Barker chris.bar...@noaa.gov wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, Huh? I'm confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-) but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment…. I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages. A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, Huh? I'm confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-) but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
