Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. Rob ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sun, May 17, 2015 at 5:12 PM, Robert Collins robe...@robertcollins.net wrote: But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. +1 -- and rip out setuptools installing of dependencies, while we're at it :-) (OK, I know e can't do that...) But is the up shot that using pip to install won't buy anythign over setuptools right now? (except for packages that aren't already using setuptools...) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
Trying to keep this brief, because the odds of my finding time to do much with this are slim.. I'm not proposing that we drop it -- just that we push pip and wheel a bit farther to broaden the supported user-base. I can't stop you working on something I consider a deep rabbithole, no -- but I do appreciate your assessment of how deep that hole is -- you certainly have a while lot more background with all this than I do -- I could well be being very naive here. but why not just recommend the use of conda, and only pubish sdists on PyPI? conda needs more users and contributors seeking better integration with the PyPA tooling, and minimising the non-productive competition. I essentially where two hats here: 1) I produce software built on top of the scientific python stack, and I want my users to have an easy experience with installing and running my code. For that -- I am going the conda route. Im not there yet, but close to being able to say: a) install Anaconda b) add my binstar channel to your conda environment c) conda install my_package The complication here is that we also have a web front end for our computational code, and it makes heavy use of all sorts of web-oriented packages that are not supported by Anaconda or, for the most part, the conda community (binstar). My solution is to make conda packages myself of those and put them in my binstar channel. The other option is to piip install those packages, but then you get pretty tangled up in dependencies ans conda environments, vs viirtual environments, etc... 2) Hat two is an instructor for the University of Washington Continuing Education Program's Python Certification. In that program, we do very little with the scipy stack, but have an entire course on web development. And the instructor of that class, quite rightly, pushes the standard of practice for web developers: heavy use of virtualenv and pip. Oh, and hat (3) is a long time pythonista, who, among other things, has been working for years to make using python easier to use on the Mac for folks that don't know or care what the unix command line is I guess the key thing here for me is that I don't see pushing conda to budding web developers -- but what if web developers have the need for a bit of the scipy stack? or??? We really don't have a good solution for those folks. The web development folks targeting Linux will generally be in a position to build from source (caching the resulting wheel file, or perhaps an entire container image). again, I'm not concerned about linux -- it's an ABI nightmare, so we really don't want to go there, and its users are generally more sophisticated a little building is not a big deal. It's also worth noting that one of my key intended use cases for metadata extensions is to publish platform specific external dependencies in the upstream project metadata, which would get us one step closer to fully automated repackaging into policy compliant redistributor packages. Honestly, I don't follow this! -- but I'll keep an eye out for it - sounds useful. The existence of tight ABI coupling between components both gives the scientific Python stack a lot of its power, *and* makes it almost as hard to distribute in binary form as native GUI applications. I think harder, actually :-) * No one else seems to think it's worth trying to extend the PyPa ecosystem a bit more to better support dynamic libs. (except _maybe_ Enthought?) I know Donald is keen to see this, and a lot of ideas become more feasible if(/when?) PyPI gets an integrated wheel build farm. At that point, we can use the centre of gravity approach by letting the build farm implicitly determine the standard version of particular external dependencies, even if we can't communicate those versions effectively in the metadata. That's more what I'm thinking, yes. * I still think it can be done with minimal changes, and hacked in to do the proof of concept I'm still not clear on what it is. I've been pointing out how hard it is to do this right in the general case, but I get the impression you're actually more interested in the narrower case of defining a SciPy ABI that encompasses selected third party binary dependencies. I wouldn't say SciPyABI -- that, in a way is already being done -- folks are coordinating the official bianries of at least the core scipy stack -- it's a pain -- no Windows wheels for numpy, for instance (though I think they are close) My interest is actually taking it beyond that -- honestly in my case there are only a handful of libs that I'm aware of that get common use, for instance libfreetype and libpng in wxPython, PIL, matplotlib, etc. If I were only SciPy focused -- conda would be the way to go. That's part ofteh problem I see -- there are split communities, but they DO overlap, I thin ti's a diservice to punt thes issues of to individual sub-communities to address on their own. * But I'm not sure it's something that's going to
Re: [Distutils] Making pip and PyPI work with conda packages
On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this. David However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On May 17, 2015, at 8:12 PM, Robert Collins robe...@robertcollins.net wrote: On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov mailto:chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing. So a benefit of using pip instead of setuptools is that as we move to a pluggable build system pip can act as a unified fronted to multiple build systems, instead of every system having to implement each pluggable build system themselves. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice. The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that. And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together. Continuum has a bunch of smart people, though. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the offical conda packages is the standard). But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already. So what would using pip inside a conda build script buy you that using setuptools does not? And would this be the right incantation to put in a build script: pip install --no-deps ./ (if you are in the package's main dir -- next to setup.py) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
On 17 May 2015 at 04:48, Nick Coghlan ncogh...@gmail.com wrote: A large part of what *defines* a platform is making decisions about the ABI to publish target. Linux distros, nix, conda do that for everything they redistribute. I assume chocolatey does as well I'm picking on this because it seems to be a common misconception about what Chocolatey provides on Windows. As far as I understand, Chocolatey does *not* provide a platform in this sense at all. The installers hosted by Chocolatey are typically nothing more than repackaged upstream installers (or maybe just scripting around downloading and running upstream installers directly), with a nice command line means of discovering and installing them. From the Chocolatey FAQ: What does Chocolatey do? Are you redistributing software? Chocolatey does the same thing that you would do based on the package instructions. This usually means going out and downloading an installer from the official distribution point and then silently installing it on your machine. With most packages this means Chocolatey is not redistributing software because they are going to the same distribution point that you yourself would go get the software if you were performing this process manually. So AIUI, for example, if you install Python with Chocolatey, it just downloads and runs the python.org installer behind the scenes. Also, Chocolatey explicitly doesn't handle libraries - it is an application installer only. So there's no dependency management, or sharing of libraries beyond that which application installers do natively. Disclaimer: I haven't used Chocolatey much, except for some experimenting. This is precisely *because* it doesn't add much beyond application installs, which I'm pretty much happy handling myself. But it does mean that I could have missed some aspects of what it provides. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 17 May 2015 at 19:05, Nick Coghlan ncogh...@gmail.com wrote: On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. However, having conda use pip install in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly. So you're generating a record file and then copying it into place in the rpm's? I'm not sure if thats an external-stable-thing yet, perhaps it is. The distutils data directory with the METADATA etc is what pkg_resources needs (and is preserve on Debian and derived systems already). I see no reason why we can't make this a part of the contract, but we should at least get it in a PEP first, no? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
