Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
> But it gets messy when you have two systems trying to handle dependencies -- > pip may not realize that conda has already installed something, and vice > versa. So it's really nicer to have one package manager. > > But maybe all you really need to do is teach conda to understand pip > meta-data, and/or make sure that conda write pip-compatible meta data. Forgive me, I'm trying to follow as someone who is working with PyPI but hasn't really used conda or pip. Does a conda environment contain its own site-packages directory, and does pip correctly install packages to that directory? If so, I expect supporting PEP 376 would help with this. It doesn't help either package manager install dependencies from outside their repos, it just means that pip will work if the user installs dependencies from conda first. To be able to install dependencies, either conda needs to know enough about PyPI to find a package's dependencies itself (and at that point, I wonder how much value pip adds compared to 'wheel'), or pip needs to know that it can delegate to conda when run in this way. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Help required for setup.py
Hello I was trying to create my package for distribution. I have a requirement that I need to check if one particular command is available in the system ( this command is not installed through a package - but a bundle is installed to get the command in the system ). I am using Ubuntu 14.04 Thanks in advance Salil ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
On Mon, May 18, 2015 at 10:50 AM, David Mertz wrote: > This pertains more to the other thread I started, but I'm sort of becoming > convinced--especially by Paul Moore's suggestion there--that the better > approach is to grow conda (the tool) rather than shoehorn conda packages > into pip. > I agree -- in some sense conda is pip+more, you couldn't do it without growing pip (see the other thread...) > So it might make sense to actually allow folks to push conda to budding > web developers, if conda allowed installation (and environment management) > of sdist packages on PyPI. So perhaps it would be good if *this* worked: > > % pip install conda > % conda install scientific_stuff > % conda install --sdist django_widget # we know to look on PyPI > so a point / question here: you can, right now, run pip from inside a conda environment python, and for the most part, it works -- certainly for sdists. I'm actually doing that a lot, and so are others. But it gets messy when you have two systems trying to handle dependencies -- pip may not realize that conda has already installed something, and vice versa. So it's really nicer to have one package manager. But maybe all you really need to do is teach conda to understand pip meta-data, and/or make sure that conda write pip-compatible meta data. Then a user could do: conda install some_package and conda would look it all its normal places for some_package, and if it din't find it, it would try running "pip install" under the hood. The user wouldn't know, or have to know, where the package came from (though conda might want to add that to the meta-data for use come upgrade time, etc.) In short --make it easy for conda users to use pip / pypi packages. Note: there has been various threads about this on the Anaconda list lately. The current "plan" is to have a community binstar channel that mirrors as much of pypi as possible. Until we have an automated way to grab pypi packages for conda -- this isn't bad stop gap. Also note that conda can often (but not always) build a conda package from pypi automagically -- someone could potentially run a service that does that. On Mon, May 18, 2015 at 3:17 AM, Paul Moore wrote: > >> Agreed. My personal use case is as a general programmer (mostly >> sysadmin and automation type of work) with some strong interest in >> business data analysis and a side interest in stats. >> >> For that sort of scenario, some of the scipy stack (specifically >> matplotlib and pandas and their dependencies) is really useful. But >> conda is *not* what I'd use for day to day work, so being able to >> install via pip is important to me. >> > What if "conda install" did work for virtually all pypi packages? (one way or the other) -- would you use and recommend Anaconda (or miniconda) then? > It should be noted that installing >> via pip *is* possible - via some of the relevant projects having >> published wheels, and the rest being available via Christoph Gohlke's >> site either as wheels or as wininsts that I can convert. But that's >> not a seamless process, so it's not something I'd be too happy >> explaining to a colleague should I want to share the workload for that >> type of thing. >> > right -- that could be made better right now -- or soon. Gohlke's packages can't be simply put up on PyPi for licensing reasons (he's using the Intel math libs). But some folks are working really hard on getting a numpy wheel that will work virtually everywhere, and still give good performance for numerics. From there, the core SciPy stack should follow (it's already on PyPi for OS-X). Which is a GREAT move in the right direction, but doesn't get us quite to where PyPi can support the more complex packages. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
On Mon, May 18, 2015 at 11:21 AM, Paul Moore wrote: > On 18 May 2015 at 18:50, David Mertz wrote: > > % pip install conda > > % conda install scientific_stuff > > % conda install --sdist django_widget # we know to look on PyPI > > But that doesn't give (Windows, mainly) users a solution for things > that need a C compiler, but aren't provided as conda packages. > Conda provides (or can) a C compiler (some versions of gcc). It was buggy last time I checked, but it's doable. > My honest view is that unless conda is intending to replace pip and > wheel totally, you cannot assume that people will be happy to use > conda alongside pip (or indeed, use any pair of independent packaging > tools together - people typically want one unified solution). And if > the scientific community stops working towards providing wheels for > people without compilers "because you can use conda", there is going > to be a proportion of the Python community that will lose out on some > great tools as a result. Exactly -- this idea that there are two (or more) non-overlapping communities is pretty destructive. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
A member of the conda dev team could answer this better than I, but I've used enough to _think_ I understand the basics: On Mon, May 18, 2015 at 3:30 AM, Paul Moore wrote: > One way forward in terms of building wheels is to use any build > process you like to do an isolated build (I think it's --root that > distutils uses for this sort of thing) and then use distlib to build a > wheel from the resulting directory structure (or do it by hand, it's > not much more than a bit of directory rearrangement and zipping things > up). > > That process can be automated any way you like - although ideally via > something general, so projects don't have to reinvent the wheel every > time. > sure -- you can put virtually anything in a conda build script. what conda build does is more or less: * setup an isolated environment with some handy environment variables for things like the python interpreter, etc. * run your build script * package up whatever got built. If processes like conda then used wheels as their input for building > packages, the wheels could *also* be published I'm not sure it's any easier to build a wheel, then make a conda package out of it, than to build a conda package, and then make a wheel out of it. Or have your build scrit build a wheel, and then independently build a conda package. In any case, the resulting wheel would depend on an environment like the one set up by conda build -- and that is an environment with all the dependencies installed -- which is where this gets ugly. [remember, making the wheel itself it the easy part] > not least, does the > way conda uses shared libraries make going via wheels impossible (or > at least make the wheels unusable without conda's support for > installing non-Python shared libraries)? Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as long as you, or someone else has made a conda package for this_c_lib, then all is well. But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. We need to remember what leveraging conda would buy us: conda doesn't actually make it any easier to build anything -- you need a platform-specific build script to build a conda package. conda does provide a way to manage non-python dependencies -- but that doesn't buy you anything unless you are using conda to manage your system anyway. conda DOES provide a community of people figuring out how to build complex packages, and building them, and putting them up for public dissemination. So the thing that leveraging conda can do is reduce the need for a lot of duplicated effort. And that effort is almost entirely about those third part libs -- after all, a compiled extension that has no dependencies is easy to build and put on PyPi. (OK, there is still a bit of duplicated effort in making the builds themselves on multiple platforms -- but with CI systems, that's not huge) An example: I have a complex package that not depends on all sorts of hard-to-build python packages, but also has its own C++ code that depends on the netcdf4 library. Which in turn, depends on the hdf5 lib, which depends on libcurl, and zlib, and (I think one or two others). Making binary wheels of this requires me to figure out how to build all those deps on at least two platforms (Windows being the nightmare, but OS-X is not trivial, too, if I want it to match the python.org build, and support older OS versions than I am running) Then I could have a nice binary wheel that my users can pip install and away they go. But: 1) They also need the Py_netCDF4 package, which may or may not be easy to find. If not -- they need to go through all that build hell. Then they have a package that is using a bunch of the same shared libs as mine -- and hopefully no version conflicts... 2) my package is under development -- what I really want is for it to be easy for my users to build from source, so they can keep it all up to date from the repo. Now they need to get a development version of all those libs up and running on their machines -- a heavy lift for a lot of people. So now - "use Anaconda" is a pretty good solution -- it provides all the libs I need, and someone else has figured out how to build them on the platforms I care about. But it would be nice if we could find a way for the "standard" python toolchain could support this. NOTE: as someone suggested on this list was to provide (outside of PyPi + pip), a set of static libs all built and configured, so that I could say: "install these libs from some_other_place, then you can build and run my code" -
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
I don't really see any reason conda couldn't support bdist wheels also. But yes, basically the idea is that we'd like users to be able to rely entirely on conda as their packaging (and environment configuration) system if they choose to. It may be impolitic to say so, but I think conda can and should replace pip for a large class of users. That is, it should be possible for users to use pip exactly once (as in the line I show above), and use conda forever thereafter. Since conda does a lot more (programming language independence, environments), perhaps it really does make a lot more sense for conda to be "one package manager to rule them all" much more than trying to make a pip that does so. But y'know, the truth is I'm trying to figure out the best path here. I want to get better interoperability between conda packages and the rest of the Python ecosystem, but there are stakeholders involved both in the distutils community and within Continuum (where I now work). On Mon, May 18, 2015 at 11:21 AM, Paul Moore wrote: > On 18 May 2015 at 18:50, David Mertz wrote: > > % pip install conda > > % conda install scientific_stuff > > % conda install --sdist django_widget # we know to look on PyPI > > But that doesn't give (Windows, mainly) users a solution for things > that need a C compiler, but aren't provided as conda packages. > > My honest view is that unless conda is intending to replace pip and > wheel totally, you cannot assume that people will be happy to use > conda alongside pip (or indeed, use any pair of independent packaging > tools together - people typically want one unified solution). And if > the scientific community stops working towards providing wheels for > people without compilers "because you can use conda", there is going > to be a proportion of the Python community that will lose out on some > great tools as a result. > > Paul > -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
On 18 May 2015 at 18:50, David Mertz wrote: > % pip install conda > % conda install scientific_stuff > % conda install --sdist django_widget # we know to look on PyPI But that doesn't give (Windows, mainly) users a solution for things that need a C compiler, but aren't provided as conda packages. My honest view is that unless conda is intending to replace pip and wheel totally, you cannot assume that people will be happy to use conda alongside pip (or indeed, use any pair of independent packaging tools together - people typically want one unified solution). And if the scientific community stops working towards providing wheels for people without compilers "because you can use conda", there is going to be a proportion of the Python community that will lose out on some great tools as a result. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
This pertains more to the other thread I started, but I'm sort of becoming convinced--especially by Paul Moore's suggestion there--that the better approach is to grow conda (the tool) rather than shoehorn conda packages into pip. Getting pip to recognize the archive format of conda would be easy enough alone, but that really doesn't cover the fact that 'conda ~= pip+virtualenv', and pip alone simply should not try to grow that latter aspect itself. Plus pip is not going to be fully language agnostic, for various reasons, but including the fact that apt-get and yum and homebrew and ports already exist. So it might make sense to actually allow folks to push conda to budding web developers, if conda allowed installation (and environment management) of sdist packages on PyPI. So perhaps it would be good if *this* worked: % pip install conda % conda install scientific_stuff % conda install --sdist django_widget # we know to look on PyPI Maybe that flag is mis-named, or could be omitted altogether. But there's no conceptual reason that conda couldn't build an sdist fetched from PyPI into a platform specific binary matching the current user machine (and do all the metadata dependency and environment stuff the conda tool does). On Mon, May 18, 2015 at 3:17 AM, Paul Moore wrote: > On 17 May 2015 at 23:50, Chris Barker wrote: > > I guess the key thing here for me is that I don't see pushing conda to > > budding web developers -- but what if web developers have the need for a > bit > > of the scipy stack? or??? > > > > We really don't have a good solution for those folks. > > Agreed. My personal use case is as a general programmer (mostly > sysadmin and automation type of work) with some strong interest in > business data analysis and a side interest in stats. > > For that sort of scenario, some of the scipy stack (specifically > matplotlib and pandas and their dependencies) is really useful. But > conda is *not* what I'd use for day to day work, so being able to > install via pip is important to me. It should be noted that installing > via pip *is* possible - via some of the relevant projects having > published wheels, and the rest being available via Christoph Gohlke's > site either as wheels or as wininsts that I can convert. But that's > not a seamless process, so it's not something I'd be too happy > explaining to a colleague should I want to share the workload for that > type of thing. > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making pip and PyPI work with conda packages
On 18 May 2015 at 05:32, David Cournapeau wrote: > Note that some packages will push hard against injecting setuptools, at > least until it does not offer a way to prevent from installing as an egg > directory. Most of the core scientific packages avoid setuptools because of > this. One way forward in terms of building wheels is to use any build process you like to do an isolated build (I think it's --root that distutils uses for this sort of thing) and then use distlib to build a wheel from the resulting directory structure (or do it by hand, it's not much more than a bit of directory rearrangement and zipping things up). That process can be automated any way you like - although ideally via something general, so projects don't have to reinvent the wheel every time. If processes like conda then used wheels as their input for building packages, the wheels could *also* be published (either on PyPI or on a 3rd party site such as binstar) to provide support for people not using conda for their package maintenance. There may be technical issues with this process - not least, does the way conda uses shared libraries make going via wheels impossible (or at least make the wheels unusable without conda's support for installing non-Python shared libraries)? But it does remove a lot of the duplication of effort and competition that currently seems to be involved in the conda vs wheel situation. In this scenario, there are two binary formats - wheel and conda (the format). There are a number of packaging tools, notably pip and conda (the tool). Other tools and/or formats can be built on the wheel binary format to address specific communities' needs, much as conda does for the Scientific community. Here, wheel and pip are the bottom layer, the Python specific binary format and package manager. They are appropriate for people with generalised needs, and for communities with no need for a more advanced/specialised tool. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)
On 17 May 2015 at 23:50, Chris Barker wrote: > I guess the key thing here for me is that I don't see pushing conda to > budding web developers -- but what if web developers have the need for a bit > of the scipy stack? or??? > > We really don't have a good solution for those folks. Agreed. My personal use case is as a general programmer (mostly sysadmin and automation type of work) with some strong interest in business data analysis and a side interest in stats. For that sort of scenario, some of the scipy stack (specifically matplotlib and pandas and their dependencies) is really useful. But conda is *not* what I'd use for day to day work, so being able to install via pip is important to me. It should be noted that installing via pip *is* possible - via some of the relevant projects having published wheels, and the rest being available via Christoph Gohlke's site either as wheels or as wininsts that I can convert. But that's not a seamless process, so it's not something I'd be too happy explaining to a colleague should I want to share the workload for that type of thing. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
