On Tue, May 19, 2015 at 3:04 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
> Yeah, I'm trying to never build anything for myself, just consume > binaries. Having all binaries built by "the conda people" is a > bottleneck. it is -- though the group of "conda people" is growing... > Having pip auto-build wheels once and reuse them (coming > in the next version, yay!) is good enough. Having projects upload > wheels to PyPI is ideal. Building wheels myself from wininsts provided > by others or by doing the awkward work once and hosting them in a > personal index is acceptable. > you can build conda packages from wininsts as well, actually. Haven't tried it myself yet. (I'm hoping form bdist_mpkgs on the Mac, too, though those are getting rare) I've spent too much of my life trying to build other people's C code. > I'd like to stop now :-) > no kidding -- and this whole thread is about how to help more an more people stop... > One of the key points is that when they started building conda -- pip+wheel > > where not mature, and the core folks behind them didn't want to support > what > > was needed (dynamic libs, etc) -- and still don't. > > Well, I'm not sure "don't want to" is accurate these days. "Think the > problem is harder than you're making it sound" may be accurate, as > might "have no resource to spare to do the work, but would like others > to". well, yeah, though I'm still not sure how much support there is. And I do think that no one want to extend pip to be able to install Perl, for instance ;-) >> My biggest worry is that at some point, "if you want numpy/scipy, you > >> should use conda" becomes an explicit benefit of conda, > > > > That is EXACTLY what the explicit benefit of conda is. I think we'll get > > binary wheels for numpy and scipy up on PyPi before too long, but the > rest > > of the more complex stuff is not going to be there. > > I did specifically mean numpy and scipy. People are using those, and > pandas and matplotlib, That would be the core "scipy stack", and you can't, as of today, pip install it on Windows (you can on the Mac), and you can get wheel from the Gohlke repo, or wininst from the scipy site. That is going to change, hopefully soon, and to be fair the technical hurdles have to do with building a good LAPACK without licensing issues, and figuring out if we need to support pre-SSE2 hardware, etc ... not a pip or pypi limitation. But the *intention* is that wheels will be an acceptable > replacement for wininst installers, so giving up before we reach that > goal would be a shame. I don't think we will -- some folks are doing some great work on that -- and it looks to be close. > Again, that's something for Nick to comment on - I don't know how > wheel (it's wheel more than pip in this context, I believe) fits into > RPM/deb building processes. But I do know that's how he's been talking > about things working. I can't see why conda would be any different. > yup -- it probably does make sense to do with conda what is done with rpm. Except that conda already has a bunch of python-aware stuff in it... > $PYTHON setup.py install > > That sounds like conda doesn't separate build from install - is there > a conda equivalent of a "binary distribution" like a wheel? yes, a conda pacakge is totally binary. but when you build one, it does this: 1) create a new, empty conda environment 2) install the build dependencies of the package at hand 3) download or unpack the source code 4) build and install the package (into this special, temporary, conda environment) 5) package up all the stuff that got installed into a conda package I don't know how it does it, but it essentially finds all the files that were added by the install stage, and puts those in the package. Remarkably simple, and I think, kind of elegant. I think it installs, rather than simply building, so that conda itself doesn't need to know anything about what kind of package it is -- what it wants the final package to be is an archive of everything that you want installed, where you want it installed. so actually installing it is the easiest way to do that. for instance, a C library build script might be: ./configure make make install There are > a lot of reasons why pip/wheel is working hard to move to a separation > of build and install steps, and if conda isn't following that, I think wheel and conda are quite similar in that regard, actually. > > it's that simple. And I don't know if this is what it actually does, but > > essentially the conda package is all the stuff that that script added to > the > > environment. So if changing that invocation to use pip would get us some > > better meta data or what have you, then by all means, we should change > that > > standard of practice. > > Well, it sounds like using "setup.py bdist_wheel" and then repackaging > up the contents of the wheel might be easier than playing "spot what > changed" with a setup.py install invocation. But I'm no expert. (If > the response to that is "wheel doesn't handle X" then I'd rather see > someone offering to help *improve* wheel to handle that situation!) > for python packages, maybe but the way they do it now is more generic -- why have a bunch of python package specific code you don't need? > > (but it seems like an odd thing to have to use the package manager to > build > > the package correctly -- shouldn't' that be distutils or setuptools' job? > > Wheels can be built using setuptools (bdist_wheel) or just packaging > up some files manually. All pip does in this context is orchestrate > the running of setup.py bdist_wheel. You lot use "conda" to mean the > format, the package manager, and the distribution channel - give me > some slack if I occasionally use "pip" when I mean "wheel" or > "setuptools" :-) :-) > fair enough. but does making a wheel create some meta-data, etc. that doing a raw install not create? i.e. is there something in a wheel that conda maybe should use that it's not getting? There is a point where "specialised enough to only matter to > Acaconda's target audience" is OK. And Christoph Gohlke's > distributions fill a huge gap (losing that because "you should use > conda" would be a huge issue, IMO). > His repo is fantastic, yes, but it's an awful lot of reliance on one really, really, productive and smart guy! As a note -- he is often the first to find and resolve a lot of Windows build issues on a variety of packages. AND I think some of the wheels on pypi are actually built by him, and re-distributed by the package maintainers. >> lot of the "conda vs pip" discussion is less relevant. At the moment, > >> there are projects like numpy that don't distribute Windows wheels on > >> PyPI, but Christoph Gohlke has most of them available, > > > > yes, but in a form that is not redistributable on PyPi... > > Well, it's not clear to me how insurmountable that problem is - his > page doesn't specifically mention redistribution limitations (I > presume it's license issues?) yes. > Has anyone discussed the possibility of > addressing the issues? If it is licensing (of MKL?) yes. > then could the > ones without license implications be redistributed? probably -- and, as above, some are (at least one I know of was, anyway :-) ) > Could the PSF > assist with getting a redistribution license for the remainder? I've > no idea what the issues are - I'm just asking if there are approaches > no-one has considered. > It's been talked about, and has recently -- the trick, as I understand it, is that while the Intel license allows re-distribution, you are supposed to have soem sort of info for the user about the license restrictions. when you "pip install" something, there is no way for the user to get that license info. Or something like that. Also -- the scipy devs in general really prefer a fully open source solution. But one is coming soon, so we're good. > I'm not sure about "conda-only", but not pip-installable is all too > common. > > My benchmark is everything that used to distribute wininst installers > or binary eggs distributes wheels. we're probably getting pretty close there. wxPython is an exception, but Robin is distributing the new and improved beta version as wheels. I think he's got enough on his plate that he doesn't want to wrestle with the old and crufty build system. I don't know what else there is of the major packages. > I'd *like* to go beyond that, but > we haven't got the resources to help projects that never offered > binaries. Asking (and assisting) people to move from the old binary > standards to the new one should be a reasonable goal, though. > yup. which makes me think -- maybe not that hard to do a wininst to wheel converter for wxPython -- that would be nice. We also need it for the Mac, and that would be harder -- he's got some trickery in placing the libs in that one... > >> > And for even that to work, we need a way for everything installable by > >> > pip > >> > to be installable within that conda environment -- which we could > >> > probably > >> > achieve. > >> > >> See above - without some serious resource on the conda side, that > >> "everything" is unlikely. > > > > conda can provide a full, pretty standard, python environment -- why do > you > > think "everything" is unlikely? > > Because the pip/wheel ecosystem relies on projects providing their own > wheels. Conda relies on the "conda guys" building packages. It's the > same reason that a PyPI build farm is still only a theory :-) Or were > you assuming that package authors would start providing conda > packages? no -- THAT is unlikely. I meant that if we can get the automatic build a conda package from a pypi sdist working, then we're good to go. Or maybe build-a-conda-package-from-a-binary-wheel. After all, if a package can be pip-installed, then it should be able to be pip-installed inside a conda environment, which means it should be able to be done automatically. I've done a bunch of these -- it's mostly boilerplate. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
