On Thu, Dec 5, 2013 at 10:12 PM, Oscar Benjamin <oscar.j.benja...@gmail.com>wrote:
> On 4 December 2013 20:56, Ralf Gommers <ralf.gomm...@gmail.com> wrote: > > On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal > > <chris.bar...@noaa.gov> wrote: > >> > >> So a lowest common denominator wheel would be very, very, useful. > >> > >> As for what that would be: the superpack is great, but it's been around > a > >> while (long while in computer years) > >> > >> How many non-sse machines are there still out there? How many non-sse2? > > > > Hard to tell. Probably <2%, but that's still too much. Some older Athlon > XPs > > don't have it for example. And what if someone submits performance > > optimizations (there has been a focus on those recently) to numpy that > use > > SSE4 or AVX for example? You don't want to reject those based on the > > limitations of your distribution process. > > > >> And how big is the performance boost anyway? > > > > Large. For a long time we've put a non-SSE installer for numpy on pypi so > > that people would stop complaining that ``easy_install numpy`` didn't > work. > > Then there were regular complaints about dot products being an order of > > magnitude slower than Matlab or R. > > Yes, I wouldn't want that kind of bad PR getting around about > scientific Python "Python is slower than Matlab" etc. > > It seems as if there is a need to extend the pip+wheel+PyPI system > before this can fully work for numpy. I'm sure that the people here > who have been working on all of this would be very interested to know > what kinds of solutions would work best for numpy and related > packages. > > You mentioned in another message that a post-install script seems best > to you. I suspect there is a little reluctance to go this way because > one of the goals of the wheel system is to reduce the situation where > users execute arbitrary code from the internet with admin privileges > e.g. "sudo pip install X" will download and run the setup.py from X > with root privileges. Part of the point about wheels is that they > don't need to be "executed" for installation. I know that post-install > scripts are common in .deb and .rpm packages but I think that the use > case there is slightly different as the files are downloaded from > controlled repositories whereas PyPI has no quality assurance. > I don't think it's avoidable - anything that is transparant to the user will have to execute code. The multiwheel idea of Nick looks good to me. > BTW, how do the distros handle e.g. SSE? > I don't know exactly to be honest. > My understanding is that they > just strip out all the SSE and related non-portable extensions and > ship generic 686 binaries. My experience is with Ubuntu and I know > they're not very good at handling BLAS with numpy and they don't seem > to be able to compile fftpack as well as Cristoph can. > > Perhaps a good near-term plan might be to > 1) Add the bdist_wheel command to numpy - which may actually be almost > automatic with new enough setuptools/pip and wheel installed. > 2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred > from OS version which wheels can currently handle. > 3) Upload wheels for Windows to somewhere other than PyPI e.g. > SourceForge pending a distribution solution that can detect SSE > support on Windows. > That's a reasonable plan. I have an OS X wheel already, which required only a minor change to numpy's setup.py. > I think it would be good to have a go at wheels even if they're not > fully ready for PyPI (just in case some other issue surfaces in the > process). > Agreed. Ralf
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig