Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
FYI: https://github.com/numpy/numpy/pull/3439 -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Thu, Jun 13, 2013 at 1:25 PM, Andrew Jaffe wrote: > On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote: > > On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers > wrote: > >> The binaries will still be built against python.org Python, so there > >> shouldn't be an issue here. Same for building from source. > > > > My point was that it's nice to be able to have it build with an out of > > teh box wetup.py with accelerated LAPACK and all... If whoever is > > building binaries wants to get fancy, great. > > Yes, please. The current system does seem to work for at least some of > us. And, if I understand the thread in the scipy mailing list, it's not > actually clear that there's a bug, as opposed to incompatible fortran > ABIs (which doesn't seem like a bug to me). > > But I guess the most important thing would be that it can be used with > apple or python.org Python builds (my reading of some of the suggestions > would be requiring one of homebrew/fink/macports), No we should support all pythons; python.org being the one supported by our binary installers. Ralf > preferably out-of-the-box -- even if that meant restricting to prebuilt > binaries. > Being able to run non-obscure installers (i.e., from the main python.org > and scipy.org sites) for Python + numpy + scipy + matplotlib and get > optimized versions would be sufficient. > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
12.06.2013 00:29, Ralf Gommers kirjoitti: [clip] > Sounds like a good idea. Would still make sense to move Accelerate down > in the list of preferred libs, so that one can install ATLAS, MKL or > OpenBLAS once and be done, instead of always having to remember these > envvars. It goes like this: https://github.com/pv/numpy/tree/fortran-abicheck Interested parties on OSX should check whether they manage to build this version of Numpy with Accelerate when this is enabled, and whether Scipy's ARPACK tests still fail when building against this version of Numpy. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote: > On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers wrote: >> The binaries will still be built against python.org Python, so there >> shouldn't be an issue here. Same for building from source. > > My point was that it's nice to be able to have it build with an out of > teh box wetup.py with accelerated LAPACK and all... If whoever is > building binaries wants to get fancy, great. Yes, please. The current system does seem to work for at least some of us. And, if I understand the thread in the scipy mailing list, it's not actually clear that there's a bug, as opposed to incompatible fortran ABIs (which doesn't seem like a bug to me). But I guess the most important thing would be that it can be used with apple or python.org Python builds (my reading of some of the suggestions would be requiring one of homebrew/fink/macports), preferably out-of-the-box -- even if that meant restricting to prebuilt binaries. Being able to run non-obscure installers (i.e., from the main python.org and scipy.org sites) for Python + numpy + scipy + matplotlib and get optimized versions would be sufficient. Andrew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 11:57 PM, Pauli Virtanen wrote: > 12.06.2013 00:29, Ralf Gommers kirjoitti: > [clip] > > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, > > so doing something like > > > > export FOPT="-ff2c" > [clip] > > Sounds like a good idea. Would still make sense to move Accelerate down > > in the list of preferred libs, so that one can install ATLAS, MKL or > > OpenBLAS once and be done, instead of always having to remember these > > envvars. > > Btw, it would be interesting to actually check if some combination of > "-ff2c" in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues. https://github.com/scipy/scipy/issues/2256#issuecomment-17028321 Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
12.06.2013 00:29, Ralf Gommers kirjoitti: [clip] > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, > so doing something like > > export FOPT="-ff2c" [clip] > Sounds like a good idea. Would still make sense to move Accelerate down > in the list of preferred libs, so that one can install ATLAS, MKL or > OpenBLAS once and be done, instead of always having to remember these > envvars. Btw, it would be interesting to actually check if some combination of "-ff2c" in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On 11 Jun 2013 22:31, "Ralf Gommers" wrote: > Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. These days it might make sense to just make OpenBLAS the default on all platforms. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 11:17 AM, Pauli Virtanen wrote: > David Cournapeau gmail.com> writes: > [clip] > > What is the default ABI used on homebrew ? I think we should just > > follow that, given that Apple cannot figure it out. > > I think for Scipy homebrew uses the Gfortran ABI: > https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile > > But that's probably the wrong thing to do, it doesn't work: > http://trac.macports.org/ticket/36694 > > For Octave, they have -ff2c: > https://trac.macports.org/browser/trunk/dports/math/octave/Portfile > > *** > > A third option (maybe the best one) could be to add an ABI check > to numpy.distutils BLAS/LAPACK detection --- compile a small test > program that checks SDOT/CDOTU/DDOT etc., and refuse to use the > BLAS/LAPACK libraries if they give incorrect results. After that, > we can also remove the sdot/cdotu wrappers. > > This approach is used by Octave. > > This leaves the problem of dealing with Fortran ABI to those in > charge of the build environment, e.g., macports, Enthought, ..., > who are also in the best position to pick the correct solution > per each platform supported. > > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, > so doing something like > > export FOPT="-ff2c" > > or > > export LDFLAGS="-ldotwrp -lblas" > > works? This makes things a bit more complicated to the builder, an > issue that can be solved with documentation, and keeping that up to > date is easier than hardcoding stuff into numpy.distutils. > Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. https://github.com/scipy/scipy/issues/2256#issuecomment-17028321 has a sample test. Minor issue is that it segfaults with the wrong ABI, instead of giving an incorrect result. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers wrote: > The binaries will still be built against python.org Python, so there > shouldn't be an issue here. Same for building from source. My point was that it's nice to be able to have it build with an out of teh box wetup.py with accelerated LAPACK and all... If whoever is building binaries wants to get fancy, great. > Yeah, I still have an OS X 10.6 machine and Accelerate works great there. > But they screwed it up pretty bad in 10.7, and then made it worse for 10.8. > Not so cool anymore oh well, not on 10.8 yet myself -- if it's broken, it's broken :-( > I think we have to indeed keep it easy to build from source. Maybe providing > a script to automatically fetch all dependencies and then build (like MPL > does) would be a good way to go. Maybe, but stand alone scripts like that get pretty fragile ( I know, I have a few of them myself...) I really want to have a better system in place to build binaries for the mac -- ideally a system that uses the same infrastructure to build a variety of Mac packages, rather than each package having it's own build scripts that need to be maintained and probably overlap with each other. I"m looking at gattai as a system to build on. (http://sourceforge.net/projects/gattai/) There is also the issue of third-party dependencies (libpng, libfreetype, blas, etc, etc) I kind of like how Anaconda appears to be doing it. They have packages that have the shared libs in them, and then other packages can depend on those. Anyway, lots to do! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 2:09 PM, Matthew Brett wrote: > Hi, > > On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen wrote: > > David Cournapeau gmail.com> writes: > > [clip] > >> What is the default ABI used on homebrew ? I think we should just > >> follow that, given that Apple cannot figure it out. > > > > I think for Scipy homebrew uses the Gfortran ABI: > > https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile > > > > But that's probably the wrong thing to do, it doesn't work: > > http://trac.macports.org/ticket/36694 > > > > For Octave, they have -ff2c: > > https://trac.macports.org/browser/trunk/dports/math/octave/Portfile > > > > *** > > > > A third option (maybe the best one) could be to add an ABI check > > to numpy.distutils BLAS/LAPACK detection --- compile a small test > > program that checks SDOT/CDOTU/DDOT etc., and refuse to use the > > BLAS/LAPACK libraries if they give incorrect results. After that, > > we can also remove the sdot/cdotu wrappers. > > > > This approach is used by Octave. > > > > This leaves the problem of dealing with Fortran ABI to those in > > charge of the build environment, e.g., macports, Enthought, ..., > > who are also in the best position to pick the correct solution > > per each platform supported. > > > > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, > > so doing something like > > > > export FOPT="-ff2c" > > > > or > > > > export LDFLAGS="-ldotwrp -lblas" > > > > works? This makes things a bit more complicated to the builder, an > > issue that can be solved with documentation, and keeping that up to > > date is easier than hardcoding stuff into numpy.distutils. > > What will be the performance drop for the default OSX installer > version of numpy, if we drop Accelerate / veclib support? > Answer on scipy-dev: http://article.gmane.org/gmane.comp.python.scientific.devel/17864 Summary: it depends. If anyone knows of better benchmarks, please share. Joern Hees just wrote up how to install with OpenBLAS, if you want to know for your application you can give it a try: http://joernhees.de/blog/2013/06/08/mac-os-x-10-8-scientific-python-with-homebrew/ Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Tue, Jun 11, 2013 at 9:44 PM, Chris Barker - NOAA Federal < chris.bar...@noaa.gov> wrote: > On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker wrote: > > fwiw, homebrew is not macports. it's a more recent replacement that > > seems to be taking over gradually. > > And then there is (or was) fink > > Anyway, it would be really nice if numpy could work well out-of-the > box with the pyton.org python (and or the Apple-supplied one) without > any need fro macports, homebrew, etc. The binaries will still be built against python.org Python, so there shouldn't be an issue here. Same for building from source. > It's actually pretty cool that > Apple provides veclib, and and I liked that it got used by default > with a simple "setup.py build". > Yeah, I still have an OS X 10.6 machine and Accelerate works great there. But they screwed it up pretty bad in 10.7, and then made it worse for 10.8. Not so cool anymore It would be nice to keep it that way. > > If a user (or distributor of binaries) wants to build the whole scipy > stack, they'll need to figure out all this *&^%& anyway, changing a > config in numpy to build differently would be the easiest part. > > And yes, there really are folks that use numpy a lot without scipy. > I think we have to indeed keep it easy to build from source. Maybe providing a script to automatically fetch all dependencies and then build (like MPL does) would be a good way to go. Ralf > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker wrote: > fwiw, homebrew is not macports. it's a more recent replacement that > seems to be taking over gradually. And then there is (or was) fink Anyway, it would be really nice if numpy could work well out-of-the box with the pyton.org python (and or the Apple-supplied one) without any need fro macports, homebrew, etc. It's actually pretty cool that Apple provides veclib, and and I liked that it got used by default with a simple "setup.py build". It would be nice to keep it that way. If a user (or distributor of binaries) wants to build the whole scipy stack, they'll need to figure out all this *&^%& anyway, changing a config in numpy to build differently would be the easiest part. And yes, there really are folks that use numpy a lot without scipy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
> I think for Scipy homebrew uses the Gfortran ABI: > https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile fwiw, homebrew is not macports. it's a more recent replacement that seems to be taking over gradually. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
Hi, On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen wrote: > David Cournapeau gmail.com> writes: > [clip] >> What is the default ABI used on homebrew ? I think we should just >> follow that, given that Apple cannot figure it out. > > I think for Scipy homebrew uses the Gfortran ABI: > https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile > > But that's probably the wrong thing to do, it doesn't work: > http://trac.macports.org/ticket/36694 > > For Octave, they have -ff2c: > https://trac.macports.org/browser/trunk/dports/math/octave/Portfile > > *** > > A third option (maybe the best one) could be to add an ABI check > to numpy.distutils BLAS/LAPACK detection --- compile a small test > program that checks SDOT/CDOTU/DDOT etc., and refuse to use the > BLAS/LAPACK libraries if they give incorrect results. After that, > we can also remove the sdot/cdotu wrappers. > > This approach is used by Octave. > > This leaves the problem of dealing with Fortran ABI to those in > charge of the build environment, e.g., macports, Enthought, ..., > who are also in the best position to pick the correct solution > per each platform supported. > > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, > so doing something like > > export FOPT="-ff2c" > > or > > export LDFLAGS="-ldotwrp -lblas" > > works? This makes things a bit more complicated to the builder, an > issue that can be solved with documentation, and keeping that up to > date is easier than hardcoding stuff into numpy.distutils. What will be the performance drop for the default OSX installer version of numpy, if we drop Accelerate / veclib support? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
David Cournapeau gmail.com> writes: [clip] > What is the default ABI used on homebrew ? I think we should just > follow that, given that Apple cannot figure it out. I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile But that's probably the wrong thing to do, it doesn't work: http://trac.macports.org/ticket/36694 For Octave, they have -ff2c: https://trac.macports.org/browser/trunk/dports/math/octave/Portfile *** A third option (maybe the best one) could be to add an ABI check to numpy.distutils BLAS/LAPACK detection --- compile a small test program that checks SDOT/CDOTU/DDOT etc., and refuse to use the BLAS/LAPACK libraries if they give incorrect results. After that, we can also remove the sdot/cdotu wrappers. This approach is used by Octave. This leaves the problem of dealing with Fortran ABI to those in charge of the build environment, e.g., macports, Enthought, ..., who are also in the best position to pick the correct solution per each platform supported. AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT="-ff2c" or export LDFLAGS="-ldotwrp -lblas" works? This makes things a bit more complicated to the builder, an issue that can be solved with documentation, and keeping that up to date is easier than hardcoding stuff into numpy.distutils. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 9:22 AM, Pauli Virtanen wrote: > Hi, > > numpy.distutils provides support for Accelerate+veclib on OSX, > but does not provide Fortran compiler flags that ensure that > the Fortran ABI used is compatible. As a result ASUM, SDOT, > (apparently also DDOT), and other common BLAS functions > return garbage when called with the wrong ABI. > > This breaks parts of Scipy. (There's some code to hack around > this --- it worked on earlier versions of OSX, but apparently > is unreliable on 10.8, demonstrating that this approach is just > too brittle.) > > Moreover, third-party module developers who use Fortran will not > be aware of this issue, so it may also break 3rd party code. > > As far as I see, the options are: > > 1. Add -ff2c (or whatever is the correct flag for Accelerate) >to the Fortran flags on OSX. What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. > > 2. Drop support for Veclib/Accelerate. > > I think Accelerate was also incompatible with multiprocessing, > which would weigh for option 2. As for multiprocessing support, the problem will happen with other libraries as well. Essentially, the way multiprocessing work (fork wo exec) is simply not supported on OS X. Olivier Grisel from scikits learn knows more about this (and has some workaround to use multiprocessing with numpy/scipy on os x). I will look into the issue with Accelerate on 10.8 (am still on 10.7 myself), but I would not mind dropping support for it if it is too much of an hassle. I will look into the bento issue that makes it fail on 64 bits python, and maybe using openblas should become the default ? David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Dropping support for Accelerate/veclib?
Hi, numpy.distutils provides support for Accelerate+veclib on OSX, but does not provide Fortran compiler flags that ensure that the Fortran ABI used is compatible. As a result ASUM, SDOT, (apparently also DDOT), and other common BLAS functions return garbage when called with the wrong ABI. This breaks parts of Scipy. (There's some code to hack around this --- it worked on earlier versions of OSX, but apparently is unreliable on 10.8, demonstrating that this approach is just too brittle.) Moreover, third-party module developers who use Fortran will not be aware of this issue, so it may also break 3rd party code. As far as I see, the options are: 1. Add -ff2c (or whatever is the correct flag for Accelerate) to the Fortran flags on OSX. 2. Drop support for Veclib/Accelerate. I think Accelerate was also incompatible with multiprocessing, which would weigh for option 2. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
