[Numpy-discussion] NumPy's BLAS library on macOS?

[Jerry Morrison]

Would someone please answer installation questions about NumPy's BLAS on
macOS? I'm not finding the answers in the release notes
, the PR
 source, the docs
, or Stack Overflow
.


Q1. The NumPy 1.21.0 release note
 says "This change
enables the Accelerate Framework as an option on macOS." How to set
that option on/off?

Q2. How to determine if NumPy uses Accelerate vs. its internal copy of
OpenBLAS?
After installing a wheel, `numpy.show_config()` shows the openblas_info
library_dirs et al as '/usr/local/lib'. Neither '/usr/local/lib/' nor
'site-packages/numpy/' contains a *blas*.so library (for Python 3.8.* on
macOS 10.14.6) but the doc  says "The OpenBLAS
libraries are included in the wheel."

Q3. How to pip install NumPy 1.21.0 in a way that ensures it uses its
embedded OpenBLAS on macOS as on Linux? I'm aiming for as portable results
as possible. Or should we link NumPy to an external OpenBLAS via `pip
install numpy --no-binary numpy==1.21.0` with `~/.numpy-site.cfg`? (Ditto
for SciPy.)

Q4. Can the new NPY_* environment variables select specific BLAS & LAPACK
libraries through pip install, and perhaps install faster than building
NumPy, SciPy, etc. from source? How to do that?

Q5. Is NumPy's embedded OpenBLAS compiled by gcc or clang? Is that
controllable via `pip install`?

Thank you!
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy's BLAS library on macOS?

[Jerry Morrison]

Got it!

*Summary:*
* Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its
embedded OpenBLAS on every platform that has a wheel.
  That OpenBLAS is always compiled with gcc/gfortran.
  In this case, `np.show_config()` reports `library_dirs =
['/usr/local/lib']` even though there's no libblas in that directory.

* Installing numpy from source (e.g. `pip install numpy==1.21.0 --no-binary
numpy)` looks for BLAS & LAPACK libraries at build time as influenced by
the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or by the file
~/.numpy-site.cfg.
  On macOS, 'accelerate' is in the default search order after 'openblas'.
  On macOS < 11.3, importing numpy that's linked to Accelerate will detect
an Accelerate bug and raise a RuntimeError.

On Wed, Jul 7, 2021 at 1:32 PM Ralf Gommers  wrote:

>
>
> On Wed, Jul 7, 2021 at 9:56 PM Jerry Morrison <
> jerry.morrison+nu...@gmail.com> wrote:
>
>> Would someone please answer installation questions about NumPy's BLAS on
>> macOS? I'm not finding the answers in the release notes
>> <https://github.com/numpy/numpy/releases>, the PR
>> <https://github.com/numpy/numpy/pull/18874> source, the docs
>> <https://numpy.org/doc/1.21/>, or Stack Overflow
>> <https://stackoverflow.com/search?tab=newest&q=%5bnumpy%5d%20openblas>.
>>
>>
>> Q1. The NumPy 1.21.0 release note
>> <https://github.com/numpy/numpy/releases/tag/v1.21.0> says "This change
>> enables the Accelerate Framework as an option on macOS." How to set
>> that option on/off?
>>
>
> It's autodetected at build time. If you have no other BLAS installed, it
> will be used. Or explicitly select it with NPY_BLAS_ORDER/NPY_LAPACK_ORDER
>
>
>> Q2. How to determine if NumPy uses Accelerate vs. its internal copy of
>> OpenBLAS?
>> After installing a wheel, `numpy.show_config()` shows the openblas_info
>> library_dirs et al as '/usr/local/lib'. Neither '/usr/local/lib/' nor
>> 'site-packages/numpy/' contains a *blas*.so library (for Python 3.8.* on
>> macOS 10.14.6) but the doc <https://numpy.org/install/> says "The
>> OpenBLAS libraries are included in the wheel."
>>
>
> It's a build-time option, you cannot select it at runtime.
>
>
>> Q3. How to pip install NumPy 1.21.0 in a way that ensures it uses its
>> embedded OpenBLAS on macOS as on Linux? I'm aiming for as portable results
>> as possible. Or should we link NumPy to an external OpenBLAS via `pip
>> install numpy --no-binary numpy==1.21.0` with `~/.numpy-site.cfg`? (Ditto
>> for SciPy.)
>>
>
> If you install a wheel, you will always get the bundled OpenBLAS on every
> platform for which we have binary wheels.
>
>
>>
>> Q4. Can the new NPY_* environment variables select specific BLAS & LAPACK
>> libraries through pip install, and perhaps install faster than building
>> NumPy, SciPy, etc. from source? How to do that?
>>
>
> This question seems a little bit confused. Those env vars just select the
> BLAS/LAPACK library. It will not affect build time - we're never building
> BLAS or LAPACK itself from source.
>
>
>> Q5. Is NumPy's embedded OpenBLAS compiled by gcc or clang? Is that
>> controllable via `pip install`?
>>
>
> gcc/gfortran. and no, you cannot control it through pip
>
> Cheers,
> Ralf
>
>
>> Thank you!
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy's BLAS library on macOS?

[Jerry Morrison]

On Wed, Jul 7, 2021 at 10:14 PM Matti Picus  wrote:

>
> On 8/7/21 2:23 am, Jerry Morrison wrote:
> > Got it!
> >
> > *Summary:*
> > * Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its
> > embedded OpenBLAS on every platform that has a wheel.
> >   That OpenBLAS is always compiled with gcc/gfortran.
> >   In this case, `np.show_config()` reports `library_dirs =
> > ['/usr/local/lib']` even though there's no libblas in that directory.
> >
> > * Installing numpy from source (e.g. `pip install numpy==1.21.0
> > --no-binary numpy)` looks for BLAS & LAPACK libraries at build time as
> > influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or
> > by the file ~/.numpy-site.cfg.
> >   On macOS, 'accelerate' is in the default search order after 'openblas'.
> >   On macOS < 11.3, importing numpy that's linked to Accelerate will
> > detect an Accelerate bug and raise a RuntimeError.
>
>
> That seems correct, although admittedly show_config could do a better
> job. The problem is that not every BLAS implementation provides a
> convenient method to self-report.
>

For implementations that don't self-report, could show_config detect that
it's using embedded OpenBLAS or a system Accelerate library?


>
> It might be nice to document all this somewhere more permanent, the
> docstring for show_config might be a good place to start.
>

Agreed. On the https://numpy.org/install/ installation page?
Do you want a PR? (How to get the translations?)
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Floating point precision expectations in NumPy

[Jerry Morrison]

On Tue, Jul 27, 2021 at 4:55 PM Sebastian Berg 
wrote:

> Hi all,
>
> there is a proposal to add some Intel specific fast math routine to
> NumPy:
>
> https://github.com/numpy/numpy/pull/19478
>
> part of numerical algorithms is that there is always a speed vs.
> precision trade-off, giving a more precise result is slower.
>
> So there is a question what the general precision expectation should be
> in NumPy.  And how much is it acceptable to diverge in the
> precision/speed trade-off depending on CPU/system?
>
> I doubt we can formulate very clear rules here, but any input on what
> precision you would expect or trade-offs seem acceptable would be
> appreciated!
>
>
> Some more details
> -
>
> This is mainly interesting e.g. for functions like logarithms,
> trigonometric functions, or cubic roots.
>
> Some basic functions (multiplication, addition) are correct as per IEEE
> standard and give the best possible result, but these are typically
> only correct within very small numerical errors.
>
> This is typically measured as "ULP":
>
>  https://en.wikipedia.org/wiki/Unit_in_the_last_place
>
> where 0.5 ULP would be the best possible result.
>
>
> Merging the PR may mean relaxing the current precision slightly in some
> places.  In general Intel advertises 4 ULP of precision (although the
> actual precision for most functions seems better).
>
>
> Here are two tables, one from glibc and one for the Intel functions:
>
>
> https://www.gnu.org/software/libc/manual/html_node/Errors-in-Math-Functions.html
> (Mainly the LA column)
> https://software.intel.com/content/www/us/en/develop/documentation/onemkl-vmperfdata/top/real-functions/measured-accuracy-of-all-real-vm-functions.html
>
>
> Different implementation give different accuracy, but formulating some
> guidelines/expectation (or referencing them) would be useful guidance.
>

"Close enough" depends on the application but non-linear models can get the
"butterfly effect" where the results diverge if they aren't identical.

For a certain class of scientific programming applications, reproducibility
is paramount.

Development teams may use a variety of development laptops, workstations,
scientific computing clusters, and cloud computing platforms. If the tests
pass on your machine but fail in CI, you have a debugging problem.

If your published scientific article links to source code that replicates
your computation, scientists will expect to be able to run that code, now
or in a couple decades, and replicate the same outputs. They'll be using
different OS releases and maybe different CPU + accelerator architectures.

Reproducible Science is good. Replicated Science is better.


Clearly there are other applications where it's easy to trade
reproducibility and some precision for speed.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Floating point precision expectations in NumPy

[Jerry Morrison]

On Fri, Jul 30, 2021 at 12:22 PM Sebastian Berg 
wrote:

> On Fri, 2021-07-30 at 11:04 -0700, Jerry Morrison wrote:
> > On Tue, Jul 27, 2021 at 4:55 PM Sebastian Berg <
> > sebast...@sipsolutions.net>
> > wrote:
> >
> > > Hi all,
> > >
> > > there is a proposal to add some Intel specific fast math routine to
> > > NumPy:
> > >
> > > https://github.com/numpy/numpy/pull/19478
> > >
> > > part of numerical algorithms is that there is always a speed vs.
> > > precision trade-off, giving a more precise result is slower.
> > >
>
> 
>


> > "Close enough" depends on the application but non-linear models can
> > get the
> > "butterfly effect" where the results diverge if they aren't
> > identical.
>
>
> Right, so my hope was to gauge what the general expectation is.  I take
> it you expect a high accuracy.
>
> The error for the computations itself is seems low on first sight, but
> of course they can explode quickly in non-linear settings...
> (In the chaotic systems I worked with, the shadowing theorem would
> usually alleviate such worries. And testing the integration would be
> more important.  But I am sure for certain questions things may be far
> more tricky.)
>

I'll put forth an expectation that after installing a specific set of
libraries, the floating point results would be identical across platforms
and into the future. Ideally developers could install library updates (for
hardware compatibility, security fixes, or other reasons) and still get
identical results.

That expectation is for reproducibility, not high accuracy. So it'd be fine
to install different libraries [or maybe use those pip package options in
brackets, whatever they do?] to trade accuracy for speed. Could any
particular choice of accuracy still provide reproducible results across
platforms and time?



> > For a certain class of scientific programming applications,
> > reproducibility
> > is paramount.
> >
> > Development teams may use a variety of development laptops,
> > workstations,
> > scientific computing clusters, and cloud computing platforms. If the
> > tests
> > pass on your machine but fail in CI, you have a debugging problem.
> >
> > If your published scientific article links to source code that
> > replicates
> > your computation, scientists will expect to be able to run that code,
> > now
> > or in a couple decades, and replicate the same outputs. They'll be
> > using
> > different OS releases and maybe different CPU + accelerator
> > architectures.
> >
> > Reproducible Science is good. Replicated Science is better.
> > <http://rescience.github.io/>
> >
> > Clearly there are other applications where it's easy to trade
> > reproducibility and some precision for speed.
>
>
> Agreed, although there are so many factors, often out of our control,
> that I am not sure that true replicability is achievable without
> containers :(.
>
> It would be amazing if NumPy could have a "replicable" mode, but I am
> not sure how that could be done, or if the "ground work" in the math
> and linear algebra libraries even exists.
>
>
> However, even if it is practically impossible to make things
> replicable, there is an argument for improving reproducibility and
> replicability, e.g. by choosing the high-accuracy version here.  Even
> if it is impossible to actually ensure.
>

Yes! Let's at least have reproducibility in mind and work on improving
it, e.g. by removing failure modes.
(Ditto for security :-)
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Re: Consider linking to Accelerate in Apple Silicon ARM64 wheels

[Jerry Morrison]

This sounds terrific. Is there a technique to get such a performance
improvement with Accelerate?

We tried:

pip install numpy==1.24.3 --no-binary numpy


with and without a ~/.numpy-site.cfg file that sets

[accelerate]
libraries = Accelerate, vecLib


on Intel and M2 Macs on macOS 13.3. While np.show_config() reports that
numpy is linked to Accelerate, a large dot product runs maybe twice as fast
as with OpenBLAS, and our full bio simulation's performance is within
measurement noise.

Trying the --no-use-pep517 argument had no obvious impact.

Best,
Jerry


On Thu, Mar 23, 2023 at 3:52 AM Ralf Gommers  wrote:

>
>
> On Thu, Mar 23, 2023 at 10:43 AM Clemens Brunner <
> clemens.brun...@gmail.com> wrote:
>
>> Thanks Ralf, this sounds great! Just making sure I understand, this means
>> that for macOS 13, we have to enable Accelerate by building NumPy from
>> source.
>
>
> Indeed. Either that, or use a packaging system that's more capable in this
> regard - conda-forge for example will give you runtime switching of BLAS
> and LAPACK libraries out of the box (
> https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation).
> Several Linux distros support this too, via mechanisms like
> `update-alternatives` - but that won't help you much on macOS of course.
>
> Once macOS 13.3 is out, building SciPy from source will also link to
>> Accelerate. Finally, Accelerate support will be enabled by default in
>> binary wheels for NumPy and SciPy once macOS 14 is out (presumably some
>> time in October this year). Correct?
>>
>
> Yes, if I had to guess now then I'd say that this will be the default in
> NumPy 2.0 at the end of the year.
>
> Cheers,
> Ralf
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Consider linking to Accelerate in Apple Silicon ARM64 wheels

[Jerry Morrison]

On Tue, May 9, 2023 at 12:13 PM Ralf Gommers  wrote:

>
>
> On Tue, May 9, 2023 at 5:11 AM Jerry Morrison <
> jerry.morrison+nu...@gmail.com> wrote:
>
>> This sounds terrific. Is there a technique to get such a performance
>> improvement with Accelerate?
>>
>> We tried:
>>
>> pip install numpy==1.24.3 --no-binary numpy
>>
>>
>> with and without a ~/.numpy-site.cfg file that sets
>>
>> [accelerate]
>> libraries = Accelerate, vecLib
>>
>>
>> on Intel and M2 Macs on macOS 13.3. While np.show_config() reports that
>> numpy is linked to Accelerate, a large dot product runs maybe twice as fast
>> as with OpenBLAS, and our full bio simulation's performance is within
>> measurement noise.
>>
>> Trying the --no-use-pep517 argument had no obvious impact.
>>
>
> From the release notes: "To use the new interfaces, define
> ACCELERATE_NEW_LAPACK before including the Accelerate or vecLib headers".
> So you are still using the old version of the library if you did not do
> that. This is not entirely trivial to do now; I expect we'll see a PR to
> enable the new libraries quite soon.
>

... macOS Ventura 13.3 release notes. So Clemens must've tweaked the Numpy
source code or build commands to do that.

OK, I'll wait for this PR.

Thanks, Ralf!



>
> Cheers,
> Ralf
>
>
>
>>
>> Best,
>> Jerry
>>
>>
>> On Thu, Mar 23, 2023 at 3:52 AM Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>> On Thu, Mar 23, 2023 at 10:43 AM Clemens Brunner <
>>> clemens.brun...@gmail.com> wrote:
>>>
>>>> Thanks Ralf, this sounds great! Just making sure I understand, this
>>>> means that for macOS 13, we have to enable Accelerate by building NumPy
>>>> from source.
>>>
>>>
>>> Indeed. Either that, or use a packaging system that's more capable in
>>> this regard - conda-forge for example will give you runtime switching of
>>> BLAS and LAPACK libraries out of the box (
>>> https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation).
>>> Several Linux distros support this too, via mechanisms like
>>> `update-alternatives` - but that won't help you much on macOS of course.
>>>
>>> Once macOS 13.3 is out, building SciPy from source will also link to
>>>> Accelerate. Finally, Accelerate support will be enabled by default in
>>>> binary wheels for NumPy and SciPy once macOS 14 is out (presumably some
>>>> time in October this year). Correct?
>>>>
>>>
>>> Yes, if I had to guess now then I'd say that this will be the default in
>>> NumPy 2.0 at the end of the year.
>>>
>>> Cheers,
>>> Ralf
>>>
>>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com