[email protected] wrote:
> After looking through old threads I came across [1], which discusses the
> effect
> LAPACK-level parallelization has on preformance. For the user that started
> that
> thread, they saw improvements by changing their environmental variables. This
> I understand, what is confusing me is how you figure out which library a Kwant
> distribution is linked to.
>
> E.g. without setting these independently and observing what effect each has on
> performance, how can I tell which of the following environmental variables
> need
> to be changed?
> OPENBLAS_NUM_THREADS
> OMP_NUM_THREADS
> MKL_DYNAMIC
> MKL_NUM_THREADS
Short answer:
Experiment! For OpenBLAS the environment variable OPENBLAS_NUM_THREADS
is specific to OpenBLAS, while OMP_NUM_THREADS applies to everything
that uses OpenMP (including OpenBLAS).
Use top/htop or some similar program to monitor CPU usage as you run
your computation. You want to use all cores, but avoid
oversubscription. Slight oversubscription is OK, but running 16*16
threads on 16 cores will not work well.
Long answer:
Kwant potentially uses two different LAPACKs:
• The module kwant.linalg.lapack exposes the LAPACK that is used by
SciPy. This LAPACK is used whenever Kwant itself needs to call some
LAPACK functionality, for example when computing modes.
• The module kwant.linalg.mumps exposes MUMPS (a library for sparse
linear algebra). MUMPS requires LAPACK, and so Kwant’s mumps module
must be linked to some (potentially different) LAPACK.
To check which LAPACK is used by Kwant with MUMPS, you can use a tool
like ldd (it exists on Linux, no idea about MacOS). For example:
$ cd $(python3 -c 'import kwant; print(kwant.linalg.__file__)' | xargs dirname)
$ ldd *.so
lapack.cpython-311-x86_64-linux-gnu.so:
linux-vdso.so.1 (0x00007ffc0866e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f045035e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0450657000)
_mumps.cpython-311-x86_64-linux-gnu.so:
linux-vdso.so.1 (0x00007fff1a57a000)
libzmumps_scotch-5.5.so =>
/lib/x86_64-linux-gnu/libzmumps_scotch-5.5.so (0x00007fb2d3400000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb2d321f000)
liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3
(0x00007fb2d2a00000)
libmumps_common_scotch-5.5.so =>
/lib/x86_64-linux-gnu/libmumps_common_scotch-5.5.so (0x00007fb2d371f000)
libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
(0x00007fb2d2600000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb2d3140000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb2d37d9000)
libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0
(0x00007fb2d0318000)
libesmumps-7.0.so => /lib/x86_64-linux-gnu/libesmumps-7.0.so
(0x00007fb2d3714000)
libscotch-7.0.so => /lib/x86_64-linux-gnu/libscotch-7.0.so
(0x00007fb2d3683000)
libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
(0x00007fb2d29b9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007fb2d3661000)
libscotcherr-7.0.so => /lib/x86_64-linux-gnu/libscotcherr-7.0.so
(0x00007fb2d365c000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb2d3121000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0
(0x00007fb2d310e000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fb2d298a000)
We can see that my Kwant/MUMPS is linked with OpenBLAS.
One can do the same exercise to find out the LAPACK with which SciPy has
been linked:
$ cd $(python3 -c 'import scipy; print(scipy.linalg.lapack.__file__)' | xargs
dirname)
$ ldd *.so
Cheers
Christoph
