Re: Help for emmax needed (Was: Removing ATLAS?)

[Sébastien Villemot]

Control: tags -1 + patch

Hi Andreas,

Le mercredi 29 novembre 2023 à 10:06 +0100, Andreas Tille a écrit :
> Control: tags -1 help
> 
> Am Fri, Jul 14, 2023 at 01:40:22AM +0200 schrieb Sébastien Villemot:
> > Le lundi 10 juillet 2023 à 22:01 +0200, Andreas Tille a écrit :
> > > I've checked my responsibility for the dependencies and stumbled about
> > > emmax
> > > 
> > > 
> > > emmax.c:10:10: fatal error: clapack.h: No such file or directory
> > >10 | #include 
> > >   |  ^~~
> > > compilation terminated.
> > > ...
> > 
> > clapack.h is apparently an early attempt at providing a C interface to
> > some LAPACK functions (which are in Fortran). It indeed looks ATLAS-
> > specific.
> > 
> > The modern solution for that problem is to use LAPACKE (note the final
> > “E”), which is provided by liblapacke-dev. It is a standardized and
> > maintained C interface to all LAPACK routines.
> > 
> > I guess it should be feasible to adapt emmax to make it work with
> > LAPACKE.
> 
> I tried to do so in Salsa.  Unfortunately LAPACKE is not a drop in
> replacement and I'm lacking the necessary knowledge about all these
> LAPACK implementations.  For instance I have no idea how to fix the
> error you can see in Salsa CI[1] which has
> 
> ...
> emmax.c: In function 'dsyevd':
> emmax.c:1633:17: error: conflicting types for 'dsyevd_'; have 'void(char *, 
> char *, int *, double *, int *, double *, double *, int *, int *, int *, int 
> *)'
>  1633 |   extern  void  dsyevd_ (char *JOBZp, char *UPLOp, int *Np,
>   | ^~~
> In file included from /usr/include/lapack.h:11,
>  from emmax.c:10:
> /usr/include/lapack.h:17096:6: note: previous declaration of 'dsyevd_' with 
> type 'void(const char *, const char *, const int32_t *, double *, const 
> int32_t *, double *, double *, const int32_t *, int32_t *, const int32_t *, 
> int32_t *, size_t,  size_t)' {aka 'void(const char *, const char *, const int 
> *, double *, const int *, double *, double *, const int *, int *, const int 
> *, int *, long unsigned int,  long unsigned int)'}
> 17096 | void LAPACK_dsyevd_base(
>   |  ^~
> emmax.c: In function 'dsyevr':
> emmax.c:1652:17: error: conflicting types for 'dsyevr_'; have 'void(char *, 
> char *, char *, int *, double *, int *, double *, double *, int *, int *, 
> double *, int *, double *, double *, int *, int *, double *, int *, int *, 
> int *, int *)'
>  1652 |   extern  void  dsyevr_ (char *JOBZp, char *RANGEp, char *UPLOp, int 
> *Np,
>   | ^~~
> 
> Looking at the last line of this snippet (line 901 in the linked CI
> log[1]) the parameter char *RANGEp is not part of the declaration of
> LAPACK_dsyevd_base given in line 17096 of /usr/include/lapack.h.
> 
> I admit this is hard to solve for me and I would love if someone
> with some knowledge of LAPACK could provide some patch.  A similar
> issue exists for dgetrf.
> 
> Any help would be appreciated.

I opened a merge request that completes the migration away from ATLAS:

 https://salsa.debian.org/med-team/emmax/-/merge_requests/1

Best,

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Help for emmax needed (Was: Removing ATLAS?)

[Andreas Tille]

Control: tags -1 help

Hi,

Am Fri, Jul 14, 2023 at 01:40:22AM +0200 schrieb Sébastien Villemot:
> Le lundi 10 juillet 2023 à 22:01 +0200, Andreas Tille a écrit :
> > I've checked my responsibility for the dependencies and stumbled about
> > emmax
> > 
> > 
> > emmax.c:10:10: fatal error: clapack.h: No such file or directory
> >10 | #include 
> >   |  ^~~
> > compilation terminated.
> > ...
> 
> clapack.h is apparently an early attempt at providing a C interface to
> some LAPACK functions (which are in Fortran). It indeed looks ATLAS-
> specific.
> 
> The modern solution for that problem is to use LAPACKE (note the final
> “E”), which is provided by liblapacke-dev. It is a standardized and
> maintained C interface to all LAPACK routines.
> 
> I guess it should be feasible to adapt emmax to make it work with
> LAPACKE.

I tried to do so in Salsa.  Unfortunately LAPACKE is not a drop in
replacement and I'm lacking the necessary knowledge about all these
LAPACK implementations.  For instance I have no idea how to fix the
error you can see in Salsa CI[1] which has

...
emmax.c: In function 'dsyevd':
emmax.c:1633:17: error: conflicting types for 'dsyevd_'; have 'void(char *, 
char *, int *, double *, int *, double *, double *, int *, int *, int *, int *)'
 1633 |   extern  void  dsyevd_ (char *JOBZp, char *UPLOp, int *Np,
  | ^~~
In file included from /usr/include/lapack.h:11,
 from emmax.c:10:
/usr/include/lapack.h:17096:6: note: previous declaration of 'dsyevd_' with 
type 'void(const char *, const char *, const int32_t *, double *, const int32_t 
*, double *, double *, const int32_t *, int32_t *, const int32_t *, int32_t *, 
size_t,  size_t)' {aka 'void(const char *, const char *, const int *, double *, 
const int *, double *, double *, const int *, int *, const int *, int *, long 
unsigned int,  long unsigned int)'}
17096 | void LAPACK_dsyevd_base(
  |  ^~
emmax.c: In function 'dsyevr':
emmax.c:1652:17: error: conflicting types for 'dsyevr_'; have 'void(char *, 
char *, char *, int *, double *, int *, double *, double *, int *, int *, 
double *, int *, double *, double *, int *, int *, double *, int *, int *, int 
*, int *)'
 1652 |   extern  void  dsyevr_ (char *JOBZp, char *RANGEp, char *UPLOp, int 
*Np,
  | ^~~

Looking at the last line of this snippet (line 901 in the linked CI
log[1]) the parameter char *RANGEp is not part of the declaration of
LAPACK_dsyevd_base given in line 17096 of /usr/include/lapack.h.

I admit this is hard to solve for me and I would love if someone
with some knowledge of LAPACK could provide some patch.  A similar
issue exists for dgetrf.

Any help would be appreciated.

Kind regards
   Andreas.

[1] https://salsa.debian.org/med-team/emmax/-/jobs/4977705#L901

-- 
http://fam-tille.de

Re: Removing ATLAS?

[Sébastien Villemot]

Le dimanche 05 novembre 2023 à 14:02 +0100, Sébastien Villemot a
écrit :
> Le samedi 08 juillet 2023 à 10:01 +0200, Sébastien Villemot a écrit :
> > As the maintainer of the atlas package over the last decade, I now
> > wonder whether we should remove it from the archive.
> 
> Since the present thread seems to indicate that there to be a consensus
> towards removing atlas from Debian, I am going to move forward.

Thanks to those who gave me feedback.

I went ahead and reported the bugs, here is the list:

 
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=atlas-rm;users=debian-science@lists.debian.org

Any help in fixing those is of course much welcome.

Best,

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Sébastien Villemot]

Hi,

Le dimanche 05 novembre 2023 à 21:49 -0500, Mo Zhou a écrit :
> The following line is what I use for the PyTorch package (ignore the 
> fact that I forgot to bump the BLIS abi from 3 to 4):
> 
> Recommends: libopenblas0 | libblis3 | libatlas3-base | libmkl-rt | libblas3
> 
> So the latest Recommends line for performance critical packages relying 
> on BLAS/LAPACK should use:
> 
> Recommends: libopenblas0 | libblis4 | libmkl-rt | libblas3
> 
> libmkl-rt is oudated (I knew it), but anyway usable. Quite a few months 
> ago I wrote the plan for overhauling the MKL package. Will carry it out 
> when I'm available for that.

Indeed I think it is ok to add libmkl-rt here despite the fact that it
is non-free, because it is not the default alternative (see Policy
§2.2.1). I’m not really sure that I want to encourage that though.

> libblas3 as the last fallback when none of the others are available. 
> This seems missing from the message draft.

It is not necessary to mention libblas3 in Recommends because the
Depends field normally already includes libblas.so.3 | libblas3.
Hence libblas3 is already the last fallback.

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Mo Zhou]

The following line is what I use for the PyTorch package (ignore the 
fact that I forgot to bump the BLIS abi from 3 to 4):


Recommends: libopenblas0 | libblis3 | libatlas3-base | libmkl-rt | libblas3

So the latest Recommends line for performance critical packages relying 
on BLAS/LAPACK should use:


Recommends: libopenblas0 | libblis4 | libmkl-rt | libblas3

libmkl-rt is oudated (I knew it), but anyway usable. Quite a few months 
ago I wrote the plan for overhauling the MKL package. Will carry it out 
when I'm available for that.


libblas3 as the last fallback when none of the others are available. 
This seems missing from the message draft.


BLIS has a quite good architecture coverage compared to openblas. 
Currently we only miss libblis4 on hurd-amd64 (I'm surprised. Hurd has 
finally got amd64 version), and arc. While BLIS does not really optimize 
every architecture, it should be anyway better than the very naive 
fortran implementation.


On 11/5/23 08:02, Sébastien Villemot wrote:

Le samedi 08 juillet 2023 à 10:01 +0200, Sébastien Villemot a écrit :

As the maintainer of the atlas package over the last decade, I now
wonder whether we should remove it from the archive.

Since the present thread seems to indicate that there to be a consensus
towards removing atlas from Debian, I am going to move forward.

Please find below a bug report template, which I plan to use for
reporting bugs against the ~20 packages that currently have a (build-
)dependency against atlas. Feedback is welcome.

--

Subject: (build-)depends on atlas, which is obsolete and scheduled for removal

Package: $PACKAGE / Source: $SOURCE
Version: $VERSION
Tags: sid trixie
User: debian-science@lists.debian.org
Usertags: atlas-rm

Dear Maintainer,

$SOURCE build-depends on libatlas-base-dev / $PACKAGE depends on
libatlas3-base, which is produced by the source package atlas.

atlas is obsolete and scheduled to be removed from Debian, ideally by the
trixie release. See the following thread on the Debian Science list for more
details:

  
https://lists.debian.org/msgid-search/4311acc16afb473599c79bd5b17a8b734c2f8d2b.ca...@debian.org

As a consequence, please drop any (build-)dependency on atlas.

This should normally be straightforward to achieve, by simply replacing atlas
with another BLAS (and possibly also LAPACK) implementation.

The typical setup is to build depend on the Netlib reference implementation
(libblas-dev and possibly also liblapack-dev), and to not enforce anything in
dependencies of binary packages.

If an optimized BLAS/LAPACK implementation is needed at build time (for example
for accelerating tests), then libopenblas-dev and libblis-dev offer good
options (keeping in mind that openblas is only available on some
architectures).

If one wants to encourage users to install an optimized
implementation, then one can use “Recommends: libopenblas0 | libblis4” in
binary packages.

Also note that if your package needs libcblas (which is currently only provided
by libatlas-base-dev), then the solution is to modify the build system so that
it rather uses libblas (because, under Debian, the latter already incorporates
the symbols provided by libcblas).

Thanks for your work,

Re: Removing ATLAS?

[Drew Parsons]

On 2023-11-05 14:02, Sébastien Villemot wrote:

Le samedi 08 juillet 2023 à 10:01 +0200, Sébastien Villemot a écrit :

As the maintainer of the atlas package over the last decade, I now
wonder whether we should remove it from the archive.


Since the present thread seems to indicate that there to be a consensus
towards removing atlas from Debian, I am going to move forward.

Please find below a bug report template, which I plan to use for
reporting bugs against the ~20 packages that currently have a (build-
)dependency against atlas. Feedback is welcome.

...


As a consequence, please drop any (build-)dependency on atlas.

This should normally be straightforward to achieve, by simply replacing 
atlas

with another BLAS (and possibly also LAPACK) implementation.

The typical setup is to build depend on the Netlib reference 
implementation
(libblas-dev and possibly also liblapack-dev), and to not enforce 
anything in

dependencies of binary packages.



I would make these paragraphs stronger, emphasising that the build 
should always normally be done against generic BLAS/LAPACK, with the 
optimised libraries to be used at runtime rather than build-time.  For 
Build-Depends, we tend to use alternatives with the generic build placed 
first to be picked up by buildds, while not getting in the way of anyone 
building for themselves locally (with an optimised implementation 
installed)


My suggested wording might be

...
As a consequence, please drop any (build-)dependency on atlas.

This should be straightforward to achieve, by simply replacing atlas
with another BLAS (and possibly also LAPACK) implementation.

Normally, packages should Build-Depend on the Netlib reference 
implementation
(libblas-dev, and liblapack-dev where required), and to *not* enforce 
anything
in the Depends field of binary packages (dpkg-shlibs will automatically 
add the

appropriate "libblas3 | libblas.so.3" entry to ${shlibs:Depends}).

Alternative implementations may be given in Build-Depends for the ease 
of users making local builds with an optimised implementation installed, 
but the generic reference implementation should be placed first to be 
used by buildds.

The simplest example is
  Build-Depends: libblas-dev | libblas.so
  Build-Depends: liblapack-dev | liblapack.so
where specific optimised implementations may provide 
libblas.so/liblapack.so


If a specific optimised implementation is preferred at runtime, that 
could be hinted in the order or alternative Build-Depends,

e.g.
  Build-Depends: libblas-dev | libopenblas-dev | libblis-dev | 
libblas.so

or
  Build-Depends: libblas-dev | libblis-dev | libopenblas-dev | 
libblas.so
though that will not affect buildds which will use the first alternative 
libblas-dev in any case.
(it will also not affect the entry automatically added to 
${shlibs:Depends}).

...




Drew

Re: Removing ATLAS?

[Sébastien Villemot]

Le samedi 08 juillet 2023 à 10:01 +0200, Sébastien Villemot a écrit :
> As the maintainer of the atlas package over the last decade, I now
> wonder whether we should remove it from the archive.

Since the present thread seems to indicate that there to be a consensus
towards removing atlas from Debian, I am going to move forward.

Please find below a bug report template, which I plan to use for
reporting bugs against the ~20 packages that currently have a (build-
)dependency against atlas. Feedback is welcome.

--

Subject: (build-)depends on atlas, which is obsolete and scheduled for removal

Package: $PACKAGE / Source: $SOURCE
Version: $VERSION
Tags: sid trixie
User: debian-science@lists.debian.org
Usertags: atlas-rm

Dear Maintainer,

$SOURCE build-depends on libatlas-base-dev / $PACKAGE depends on
libatlas3-base, which is produced by the source package atlas.

atlas is obsolete and scheduled to be removed from Debian, ideally by the
trixie release. See the following thread on the Debian Science list for more
details:

 
https://lists.debian.org/msgid-search/4311acc16afb473599c79bd5b17a8b734c2f8d2b.ca...@debian.org

As a consequence, please drop any (build-)dependency on atlas.

This should normally be straightforward to achieve, by simply replacing atlas
with another BLAS (and possibly also LAPACK) implementation.

The typical setup is to build depend on the Netlib reference implementation
(libblas-dev and possibly also liblapack-dev), and to not enforce anything in
dependencies of binary packages.

If an optimized BLAS/LAPACK implementation is needed at build time (for example
for accelerating tests), then libopenblas-dev and libblis-dev offer good
options (keeping in mind that openblas is only available on some
architectures).

If one wants to encourage users to install an optimized
implementation, then one can use “Recommends: libopenblas0 | libblis4” in
binary packages.

Also note that if your package needs libcblas (which is currently only provided
by libatlas-base-dev), then the solution is to modify the build system so that
it rather uses libblas (because, under Debian, the latter already incorporates
the symbols provided by libcblas).

Thanks for your work,

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Rafael Laboissière]

* M. Zhou  [2023-07-17 12:05]:


On Fri, 2023-07-14 at 01:51 +0200, Sébastien Villemot wrote:


Your fix looks good. Note that an even better fix is to simply Build- 
Depend on libblas-dev. Linking against an optimized BLAS does not 
really help at build time, because since all variants are ABI 
compatible and use the same SONAME, it’s the runtime dependency that 
really matters.


I agree that an optimized blas is not necessary as build-depeneds. 
Just want to mention for some computational intensive packages 
during dh_auto_test, an optimized BLAS may help. Or the tests 
can take forever to run.


The xmds2 package is an Arch:all package. It does not compile binary 
programs or libraries at build time that are be linked against BLAS. It 
needs an optimized BLAS library for running the unit tests, whose code in 
C++ is automatically generated and compiled on the fly.


I will release a new version of the package that build-depends and 
depends on libopenblas-dev | libblis-dev.


I am wondering whether we should propose my patch [*] (or something 
similar) to the upstream authors. Thoughts are welcome.


Best,

Rafael Laboissière

[*] 
https://salsa.debian.org/science-team/xmds2/-/blob/cf0fc3d85e86ebdeba9d929d5ecf656ce7657259/debian/patches/vanilla-cblas.patch

Re: Removing ATLAS?

[M. Zhou]

On Fri, 2023-07-14 at 01:51 +0200, Sébastien Villemot wrote:
> 
> Your fix looks good. Note that an even better fix is to simply Build-
> Depend on libblas-dev. Linking against an optimized BLAS does not
> really help at build time, because since all variants are ABI
> compatible and use the same SONAME, it’s the runtime dependency that
> really matters.
> 

I agree that an optimized blas is not necessary as build-depeneds.
Just want to mention for some computational intensive packages
during dh_auto_test, an optimized BLAS may help. Or the tests
can take forever to run.

Re: Removing ATLAS?

[Dave Love]

Nilesh Patra  writes:

> You can send patches via BTS, if you don't want to create a salsa
> account. BTS is the standard way of filing/reacting to bug reports.

Sure, I know and approve of the email BTS, but it doesn't help for
contributing.  I'm happy to have a salsa account, as I did alioth(?),
just not to interact with Google and be put into semi-impossible captcha
hell.

Re: Removing ATLAS?

[Nilesh Patra]

On Fri, Jul 14, 2023 at 11:33:37AM +0100, Dave Love wrote:
> (I'd contribute to Debian packaging if it didn't require an account on
> salsa, which requires Recaptcha.)

You can send patches via BTS, if you don't want to create a salsa
account. BTS is the standard way of filing/reacting to bug reports.

All it needs is an email address (which you have).

https://www.debian.org/Bugs/

Best,
Nilesh


signature.asc
Description: PGP signature

Re: Removing ATLAS?

[Dave Love]

Rafael Laboissière  writes:

> * Sébastien Villemot  [2023-07-08 10:01]:
>
>> As the maintainer of the atlas package over the last decade, I now
>> wonder whether we should remove it from the archive.

I'd say definitely.  I don't know of any value in atlas these days.

I'm not a Debian developer/maintainer, but I noticed this and know a
bit about it as a sometime HPC system manager and packager who lost the
battle getting Fedora to adopt a Debian-style linear algebra policy.
(I'd contribute to Debian packaging if it didn't require an account on
salsa, which requires Recaptcha.)

>> Moreover, there are better alternatives. Most people today use
>> OpenBLAS. There is also BLIS, which can in particular be used on
>> architectures not supported by OpenBLAS.

I don't suppose high performance for anything not supported by OpenBLAS
is terribly interesting.  However, if you're interested in the sort of
performance you might expect with a GEMM micro-kernel in C on a
simple-ish micro-architecture (not AVX512), maybe see
.  That has a
reference to choosing a priori block sizes analytically.  (I think there
are also pure C micro-kernels in OpenBLAS, but I haven't checked, and
they will be harder to deal with.)

>> Also note that ATLAS has never been really well-suited to our
>> distribution model. To get the most of ATLAS, you have to recompile
>> it locally using the specific CPU that you want to target; a generic
>> binary package like the one we distribute is a suboptimal solution,
>> since it is not adapted to the local CPU cache size.

Or the SIMD, which loses a factor of four on Haswell, for instance,
without dynamic dispatch or building for -- finally! -- x86_64 hwcaps.

> When generating the C++ code, the xmds tool tries to link against
> libcblas.so, which, currently, only exists in the libatlas-base-dev
> package.

For what it's worth, you can trivially make a shim to supply the shared
object, assuming you have the header from openblas or blis and that
atlas provides the same cblas interface.  See
.  (The
Fedora blis package ships with one for libblas.)

Re: Removing ATLAS?

[Sébastien Villemot]

Le mercredi 12 juillet 2023 à 15:34 +0200, Rafael Laboissière a écrit :
> * Sébastien Villemot  [2023-07-08 10:01]:
> 
> > As the maintainer of the atlas package over the last decade, I now 
> > wonder whether we should remove it from the archive.

[…]

> Thanks for starting this discussion and proposing a plan, Sébastien.
> 
> I looked at xmds2, which currently build-depends and depends on 
> libatlas-base-dev. For those who do not know, xmds2 is a tool written in 
> Python for generating C++ code that solvesq differential equations, using 
> a XML file with the description of the problem.
> 
> When generating the C++ code, the xmds tool tries to link against 
> libcblas.so, which, currently, only exists in the libatlas-base-dev 
> package.
> 
> I “fixed” the problem with a hacky patch to the file 
> xpdeint/support/wscript and by changing the dependencies to 
> libopenblas-dev | libblis-dev. If someone can review my changes [*], I 
> will be grateful.

Your fix looks good. Note that an even better fix is to simply Build-
Depend on libblas-dev. Linking against an optimized BLAS does not
really help at build time, because since all variants are ABI
compatible and use the same SONAME, it’s the runtime dependency that
really matters.


-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Sébastien Villemot]

Le mercredi 12 juillet 2023 à 16:05 +0200, Andreas Tille a écrit :
> Am Wed, Jul 12, 2023 at 03:34:09PM +0200 schrieb Rafael Laboissière:
> > When generating the C++ code, the xmds tool tries to link against
> > libcblas.so, which, currently, only exists in the libatlas-base-dev package.
> 
> I admit this is actually the reason why any of the packages I'm maintaining
> depends from libatlas-base-dev:  These are linking against libcblas which is
> only provided in this package.  If we might find a way to provide some kind
> of "wrapper" so other implementations could provide libcblas as well, this
> could be some generic solution.  Unfortunately I'm probably pretty naive
> and I'm asking for the impossible.

Short version: any package that links against libcblas can normally be
fixed by linking against libblas.

Long version: for historical reasons, libblas.so in Debian contains
both the BLAS Fortran routines, and a C interface to these routines
(called CBLAS). Hence all the symbols that are in libcblas.so (which
have the common “cblas_” prefix) are present in libblas.so. (Note that
this is different from the LAPACK case: liblapack.so only includes the
Fortran routines, while the C interface is provided by the separate
liblapacke.so). You can check this for yourself by running the
following commands:
$ objdump -T /usr/lib/x86_64-linux-gnu/libblas.so
$ objdump -T /usr/lib/x86_64-linux-gnu/libcblas.so

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Sébastien Villemot]

Le lundi 10 juillet 2023 à 22:01 +0200, Andreas Tille a écrit :
> Hi Sébastian,
> 
> Am Sat, Jul 08, 2023 at 10:01:15AM +0200 schrieb Sébastien Villemot:
> > 
> > So, given all that, I’m inclined to (try to) remove atlas during the
> > trixie development cycle.
> 
> Sounds reasonable.
>  
> > Any thought on this?
> 
> I've checked my responsibility for the dependencies and stumbled about
> emmax
> 
> 
> emmax.c:10:10: fatal error: clapack.h: No such file or directory
>10 | #include 
>   |  ^~~
> compilation terminated.
> 
> 
> and noticed
> 
> $ apt-file search clapack.h
> libatlas-base-dev: /usr/include/x86_64-linux-gnu/clapack.h
> ... (other instances are not blas related)
> 
> which was probably the reason why I've picked libatlas-base-dev
> originally.  I would not say that emmax is important and its
> also a not maintained upstream any more.  However, I vaguely
> remember that this
>#include 
> in some code pieces of Debian Med was the reason to actually
> pick this blas implementation.  Any hint how to deal with such
> cases?

clapack.h is apparently an early attempt at providing a C interface to
some LAPACK functions (which are in Fortran). It indeed looks ATLAS-
specific.

The modern solution for that problem is to use LAPACKE (note the final
“E”), which is provided by liblapacke-dev. It is a standardized and
maintained C interface to all LAPACK routines.

I guess it should be feasible to adapt emmax to make it work with
LAPACKE.

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org



signature.asc
Description: This is a digitally signed message part

Re: Removing ATLAS?

[Andreas Tille]

Hi Rafael,

Am Wed, Jul 12, 2023 at 03:34:09PM +0200 schrieb Rafael Laboissière:
> When generating the C++ code, the xmds tool tries to link against
> libcblas.so, which, currently, only exists in the libatlas-base-dev package.

I admit this is actually the reason why any of the packages I'm maintaining
depends from libatlas-base-dev:  These are linking against libcblas which is
only provided in this package.  If we might find a way to provide some kind
of "wrapper" so other implementations could provide libcblas as well, this
could be some generic solution.  Unfortunately I'm probably pretty naive
and I'm asking for the impossible.
 
Kind regards
Andreas.

-- 
http://fam-tille.de

Re: Removing ATLAS?

[Rafael Laboissière]

* Sébastien Villemot  [2023-07-08 10:01]:

As the maintainer of the atlas package over the last decade, I now 
wonder whether we should remove it from the archive.


As a reminder, ATLAS is an optimized BLAS implementation, that fits 
into our BLAS/LAPACK alternatives framework.¹ Its strategy for 
achieving good performance is to adjust various internal array sizes 
(at build time) so that they fit in the processor cache. It was 
probably the first optimized BLAS added to Debian (in 1999).


Today, the project looks dead. The last stable release (3.10.3) 
happened in 2016. The last development release (3.11.41, not packaged) 
was in 2018. The git repository has seen no commit since 2019.²


Moreover, there are better alternatives. Most people today use 
OpenBLAS. There is also BLIS, which can in particular be used on 
architectures not supported by OpenBLAS.


Also note that ATLAS has never been really well-suited to our 
distribution model. To get the most of ATLAS, you have to recompile it 
locally using the specific CPU that you want to target; a generic 
binary package like the one we distribute is a suboptimal solution, 
since it is not adapted to the local CPU cache size.


So, given all that, I’m inclined to (try to) remove atlas during the 
trixie development cycle.


There are quite a few package which (build-)depend on atlas, I attach a 
list. But my guess is that these should be easily fixable, because most 
(if not all) do not require ATLAS specifically. One should normally not 
need to build-depend on atlas, since all our BLAS implementations are 
ABI-compatible (build-depending on libblas-dev should give an 
equivalent binary, unless one is doing static linking). For the 
dependencies of binary packages, I guess those were added to ensure 
that the user has an optimized BLAS installed; so they can probably be 
replaced by something like libopenblas0 | libblis4 (keeping in mind 
that since BLAS/LAPACK implementations are managed by the alternatives 
system, a dependency relationship cannot enforce the implementation 
used at runtime on the user machine).


Any thought on this?


Thanks for starting this discussion and proposing a plan, Sébastien.

I looked at xmds2, which currently build-depends and depends on 
libatlas-base-dev. For those who do not know, xmds2 is a tool written in 
Python for generating C++ code that solvesq differential equations, using 
a XML file with the description of the problem.


When generating the C++ code, the xmds tool tries to link against 
libcblas.so, which, currently, only exists in the libatlas-base-dev 
package.


I “fixed” the problem with a hacky patch to the file 
xpdeint/support/wscript and by changing the dependencies to 
libopenblas-dev | libblis-dev. If someone can review my changes [*], I 
will be grateful.


Another issue is whether we should propose this patch (or something 
similar) to the upstream authors. Thoughts are welcome.


Best,

Rafael

[*] 
https://salsa.debian.org/science-team/xmds2/-/commit/cf0fc3d85e86ebdeba9d929d5ecf656ce7657259

Re: Removing ATLAS?

[Andreas Tille]

Hi Steffen,

Am Tue, Jul 11, 2023 at 10:02:53AM +0200 schrieb Steffen Möller:
> EMMAX is (was) important
> https://genome.sph.umich.edu/wiki/EMMAX
> 
> but I admittedly cannot invest into maintaining this or atlas.

Sure we can't maintain this.

> My suggestion would be to talk back to upstream about transitioning to an 
> alternative. I have no idea about how difficult this would be to patch 
> ourselves.

Do you have some contact to upstream?  The version number is
 beta-07Mar2010
 
which is usually a sign that upstream has orphaned this code long
ago.  If you can reach upstream it would be great if you could ping
them somehow.

Kind regards
 Andreas.

> Best,
> Steffen
> 
> > Gesendet: Montag, 10. Juli 2023 um 22:01 Uhr
> > Von: "Andreas Tille" 
> > An: debian-science@lists.debian.org
> > Betreff: Re: Removing ATLAS?
> >
> > Hi Sébastian,
> > 
> > Am Sat, Jul 08, 2023 at 10:01:15AM +0200 schrieb Sébastien Villemot:
> > > 
> > > So, given all that, I’m inclined to (try to) remove atlas during the
> > > trixie development cycle.
> > 
> > Sounds reasonable.
> >  
> > > Any thought on this?
> > 
> > I've checked my responsibility for the dependencies and stumbled about
> > emmax
> > 
> > 
> > emmax.c:10:10: fatal error: clapack.h: No such file or directory
> >10 | #include 
> >   |  ^~~
> > compilation terminated.
> > 
> > 
> > and noticed
> > 
> > $ apt-file search clapack.h
> > libatlas-base-dev: /usr/include/x86_64-linux-gnu/clapack.h
> > ... (other instances are not blas related)
> > 
> > which was probably the reason why I've picked libatlas-base-dev
> > originally.  I would not say that emmax is important and its
> > also a not maintained upstream any more.  However, I vaguely
> > remember that this
> >#include 
> > in some code pieces of Debian Med was the reason to actually
> > pick this blas implementation.  Any hint how to deal with such
> > cases?
> >  
> > Kind regards
> > Andreas.
> >  
> > > Checking reverse dependencies...
> > > # Broken Depends:
> > > ceres-solver: libceres-dev
> > >   libceres3
> > > colmap: colmap [amd64 arm64 i386 mips64el mipsel ppc64el s390x]
> > > dune-common: libdune-common-dev
> > > emmax: emmax
> > > ergo: ergo
> > > iml: libiml0
> > > nipy: python3-nipy-lib [armel mipsel]
> > > psfex: psfex
> > > python-escript: python3-escript [armel mipsel]
> > > python3-escript-mpi [armel mipsel]
> > > qm-dsp: libqm-dsp0
> > > scamp: scamp [amd64 arm64 mips64el ppc64el s390x]
> > > scikit-misc: python3-skmisc
> > > sight: libsight [amd64]
> > > source-extractor: source-extractor
> > > xmds2: xmds2
> > > 
> > > # Broken Build-Depends:
> > > ceres-solver: libatlas-base-dev
> > > dune-common: libatlas-base-dev
> > > emmax: libatlas-base-dev
> > >libatlas3-base
> > > ergo: libatlas-base-dev
> > > ghmm: libatlas-base-dev
> > > halide: libatlas-base-dev
> > > hpcc: libatlas-base-dev
> > > iml: libatlas-base-dev
> > > ncl: libatlas3-base
> > > odin: libatlas-base-dev
> > > opm-material: libatlas-base-dev
> > > phast: libatlas-base-dev
> > > plink1.9: libatlas-base-dev
> > > plink2: libatlas-base-dev
> > > psfex: libatlas-base-dev
> > > qm-dsp: libatlas-base-dev
> > > scamp: libatlas-base-dev
> > > scikit-misc: libatlas-base-dev
> > > source-extractor: libatlas-base-dev
> > > theli: libatlas-base-dev
> > > xmds2: libatlas-base-dev
> > > 
> > > Dependency problem found.
> > > 
> > 
> > 
> > 
> > 
> > -- 
> > http://fam-tille.de
> > 
> >
> 
> 

-- 
http://fam-tille.de

Aw: Re: Removing ATLAS?

[Steffen Möller]

Hi all,

EMMAX is (was) important
https://genome.sph.umich.edu/wiki/EMMAX

but I admittedly cannot invest into maintaining this or atlas.
My suggestion would be to talk back to upstream about transitioning to an 
alternative. I have no idea about how difficult this would be to patch 
ourselves.

Best,
Steffen

> Gesendet: Montag, 10. Juli 2023 um 22:01 Uhr
> Von: "Andreas Tille" 
> An: debian-science@lists.debian.org
> Betreff: Re: Removing ATLAS?
>
> Hi Sébastian,
> 
> Am Sat, Jul 08, 2023 at 10:01:15AM +0200 schrieb Sébastien Villemot:
> > 
> > So, given all that, I’m inclined to (try to) remove atlas during the
> > trixie development cycle.
> 
> Sounds reasonable.
>  
> > Any thought on this?
> 
> I've checked my responsibility for the dependencies and stumbled about
> emmax
> 
> 
> emmax.c:10:10: fatal error: clapack.h: No such file or directory
>10 | #include 
>   |  ^~~
> compilation terminated.
> 
> 
> and noticed
> 
> $ apt-file search clapack.h
> libatlas-base-dev: /usr/include/x86_64-linux-gnu/clapack.h
> ... (other instances are not blas related)
> 
> which was probably the reason why I've picked libatlas-base-dev
> originally.  I would not say that emmax is important and its
> also a not maintained upstream any more.  However, I vaguely
> remember that this
>#include 
> in some code pieces of Debian Med was the reason to actually
> pick this blas implementation.  Any hint how to deal with such
> cases?
>  
> Kind regards
> Andreas.
>  
> > Checking reverse dependencies...
> > # Broken Depends:
> > ceres-solver: libceres-dev
> >   libceres3
> > colmap: colmap [amd64 arm64 i386 mips64el mipsel ppc64el s390x]
> > dune-common: libdune-common-dev
> > emmax: emmax
> > ergo: ergo
> > iml: libiml0
> > nipy: python3-nipy-lib [armel mipsel]
> > psfex: psfex
> > python-escript: python3-escript [armel mipsel]
> > python3-escript-mpi [armel mipsel]
> > qm-dsp: libqm-dsp0
> > scamp: scamp [amd64 arm64 mips64el ppc64el s390x]
> > scikit-misc: python3-skmisc
> > sight: libsight [amd64]
> > source-extractor: source-extractor
> > xmds2: xmds2
> > 
> > # Broken Build-Depends:
> > ceres-solver: libatlas-base-dev
> > dune-common: libatlas-base-dev
> > emmax: libatlas-base-dev
> >libatlas3-base
> > ergo: libatlas-base-dev
> > ghmm: libatlas-base-dev
> > halide: libatlas-base-dev
> > hpcc: libatlas-base-dev
> > iml: libatlas-base-dev
> > ncl: libatlas3-base
> > odin: libatlas-base-dev
> > opm-material: libatlas-base-dev
> > phast: libatlas-base-dev
> > plink1.9: libatlas-base-dev
> > plink2: libatlas-base-dev
> > psfex: libatlas-base-dev
> > qm-dsp: libatlas-base-dev
> > scamp: libatlas-base-dev
> > scikit-misc: libatlas-base-dev
> > source-extractor: libatlas-base-dev
> > theli: libatlas-base-dev
> > xmds2: libatlas-base-dev
> > 
> > Dependency problem found.
> > 
> 
> 
> 
> 
> -- 
> http://fam-tille.de
> 
>

Re: Removing ATLAS?

[Andreas Tille]

Hi Sébastian,

Am Sat, Jul 08, 2023 at 10:01:15AM +0200 schrieb Sébastien Villemot:
> 
> So, given all that, I’m inclined to (try to) remove atlas during the
> trixie development cycle.

Sounds reasonable.
 
> Any thought on this?

I've checked my responsibility for the dependencies and stumbled about
emmax


emmax.c:10:10: fatal error: clapack.h: No such file or directory
   10 | #include 
  |  ^~~
compilation terminated.


and noticed

$ apt-file search clapack.h
libatlas-base-dev: /usr/include/x86_64-linux-gnu/clapack.h
... (other instances are not blas related)

which was probably the reason why I've picked libatlas-base-dev
originally.  I would not say that emmax is important and its
also a not maintained upstream any more.  However, I vaguely
remember that this
   #include 
in some code pieces of Debian Med was the reason to actually
pick this blas implementation.  Any hint how to deal with such
cases?
 
Kind regards
Andreas.
 
> Checking reverse dependencies...
> # Broken Depends:
> ceres-solver: libceres-dev
>   libceres3
> colmap: colmap [amd64 arm64 i386 mips64el mipsel ppc64el s390x]
> dune-common: libdune-common-dev
> emmax: emmax
> ergo: ergo
> iml: libiml0
> nipy: python3-nipy-lib [armel mipsel]
> psfex: psfex
> python-escript: python3-escript [armel mipsel]
> python3-escript-mpi [armel mipsel]
> qm-dsp: libqm-dsp0
> scamp: scamp [amd64 arm64 mips64el ppc64el s390x]
> scikit-misc: python3-skmisc
> sight: libsight [amd64]
> source-extractor: source-extractor
> xmds2: xmds2
> 
> # Broken Build-Depends:
> ceres-solver: libatlas-base-dev
> dune-common: libatlas-base-dev
> emmax: libatlas-base-dev
>libatlas3-base
> ergo: libatlas-base-dev
> ghmm: libatlas-base-dev
> halide: libatlas-base-dev
> hpcc: libatlas-base-dev
> iml: libatlas-base-dev
> ncl: libatlas3-base
> odin: libatlas-base-dev
> opm-material: libatlas-base-dev
> phast: libatlas-base-dev
> plink1.9: libatlas-base-dev
> plink2: libatlas-base-dev
> psfex: libatlas-base-dev
> qm-dsp: libatlas-base-dev
> scamp: libatlas-base-dev
> scikit-misc: libatlas-base-dev
> source-extractor: libatlas-base-dev
> theli: libatlas-base-dev
> xmds2: libatlas-base-dev
> 
> Dependency problem found.
> 




-- 
http://fam-tille.de

Re: Removing ATLAS?

[M. Zhou]

I agree. The usage of ATLAS is more suitable for source based distros
like Gentoo. Plus, according to my past benchmarks, ATLAS, even if
compiled locally with -march=native flags, still falls behind OpenBLAS
and BLIS in terms of performance.

Both OpenBLAS and BLIS are still healthy, actively maintained.
So I agree it is time to let old libraries fade away.

BTW, deprecating ATLAS can also help us remove the libcblas.so
as well as fixing its reverse dependencies to use the correct libblas.so.



On Sat, 2023-07-08 at 10:01 +0200, Sébastien Villemot wrote:
> Hi,
> 
> As the maintainer of the atlas package over the last decade, I now
> wonder whether we should remove it from the archive.
> 
> As a reminder, ATLAS is an optimized BLAS implementation, that fits
> into our BLAS/LAPACK alternatives framework.¹ Its strategy for
> achieving good performance is to adjust various internal array sizes
> (at build time) so that they fit in the processor cache. It was
> probably the first optimized BLAS added to Debian (in 1999).
> 
> Today, the project looks dead. The last stable release (3.10.3)
> happened in 2016. The last development release (3.11.41, not packaged)
> was in 2018. The git repository has seen no commit since 2019.²
> 
> Moreover, there are better alternatives. Most people today use
> OpenBLAS. There is also BLIS, which can in particular be used on
> architectures not supported by OpenBLAS.
> 
> Also note that ATLAS has never been really well-suited to our
> distribution model. To get the most of ATLAS, you have to recompile it
> locally using the specific CPU that you want to target; a generic
> binary package like the one we distribute is a suboptimal solution,
> since it is not adapted to the local CPU cache size.
> 
> So, given all that, I’m inclined to (try to) remove atlas during the
> trixie development cycle.
> 
> There are quite a few package which (build-)depend on atlas, I attach a
> list. But my guess is that these should be easily fixable, because most
> (if not all) do not require ATLAS specifically. One should normally not
> need to build-depend on atlas, since all our BLAS implementations are
> ABI-compatible (build-depending on libblas-dev should give an
> equivalent binary, unless one is doing static linking). For the
> dependencies of binary packages, I guess those were added to ensure
> that the user has an optimized BLAS installed; so they can probably be
> replaced by something like libopenblas0 | libblis4 (keeping in mind
> that since BLAS/LAPACK implementations are managed by the alternatives
> system, a dependency relationship cannot enforce the implementation
> used at runtime on the user machine).
> 
> Any thought on this?
> 
> Cheers,
> 
> ¹ https://wiki.debian.org/DebianScience/LinearAlgebraLibraries
> ² https://github.com/math-atlas/math-atlas/
> 

Removing ATLAS?

[Sébastien Villemot]

Hi,

As the maintainer of the atlas package over the last decade, I now
wonder whether we should remove it from the archive.

As a reminder, ATLAS is an optimized BLAS implementation, that fits
into our BLAS/LAPACK alternatives framework.¹ Its strategy for
achieving good performance is to adjust various internal array sizes
(at build time) so that they fit in the processor cache. It was
probably the first optimized BLAS added to Debian (in 1999).

Today, the project looks dead. The last stable release (3.10.3)
happened in 2016. The last development release (3.11.41, not packaged)
was in 2018. The git repository has seen no commit since 2019.²

Moreover, there are better alternatives. Most people today use
OpenBLAS. There is also BLIS, which can in particular be used on
architectures not supported by OpenBLAS.

Also note that ATLAS has never been really well-suited to our
distribution model. To get the most of ATLAS, you have to recompile it
locally using the specific CPU that you want to target; a generic
binary package like the one we distribute is a suboptimal solution,
since it is not adapted to the local CPU cache size.

So, given all that, I’m inclined to (try to) remove atlas during the
trixie development cycle.

There are quite a few package which (build-)depend on atlas, I attach a
list. But my guess is that these should be easily fixable, because most
(if not all) do not require ATLAS specifically. One should normally not
need to build-depend on atlas, since all our BLAS implementations are
ABI-compatible (build-depending on libblas-dev should give an
equivalent binary, unless one is doing static linking). For the
dependencies of binary packages, I guess those were added to ensure
that the user has an optimized BLAS installed; so they can probably be
replaced by something like libopenblas0 | libblis4 (keeping in mind
that since BLAS/LAPACK implementations are managed by the alternatives
system, a dependency relationship cannot enforce the implementation
used at runtime on the user machine).

Any thought on this?

Cheers,

¹ https://wiki.debian.org/DebianScience/LinearAlgebraLibraries
² https://github.com/math-atlas/math-atlas/

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  https://sebastien.villemot.name
⠈⠳⣄  https://www.debian.org

sebastien@coccia:~$ dak rm -nR atlas
Will remove the following packages from unstable:
 
 atlas |  3.10.3-13 | source
libatlas-base-dev |  3.10.3-13 | amd64, arm64, armel, armhf, i386, mips64el, 
mipsel, ppc64el, s390x
libatlas-doc |  3.10.3-13 | all
libatlas-test |  3.10.3-13 | amd64, arm64, armel, armhf, i386, mips64el, 
mipsel, ppc64el, s390x
libatlas3-base |  3.10.3-13 | amd64, arm64, armel, armhf, i386, mips64el, 
mipsel, ppc64el, s390x

Maintainer: Debian Science Team 


--- Reason ---

--

Checking reverse dependencies...
# Broken Depends:
ceres-solver: libceres-dev
  libceres3
colmap: colmap [amd64 arm64 i386 mips64el mipsel ppc64el s390x]
dune-common: libdune-common-dev
emmax: emmax
ergo: ergo
iml: libiml0
nipy: python3-nipy-lib [armel mipsel]
psfex: psfex
python-escript: python3-escript [armel mipsel]
python3-escript-mpi [armel mipsel]
qm-dsp: libqm-dsp0
scamp: scamp [amd64 arm64 mips64el ppc64el s390x]
scikit-misc: python3-skmisc
sight: libsight [amd64]
source-extractor: source-extractor
xmds2: xmds2

# Broken Build-Depends:
ceres-solver: libatlas-base-dev
dune-common: libatlas-base-dev
emmax: libatlas-base-dev
   libatlas3-base
ergo: libatlas-base-dev
ghmm: libatlas-base-dev
halide: libatlas-base-dev
hpcc: libatlas-base-dev
iml: libatlas-base-dev
ncl: libatlas3-base
odin: libatlas-base-dev
opm-material: libatlas-base-dev
phast: libatlas-base-dev
plink1.9: libatlas-base-dev
plink2: libatlas-base-dev
psfex: libatlas-base-dev
qm-dsp: libatlas-base-dev
scamp: libatlas-base-dev
scikit-misc: libatlas-base-dev
source-extractor: libatlas-base-dev
theli: libatlas-base-dev
xmds2: libatlas-base-dev

Dependency problem found.



signature.asc
Description: This is a digitally signed message part