Re: SL7 with security and bug fixes forever - how much work?

[Konstantin Olchanski]

On Mon, Dec 14, 2020 at 12:54:27PM -0800, Yasha Karant wrote:
> ... Out of curiosity, how similar are the Apple Mac ARM CPUs to the CPU used 
> in
> the Fujitsu Fugaku HPC machine (A64FX 48C 2.2GHz)?

Hard to tell. (easy to tell, "just read the specs")

The Intel "magic souce" was always the "cache + memory controller" combo that
consistently runs a few cycles faster compared to competition (IBM, SGI, AMD, 
Altera,
Xilinx, etc), and people see this difference in real-world applications.

How good is the generic ARM cache and memory controller and the Apple cache and 
memory controller,
and in real world applications, remains to be seen. As of a few years ago (1 
GHz era), ARM chips
built for embedded use had pretty slow memory (i.e. single channel vs Intel 
dual/quad channel,
DDR3-1066 vs DDR4-3200, that kind of slow). Today's ARM? I guess I should run 
my memory benchmarks
on my RPi3/RPi4 boards and on the latest Xilinx ARM board we have in the lab...

And then there are the "performance/$$$", "performance/watt" and 
"performance/kg" metrics.

ARM always did well in "performance/watt", meaning that you can cram more of 
them
in the same cooling-limited volume, yielding good "performance/m^3", important 
for building
supercomputers.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: SL7 with security and bug fixes forever - how much work?

[Jon Pruente]

On Mon, Dec 14, 2020 at 2:54 PM Yasha Karant  wrote:

> I knew persons using the X86 Mac compatibility layer on PPC Macs, and
> was told that there was a noticeable performance hit because the
> emulator (more or less) functioned as an "inner interpreter" for a
> totally different ISA.  The same is true between X86-64 and the 64 bit
> ARM ISAs (along with other architectural differences).  Out of
> curiosity, how similar are the Apple Mac ARM CPUs to the CPU used in the
> Fujitsu Fugaku HPC machine (A64FX 48C 2.2GHz)?
>

The big difference is that Apple included, in hardware, the ability to set
the memory ordering to the model x86 uses. This way every translated
instruction doesn't have to move bytes around, which is why the x86
emulation of the recent Microsoft ARM hardware was so terrible. By all
accounts if the code runs in Rosetta 2, it runs very well.

Re: SL7 with security and bug fixes forever - how much work?

[Yasha Karant]

I knew persons using the X86 Mac compatibility layer on PPC Macs, and 
was told that there was a noticeable performance hit because the 
emulator (more or less) functioned as an "inner interpreter" for a 
totally different ISA.  The same is true between X86-64 and the 64 bit 
ARM ISAs (along with other architectural differences).  Out of 
curiosity, how similar are the Apple Mac ARM CPUs to the CPU used in the 
Fujitsu Fugaku HPC machine (A64FX 48C 2.2GHz)?


On 12/14/20 12:12 PM, Jon Pruente wrote:
On Mon, Dec 14, 2020 at 2:07 PM Yasha Karant > wrote:


For the Apple Mac community:  as is known, Apple is leaving the X86-64
platform for an ARM platform.  Will older applications be updated, or
will new (and in some cases, newly licensed-for-fee) applications be
required?


The recent releases of macOS went 64-bit only, which cut out a lot of 
old software. The most recent release that added support for ARM also 
includes a binary compatibility layer to run x86-64 programs called 
Rosetta 2, after the similar layer they used when they transitioned from 
PPC to x86.

Re: SL7 with security and bug fixes forever - how much work?

[Konstantin Olchanski]

> 
> > ... Apple is leaving the X86-64 platform for an ARM platform. ...
> 
> The recent releases of macOS went 64-bit only ...
> ... transitioned from PPC to x86.
> ... cut out a lot of old software
>

I have one leg in the MacOS world and I went through both transitions,
PPC to Intel and 32-bit to "64-bit only".

I remember both were pretty much no-events.

Ran "fink update", "ports update", and all the linux-side software was good.
Joints like VLC, Firefox, Google-chrome, issued updated packages quickly.

Commercial things like the "amazing slow downer", I remember paying
for an updated license (an app well worth the money, recommended.
but very specialized of course).

Moving our linux work from x86 to ARM is similarly a non-event, except
for the need to cross-compile for some non-self-hosting embedded
machines.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: SL7 with security and bug fixes forever - how much work?

[Jon Pruente]

On Mon, Dec 14, 2020 at 2:07 PM Yasha Karant  wrote:

> For the Apple Mac community:  as is known, Apple is leaving the X86-64
> platform for an ARM platform.  Will older applications be updated, or
> will new (and in some cases, newly licensed-for-fee) applications be
> required?
>

The recent releases of macOS went 64-bit only, which cut out a lot of old
software. The most recent release that added support for ARM also includes
a binary compatibility layer to run x86-64 programs called Rosetta 2, after
the similar layer they used when they transitioned from PPC to x86.

Re: SL7 with security and bug fixes forever - how much work?

[Yasha Karant]

I fully agree.  The security-against-compromise integrity of up-ported 
"SL7" using C++n, for some n > n(RHEL 7."latest or last" distro) 
requires "professional" re-evaluation, penetration testing, and 
monitoring.  Will this be done?  I too have run into the same issues 
with backporting, and for that reason switched to Ubuntu current LTS 
(for now).  As for IA-32 support, this will be a larger issue -- 
particularly as much basic research without the possibility of a 
profit-driven product (e.g., a vaccine because of public funds being 
expended for basic research into biology and medical science, both human 
and non-human) is heavily underfunded; hence, your reference to using 
hardware obsolete but fully adequate systems for which current release 
software is becoming increasingly scarce.


For the Apple Mac community:  as is known, Apple is leaving the X86-64 
platform for an ARM platform.  Will older applications be updated, or 
will new (and in some cases, newly licensed-for-fee) applications be 
required?


On 12/14/20 11:55 AM, Konstantin Olchanski wrote:

On Mon, Dec 14, 2020 at 02:36:20PM -0500, Serguei Mokhov wrote:


I have done this in the past with mixed success, typical problems
include "cmake is too old", "autoconf/autotools are too old".


Having run into this at a smaller scale for one of the projects I ended up using
devtoolset-6 (and -7 as appropriate) that ships all this stuff such as
newer GCC,
GLIBC, etc.



Yes, devtoolset is the advertised solution. I bought into it myself,
and then got burned after discovering that is it is missing one critical
32-bit shared library and it cannot be used to cross-compile 32-bit executables,
a "must have" for a couple of experiments that use hardware
with 600-1200 MHz Pentium-3 processors. (replacement hardware is in the USD$5000
price ranges, quantity "to be upgraded" is around 10, none of this has
been budgeted and 64-bit upgrade is not needed for the function
they perform).

Anyhow, is there commitment to devtoolset updates to c++17, c++20, c++21, etc?

For our MIDAS Data Acquisition package, we just started using c++11 recently,
and already we desire to use c++14 and c++17 features. (cannot, because of el7).

Then, there is the question whether SL7 "upgraded to latest versions of 
everything"
is still SL7. ("grand father's axe", etc). For a purist, SL7 is something
installed from an official installer image, adding packages from EPEL and ELREPO
starts pushing it. SL7 with custom-built linux kernel, glibc, sshd, httpd, kde,
php100 and python3000 is probably not SL7 anymore.

The label "SL7" is only useful if it refers to something well defined. If every
instance of SL7 has a different linux kernel, different versions of devtoolset,
different cmake, different python, etc, I would say SL7 no longer exists.

Re: SL7 with security and bug fixes forever - how much work?

[Konstantin Olchanski]

On Mon, Dec 14, 2020 at 02:36:20PM -0500, Serguei Mokhov wrote:
> >
> > I have done this in the past with mixed success, typical problems
> > include "cmake is too old", "autoconf/autotools are too old".
> 
> Having run into this at a smaller scale for one of the projects I ended up 
> using
> devtoolset-6 (and -7 as appropriate) that ships all this stuff such as
> newer GCC,
> GLIBC, etc.
> 

Yes, devtoolset is the advertised solution. I bought into it myself,
and then got burned after discovering that is it is missing one critical
32-bit shared library and it cannot be used to cross-compile 32-bit executables,
a "must have" for a couple of experiments that use hardware
with 600-1200 MHz Pentium-3 processors. (replacement hardware is in the USD$5000
price ranges, quantity "to be upgraded" is around 10, none of this has
been budgeted and 64-bit upgrade is not needed for the function
they perform).

Anyhow, is there commitment to devtoolset updates to c++17, c++20, c++21, etc?

For our MIDAS Data Acquisition package, we just started using c++11 recently,
and already we desire to use c++14 and c++17 features. (cannot, because of el7).

Then, there is the question whether SL7 "upgraded to latest versions of 
everything"
is still SL7. ("grand father's axe", etc). For a purist, SL7 is something
installed from an official installer image, adding packages from EPEL and ELREPO
starts pushing it. SL7 with custom-built linux kernel, glibc, sshd, httpd, kde,
php100 and python3000 is probably not SL7 anymore.

The label "SL7" is only useful if it refers to something well defined. If every
instance of SL7 has a different linux kernel, different versions of devtoolset,
different cmake, different python, etc, I would say SL7 no longer exists.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: SL7 with security and bug fixes forever - how much work?

[Serguei Mokhov]

On Mon, Dec 14, 2020 at 2:27 PM Konstantin Olchanski  wrote:
>
> On Sun, Dec 13, 2020 at 05:36:30PM -0800, Keith Lofstrom wrote:

> For items (4) and (5), one has to take the current source code
> of the applications (and critical system services like httpd),
> and "back port" them to el7.
>
> I have done this in the past with mixed success, typical problems
> include "cmake is too old", "autoconf/autotools are too old".
>
> Each "xxx too old" problem is solved by rebuilding "xxx" from
> current sources, this usually creates a few more "xxx is too old"
> dependancies. By the time you run into "glibc is too old" and "gcc is too 
> old",
> it is time to give up. (notice how there is no KDE5 for el7!).

Having run into this at a smaller scale for one of the projects I ended up using
devtoolset-6 (and -7 as appropriate) that ships all this stuff such as
newer GCC,
GLIBC, etc.

Me too, I ran the 4.x kernels from elrepo on EL6 and EL7 for newer
NVIDIA drivers as well...
All scripted as well, e.g.

   github.com/OpenISS/OpenISS/blob/master/src/scripts/dependencies/el7.sh

for CI servers or containers or similar.

-- 
Serguei Mokhov

Re: SL7 with security and bug fixes forever - how much work?

[Konstantin Olchanski]

On Sun, Dec 13, 2020 at 05:36:30PM -0800, Keith Lofstrom wrote:
> New distro releases imply:
> 
> 1) security fixes,
> 2) bug fixes,
> 3) new hardware drivers,
> 4) new applications,
> 5) and changed behavior (ie, Gnome2 to Gnome3).  
> (the list is probably not complete)
> 

I think "SL7 forever" is a good idea.
I think "SL6 forever" is not practical due to lack of c++11,
and el7 will run into same problem eventually (c++14, c++20, etc).

I think in practice, one would want to retrofit el7 machines
to the current-release linux kernel. This is not difficult
to do, as linux kernel backward compatibility is exemplary.
elrepo already provides the required kernel packages, and it is
easy to retrofit them into the el7 installer image. (ask me).

If you go this route, items (3) is fully covered, items (1) and (2)
are covered for kernel security and bugs.

For application side security, you need to scan all open network ports,
and ensure all applications that talk on that port are covered
for security and bug fixes. A partial list would include sshd, httpd,
email server, dhcp client, ntp client, nfs client and server (userland 
components),
and this brings us to items (4) and (5).

For items (4) and (5), one has to take the current source code
of the applications (and critical system services like httpd),
and "back port" them to el7.

I have done this in the past with mixed success, typical problems
include "cmake is too old", "autoconf/autotools are too old".

Each "xxx too old" problem is solved by rebuilding "xxx" from
current sources, this usually creates a few more "xxx is too old"
dependancies. By the time you run into "glibc is too old" and "gcc is too old",
it is time to give up. (notice how there is no KDE5 for el7!).

All these problems are already solved by "ports" and "homebrew" in the Mac 
world,
where these tools bring the latest versions of linux software to run on
the antique bsd-ish operating system hidden behind the glamour/glitter of MacOS.

As summary, "SL7 is forever" is possible:

a) retrofit current production release linux kernels (i.e. from elrepo, or 
build from source)
b) retrofit the "ports" or "homebrew" system from the Mac world.

How much work it takes? Ask elrepo people (linux kernel) and homebrew people.

P.S. RedHat already do "b" with their "software collections", minus the 
"re-build from source" part.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: SL7 with security and bug fixes forever - how much work?

[Yasha Karant]

I respectfully must slightly disagree with you.

In terms of security fixes and new hardware drivers, the issues of 
backporting (that is, to make a bit of software compatible -- buildable 
-- with the core gcc/g++ environment of a "very old" Linux, including 
SL) may require a great deal of work.  Unlike those environments that 
use "micro-kernels", Linux (thanks to Torvalds' insistence) is a 
monolith, and changing one aspect of a monolith often changes many other 
aspects.  (A micro-kernel -- "Mach" -- approach more easily can have 
encapsulation and "isolation.)


New hardware drivers are needed if one is using the environment on a 
platform that has "current" hardware for which new hardware drivers are 
needed.  This is more common on a laptop workstation, but happens on 
servers and real-time control and data acquisition systems (e.g., the 
experimental environment of HEP).


Could an SL7 base be kept up to date on these issues?  Probably. 
However, a technical staff of a few (less than five) full time 
professionals might not suffice given the number of lines of source code 
that are involved.  Moreover, the current professionals who maintained 
SL at Fermilab/CERN may not want to be involved, given that each may 
have a "permanent" "real job" position at Fermilab/CERN or a 
participating entity (e.g. a university).  Thus, your subscription model 
may not be practical -- and the use of volunteers or compensated "Gig 
economy" "workers" does not result in stability.  Stability requires 
compensated permanent professionals (except for those who are 
independently "wealthy" and are willing to be permanent "volunteers", an 
unlike staff arrangement).


Yasha Karant

On 12/13/20 5:36 PM, Keith Lofstrom wrote:

New distro releases imply:

1) security fixes,
2) bug fixes,
3) new hardware drivers,
4) new applications,
5) and changed behavior (ie, Gnome2 to Gnome3).
(the list is probably not complete)

How much work (staff hours per year, plus volunteer help)
would it take to do JUST (1) and (2) for Scientific Linux
7.8, 7.9,... forever, assuming that RHEL and Debian
sources were available for plagiarism?

I (and perhaps others) chose SL because it was stable.

I do not need to rebuild my working environment to adapt
to changing fashion.  I get SL for free now (thank you!)
but I wouldn't mind paying an annual subscription fee to
support a small team "keeping the plumbing water-tight
and sanitary".  WITHOUT hiding the sink and changing
the knobs, which is what I would get from RedHat/IBM.

How many of us are willing to pay for this, and to
create, contribute, and maintain scientific "extras"
(like the source code for the data reduction for my
published papers) to share with our small community?

Keith