CentOS EOL - politics?

[Keith Lofstrom]

The big physics labs that supported Scientific Linux get
much or all of their funding from the US government, and
that funding is allocated by politicians who depend on
campaign contributions. 

I wonder how much IBM contributes to the politicians who
make the funding decisions for the labs, and I wonder if
there is subtle back-channel pressure on lab software
purchases and project funding decisions?  

I'm not worried about any deep conspiracy, but if we want
the Scientific Linux ship to sail our way, we should learn
which way the winds are blowing.

Some of those winds may involve corporate users of SL and
CentOS.  There are surely some.  I'm a "corp" but a tiny
one.  I can ask friends at companies like Intel and Nike.

Which versions of Linux are ancestral to Google?  Amazon?
Netflix?  They probably have their own in-house "distros", 
but they may desire new-hires with compatible knowledge.
Lobbying for federal investment in SL may be good for 
their business.

I can also talk with my congresscritters, if any of them
are upstream of the committees that allocate US federal
funding for Fermilabs, CERN, etc. 

Which congressional committees fund our big labs?

With many things shaking loose in Washington DC over the
next few months, it is probably a good time to make our
moves before new structures freeze into place.

Keith

-- 
Keith Lofstrom  kei...@keithl.com

Re: CentOS EOL - politics?

[Maarten]

Netflix uses FreeBSD:


 1.2.2. Who Uses FreeBSD?

FreeBSD has been known for its web serving capabilities - sites that run 
on FreeBSD include Hacker News , Netcraft 
, NetEase , Netflix 
, Sina , 
Sony Japan , Rambler , 
Yahoo! , and Yandex .


https://urldefense.proofpoint.com/v2/url?u=https-3A__www.freebsd.org_doc_en-5FUS.ISO8859-2D1_books_handbook_nutshell.html&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=5fJfk8aMCJQ4SLbj_GePA3BkCcsMnlHaPouZQbObUOI&s=We4EZFPsDGEUE2jnV7YncmuJZ4xQzDkE4-ImMZz0q98&e= 


On 12/11/20 11:31 AM, Keith Lofstrom wrote:

The big physics labs that supported Scientific Linux get
much or all of their funding from the US government, and
that funding is allocated by politicians who depend on
campaign contributions.

I wonder how much IBM contributes to the politicians who
make the funding decisions for the labs, and I wonder if
there is subtle back-channel pressure on lab software
purchases and project funding decisions?

I'm not worried about any deep conspiracy, but if we want
the Scientific Linux ship to sail our way, we should learn
which way the winds are blowing.

Some of those winds may involve corporate users of SL and
CentOS.  There are surely some.  I'm a "corp" but a tiny
one.  I can ask friends at companies like Intel and Nike.

Which versions of Linux are ancestral to Google?  Amazon?
Netflix?  They probably have their own in-house "distros",
but they may desire new-hires with compatible knowledge.
Lobbying for federal investment in SL may be good for
their business.

I can also talk with my congresscritters, if any of them
are upstream of the committees that allocate US federal
funding for Fermilabs, CERN, etc.

Which congressional committees fund our big labs?

With many things shaking loose in Washington DC over the
next few months, it is probably a good time to make our
moves before new structures freeze into place.

Keith

Re: CentOS EOL - politics?

[Brett Viren]

This is not a political reply.

Keith Lofstrom  writes:

> The big physics labs that supported Scientific Linux get
> much or all of their funding from the US government,

CERN is primarily funded by CERN nation states, of which US is not one.

FNAL, being a US DOE National Lab, is primarily funded by US DOE.

> I wonder how much IBM contributes to the politicians who
> make the funding decisions for the labs, and I wonder if
> there is subtle back-channel pressure on lab software
> purchases and project funding decisions?  

The subtle pressure theory is very doubtful to me.  Here is why:

1. The various HEP/NP clusters are almost universally on SL7 so have
   until ca 2024 to figure out wtf they will do next.  So, there's
   simply nothing there to apply any subtle pressure against.  And, any
   argument to move the clusters from SL7 before we get closer to 2024
   would have to be very compelling.

2. The cost of going "full RHEL" for the clusters is prohibitive.
   Removing the previously expected "CentOS future" and applying subtle
   pressure will not magically make funding appear to pay for full RHEL
   licensing on the clusters.  IBM would have to offer a deep,
   essentially exponential, price break as a function of $(nproc).  But,
   even if they did, RHEL does not really offer anything novel and
   useful for the clusters.  Clusters update their OS far less
   frequently than the OS updates become available so something like
   RHEN licensing is useless.  Cluster admins are DEEP experts so don't
   need the paid hand holding.

3. There is already some RHEL penetration in labs for various "servers"
   so that market is partly saturated.  Even if it was still fully open,
   it's a rather mid-level salesdroid "get" (in terms of profit).  So,
   it does not seem to me to motivate any "subtle pressure" tactic.
   Where special RHEL-only drivers can't be avoided, the few systems
   that fall into this category can also use RHEL if there were no
   alternative.  

4. Little real movement to CentOS 8 has occurred.  So, few have fallen
   into that particular tar pit.  Some isolated suckers (and knowing
   some of them well, I say that with fondness) have, and they'll need
   to dig out but by and large, there is no ensnared market here to
   leverage with subtle pressure.  I think the move to CentOS 8 was just
   picking up steam.  Had IBM waited, say 1 year, they'd have ensnared
   many more in the tar pit.  So, if any dodgy tactic was going on, they
   blew it.

5. At the physicist "personal" laptop/workstation level, Mac dominates,
   with Windows next.  The small fraction of "personal" Linux I'd guess
   SL is no a majority.  There's a lot of Ubuntu and Fedora.  Like in
   general, most of those using SL are on SL7 so also fall into the "got
   until 2024 to figure wtf they will do next".

So, the way I see it, labs are simply not in a position of being "subtly
pressured" on this issue and have ample time to figure out a next step
solution for the various demographics.  A few systems are in the tar pit
but can (must) get out in some way.  A few others are already in on RHEL
so don't care.


My hope is they (we) take this current situation as a lesson and make a
radical change that puts all of our computing on more sustainable
footing as we go into the next decades.

-Brett.


signature.asc
Description: PGP signature

Re: CentOS EOL - politics?

[Yasha Karant]

I agree with your analysis, save for three comments.  Mine also is not a 
political comment, merely an analysis of fact.


Overwhelmingly throughout the world, HEP is funded by public funds 
(sometimes from totalitarian dictatorships if one can call such 
"public").  HEP addresses basic science, fundamental physics as do some 
aspects of astronomy and cosmology, which is of no interest to 
for-profits other than a bit of technology spin-off (sometimes useful in 
the "consumer" sector, more often in the weapons sector).  The only 
reasons a for-profit corporation in a "market" (including democratic 
representative government neo-liberal oligopolies) funds fundamental 
physics (I do not consider materials science fundamental, in that the 
Standard Model, with appropriate computational capability and methods of 
"solving" the underlying quantum field theory equations, including 
quantum statistical mechanics, seems to be in full agreement, and even 
has quantitative predictive power, for what has been discovered in 
materials science under terrestrial conditions) is for publicity, for 
proof of concept, and for tax writeoffs, often through vendor 
partnerships that offer "huge" discounts from the commercial prices. 
The equivalent areas of materials science, biology, etc., are of far 
greater interest, particularly if wealth transference can be used 
(public funding of the underlying research, but oligopoly profit from 
the actual deployed products).  This public funding stream is 
fundamental to carrying out experimental as well as computational 
theoretical physics -- instrumentality is required (science is not "pure 
mathematics", despite what some think looking at any modern hard science 
research paper).  To those in HEP, all of the above is known and 
"obvious", but to many in the wider SL community, it is not well understood.


There is adequate time to move servers past SL7 -- if IBM RH does not do 
for CentOS 7 what it did for CentOS 8.  Marketing promises from most 
for-profit entities are unenforceable and subject effectively to whim 
(typically perceived in the best interest of the entity, often only in 
terms of short-term stock value, not long-term, in the 
quarter-by-quarter financial market model).  However, if new hardware 
from vendors does not support the basic kernel gcc libraries of SL7, but 
requires features from later production releases (say, what currently is 
used by Ubuntu LTS), then the HEP community will have *LOTS* of 
backporting to do.


The issue of what to do past SL7 is the question.  Security compromises 
to SL7 probably will be minimal in so far as RHEL7 through CentOS 7 is 
supported, but once unsupported ("EOL"), it becomes increasingly 
hazardous to keep the OS in any "mission-critical" production 
environment.  My personal guess is that in order to allocate financial 
and personnel "resources" to other parts of the Fermilab/CERN 
activities, SL was dropped for CentOS.  This choice is not possible. 
Assuming that many of the servers actually are under a type 1 
hypervisor, it is relatively easy to start deployment of new supervisor 
environments and test these.  Otherwise, either machines must be 
de-allocated from production and put into test mode for a new supervisor 
environment, or additional platforms need to be procured.  Testing must 
be done near scale, as we all well know -- testing on a "high powered" 
workstation is not the same as testing on a clustered HPC machine, 
perhaps with a SAN, and other platform features.


As for the use of Mac OS X or MS Win (probably 10 right now) on the 
desktop, that is a matter of taste and funding.  As Apple again has 
changed the Mac platform, now from X86-64 to ARM, and as Apple strictly 
is a for-profit entity, there will need to be a massive re-investment in 
new Mac machines, unless the X86-64 platforms convert to BSD or Linux, 
and Linux is much easier to support for HEP built around SL.  A 
technical question:  for those HEP workstations that are using Mac OS X, 
is Fink or the equivalent installed so that "standard" applications 
easily can be ported?


On 12/11/20 8:09 AM, Brett Viren wrote:

This is not a political reply.

Keith Lofstrom  writes:


The big physics labs that supported Scientific Linux get
much or all of their funding from the US government,


CERN is primarily funded by CERN nation states, of which US is not one.

FNAL, being a US DOE National Lab, is primarily funded by US DOE.


I wonder how much IBM contributes to the politicians who
make the funding decisions for the labs, and I wonder if
there is subtle back-channel pressure on lab software
purchases and project funding decisions?


The subtle pressure theory is very doubtful to me.  Here is why:

1. The various HEP/NP clusters are almost universally on SL7 so have
until ca 2024 to figure out wtf they will do next.  So, there's
simply nothing there to apply any subtle pressure against.  And, any
argument to move the clusters from S

Re: CentOS EOL - politics?

[Jon Pruente]

On Fri, Dec 11, 2020 at 12:11 PM Yasha Karant  wrote:

> A technical question:  for those HEP workstations that are using Mac OS X,
> is Fink or the equivalent installed so that "standard" applications
> easily can be ported?
>

AFAICT, Fink is barely used. The main community package managers on macOS
these days are Homebrew and Mac Ports.

Sustainable computing - Re: CentOS EOL - politics?

[~Stack~]

On 12/11/20 10:09 AM, Brett Viren wrote:

My hope is they (we) take this current situation as a lesson and make a
radical change that puts all of our computing on more sustainable
footing as we go into the next decades.


I'm curious about your thoughts on what it means to have that 
sustainable footing going forward.


We have been pushing our users to Singularity images for the last two 
years (we jumped on pretty early). A LOT of our application/code base is 
already Singularity behind the scenes. The users don't know and don't 
care because their applications still run the same on the same HPC 
equipment. However, getting our users to purposefully think in terms of 
Singularity images has been a long hard road and we still have so much 
further to go.


We are on the edge of shifting a few very critical and heavy 
computations to Kubernetes. I'm not yet convinced that it will replace a 
lot of the hard-core traditional HPC workloads anytime soon, but there 
are a surprising amount of workloads that can. Plus, it allows us to 
automate from Code->Gitlab->CI/CD->Kubernetes->results delightfully well.


But one of the absolute greatest things about it from the perspective of 
what CentOS just pulled is that my dev Kubernetes has three OS's. SL7, 
Ubuntu 20.04, and CentOS 8 (I JUST spun this up the Monday before the 
announcement). As an admin, I _don't_ care about the OS at this point of 
the Kubernetes process. I kill a node and rebuild it to anything that 
supports the docker requirements (plus a few other things I need for 
company audit/security) and join it to the cluster. Done! When I killed 
that CentOS 8 node I suffered no loss in the slightest in terms of 
functionality and only about an hour of time where I had to move the 
workload and rebuild the node Ubuntu.


Bigger shops with decent sized teams, these transitions can be done over 
time. But the vast majority of my career I've supported hundreds of 
compute nodes where the entire HPC team was just me plus my manager and 
we had to support the clusters for 5-8 years (especially when I was in 
the university world). I sympathize with the small HPC teams that just 
don't have the time nor flexibility to migrate. Although, I would 
HEAVILY suggest that they make the time to learn Singularity I don't 
expect them to make the transition to Kubernetes without some drastic 
changes.


I'm just curious what you are thinking about what it means to have a 
more sustainable footing within these clusters and what we as a 
community can do to lead the way such that in the next decades it 
matters less what OS is running on the hardware of these long term 
science HPC clusters.


~Stack~

Re: Sustainable computing - Re: CentOS EOL - politics?

[Yasha Karant]

I am familiar with Kubernetes that initiated though Google engineering 
staff as I recall.  For those who are quite unfamiliar with Kubernetes, 
a brief overview with references is


https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Kubernetes&d=DwICaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=yzqqasnYexgXlPFBSgopwWoJDjJYzXUUe22jj2Ohdcs&s=_qp_loeNCERXODusF1gKpZeOqLVKfxcFppY5NEX49y0&e= 


We looked at Singularity; however, as overviewed in

https://urldefense.proofpoint.com/v2/url?u=https-3A__sc19.supercomputing.org_proceedings_bof_bof-5Fpages_bof187.html&d=DwICaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=yzqqasnYexgXlPFBSgopwWoJDjJYzXUUe22jj2Ohdcs&s=gq7xP-_56U5XegnswNWIqHt8qpOGx5e657Vgzh_tbok&e= 


Containers in HPC, Younge, Canon, and Newburn

at which time there were at least Charliecloud, Docker, Kubernetes, 
Podman, Shifter, and Singularity under consideration.


As the HPC community had not "stabilized" on one of these, and as these 
are not truly interoperable, we elected to wait and see.  (Have the CERN 
HPC collaborations now selected one of these?)


Although it is true that Ubuntu LTS, EL, etc., are not "identical", 
these each are based upon both Linux and GPL internals/applications, and 
once configured, are not all that different -- much less so from what I 
recall of the various containers.


On 12/12/20 6:42 AM, ~Stack~ wrote:

On 12/11/20 10:09 AM, Brett Viren wrote:

My hope is they (we) take this current situation as a lesson and make a
radical change that puts all of our computing on more sustainable
footing as we go into the next decades.


I'm curious about your thoughts on what it means to have that 
sustainable footing going forward.


We have been pushing our users to Singularity images for the last two 
years (we jumped on pretty early). A LOT of our application/code base is 
already Singularity behind the scenes. The users don't know and don't 
care because their applications still run the same on the same HPC 
equipment. However, getting our users to purposefully think in terms of 
Singularity images has been a long hard road and we still have so much 
further to go.


We are on the edge of shifting a few very critical and heavy 
computations to Kubernetes. I'm not yet convinced that it will replace a 
lot of the hard-core traditional HPC workloads anytime soon, but there 
are a surprising amount of workloads that can. Plus, it allows us to 
automate from Code->Gitlab->CI/CD->Kubernetes->results delightfully well.


But one of the absolute greatest things about it from the perspective of 
what CentOS just pulled is that my dev Kubernetes has three OS's. SL7, 
Ubuntu 20.04, and CentOS 8 (I JUST spun this up the Monday before the 
announcement). As an admin, I _don't_ care about the OS at this point of 
the Kubernetes process. I kill a node and rebuild it to anything that 
supports the docker requirements (plus a few other things I need for 
company audit/security) and join it to the cluster. Done! When I killed 
that CentOS 8 node I suffered no loss in the slightest in terms of 
functionality and only about an hour of time where I had to move the 
workload and rebuild the node Ubuntu.


Bigger shops with decent sized teams, these transitions can be done over 
time. But the vast majority of my career I've supported hundreds of 
compute nodes where the entire HPC team was just me plus my manager and 
we had to support the clusters for 5-8 years (especially when I was in 
the university world). I sympathize with the small HPC teams that just 
don't have the time nor flexibility to migrate. Although, I would 
HEAVILY suggest that they make the time to learn Singularity I don't 
expect them to make the transition to Kubernetes without some drastic 
changes.


I'm just curious what you are thinking about what it means to have a 
more sustainable footing within these clusters and what we as a 
community can do to lead the way such that in the next decades it 
matters less what OS is running on the hardware of these long term 
science HPC clusters.


~Stack~

Re: Sustainable computing - Re: CentOS EOL - politics?

[Brett Viren]

Hi,

~Stack~  writes:

> I'm curious about your thoughts on what it means to have that
> sustainable footing going forward.

A little bit pontificating but here is my take: "sustainable computing"
must be "community all the way down".  We must reject attempts by
flighty (or other) corporations to inject profit-motivated gatekeeping.
Embrace DIY and do-in-house expertise and reject outsourcing and brain
drain.  Prefer GPL and the AGPL licenses over MIT/BSD and certainly not
proprietary for our own software and the software we base it on.
Embrace decentralized distribution patterns for code, data and
human-to-human information and reject centralized "cloud" services.
Keep discussions (like these) on open mailing lists and out of locked up
web forums.

Of course, we may soften from this hard stance and still obtain some
measure of sustainability but must then accept an increased risk of
eventual upheaval.  The fact that we got as far as we did with RH shows
this trade off in action.  Maybe Rocky gives us another decade or so
until the cycle that CentOS started repeats.  Or, maybe its future
leadership never allow themselves to be bought out and the project
perverted.  But, even so, Rocky is not based on the effort of a
community but that of a corporation and that corporation can do other
things to strangle Rocky.  As good intentions as Rocky may have, it
isn't Debian in this regard.

> [Singularity]

Singularity and container technology in general have many benefits but
on the scale of decades, I don't see that it solves "sustainability".

It does have at least two things to offer in that direction:

A container can provide an important ingredient in a "data preservation"
effort to archive the run time environment associated with some past
data/results.

Post-2024, one may consider to run SL7 guest on a, say, CentOS 8 Stream
host.  This would give app-level stability while (maybe) still
satisfying host-level security requirements.  As time goes by, this
"solution" gets more and more insecure.


Of course, Singularity also has many other benefits and the lack of
addressing sustainability doesn't stop me from making good use of it for
other purposes.

> [Kubernetes]

I still fail to grok kubernetes so have no comment.

-Brett.




signature.asc
Description: PGP signature