I agree with Carlos here – what you’re basically doing is going back to the “monolithic build” with carefully curated libraries. Is that not what APIs and shareable libraries were supposed to get away from. Just because we can now ship 100GB images around conveniently doesn’t mean it’s a good idea.
Sure, I get it as a practical thing. The nirvana of infinitely multiversion intercompatible libraries is probably never to exist – and users “just want it to run” so the safest strategy is send “this image works on this model machine, buy that model, and we’ll support it”. And if you’re a big enough dog, you can tell people that and they will buy your product – because they have to (especially if you’ve managed to do some “regulatory capture” a’la Ma Bell) And this is not really a HPC thing – it’s everywhere – Trying to set up a build environment for an embedded processor or FPGA is like this. In theory, you *could* download all the pieces, figure out the compile switches and symbols that need to be set (because, sure, the autoconfigure always works perfectly), recompile and build. Or, you can download the “known to work set of executables for Ubuntu 18.04 LTS” and get on with your life. (and set up multiple boot environments, because, yeah, you still need to support that system that was built with 16.04 LTS 4 years ago) And I certainly don’t want to go back to the early-mid 2000s where “which version of glibc” was the question of the day when doing builds. From: Beowulf <[email protected]> on behalf of Carlos Bederián <[email protected]> Date: Saturday, December 12, 2020 at 12:36 AM To: Douglas Eadline <[email protected]> Cc: "[email protected]" <[email protected]>, Chris Samuel <[email protected]> Subject: [EXTERNAL] Re: [Beowulf] RIP CentOS 8 I'm going off on a tangent here, but how is shipping an entire distribution for a single application something good? Many things have failed for us to get to this point where such a brute force approach makes sense, and nobody wants to tackle the underlying problems. On Fri, Dec 11, 2020, 3:57 PM Douglas Eadline <[email protected]<mailto:[email protected]>> wrote: Now jump ahead to containers and HPCng (https://hpcng.org/<https://urldefense.us/v3/__https:/hpcng.org/__;!!PvBDto6Hs4WbVuu7!aUR5FTCW7oK-aG1cotqvcL7oXai5SsalvzOMh5rl-4Yi0-tY3-RkXWe3ebpg0t0-b-FFEl4$>) An open source project will release a container that "contains" everything thing it needs to run (along with the container recipe) Using Singularity you can also sign the container to assure provenance of the code. The scheduler runs containers. Simple. Software Vendors will gladly do the same. Trying to support multiple distribution goes away. Applications show up in tested containers. The scheduler runs containers. Things just work, less support issues for the vendor. Simple. The need to maintain library version trees and Modules for goes away, Of course if are developer writing your own application, you need specific libraries, but not system wide. Build the application in your working directly, include any specific libraries you need in the local source tree and fold it all into a container. Joe Landman also comments on this topic in his blog (does not seem to be showing up for me today, however) https://scalability.org/2020/12/the-future-of-linux-distributions-in-the-age-of-docker-and-k8s/<https://urldefense.us/v3/__https:/scalability.org/2020/12/the-future-of-linux-distributions-in-the-age-of-docker-and-k8s/__;!!PvBDto6Hs4WbVuu7!aUR5FTCW7oK-aG1cotqvcL7oXai5SsalvzOMh5rl-4Yi0-tY3-RkXWe3ebpg0t0-K5VkGvo$> Bottom line, it is all good, we are moving on. -- Doug
_______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
