Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
frederik.pfautsch: > > So, ideally, there is c): In a hypothetic case we would prepare > > a entire OS incl. our app (maybe via catalyst?) which would > > require a bootloader-like mini-OS on the customer's side, to > > receive updates over the internet, switch the OS at boot time, > > and fallback. I was recently playing with systemd-boot and it's > > interesting try-boot feature. > > So essentially it sounds like you want something similar to Yocto / Poky > / Petalinux for the non-embedded world (and based on Gentoo of course; > it sounds like catalyst is something like that)? I've had a look on that: Wow, another interesting approach to build customized OSes. Thanks! > Just throwing crazy ideas around, what about using net-boot for your > customer? This way they just need to store an image somewhere and can > update it whenever necessary. Or (ab-)using an initramfs. Or u-boot > bootloader, just like in the embedded world. Depending on the size of > the actual OS/rootfs, taking ideas from e.g. Android with their A/B > bootslots (i.e. two root-partitions or something, where one is active > and the other can be updated with clever scripts, after a reboot they > are swapped). ... exactly what is on my wishlist currently! I am missing such an alternative when in need for updating a remote (customer's) OS, where ssh + emerge @world is just no option. If we had that, I'd see Gentoo (e.g. with catalyst or via Yocto) shining bright here as it is perfect in stipping down things to the required minimum. Thanks.
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
peter: > > Whenever we need to deliver a updated app to customers whose OS is > > too old (but updating it is too risky), we could either > > a) keep evenly outdated dev build OSes around forever (oh no!), or > > b) post our newly built app in a container (leaving the lovely native > > world); and both ignore the fact that customers wish maintenance of > > the entire OS actually, too. > > So, ideally, there is c): In a hypothetic case we would prepare a > > entire OS incl. our app (maybe via catalyst?) which would require > > a bootloader-like mini-OS on the customer's side, to receive updates > > over the internet, switch the OS at boot time, and fallback. > > I had a) in mind. Why "oh no!" ? I didn't mean forever. Why "Oh no": Well, it works and we are currently going that way. But see the other discussion branch in this thread: It's just a growing maintainment mess over time. In short: Too many OS versions resp. according dev VMs to rebuild new apps against it. > catalyst building specific, "old-like" OSes on which you build new (binpkg) > versions of your app to be installable by emerge on old customer OSes is > admittedly a lot of work, at least initially. Yes, that brings in another approach: *If* we had catalyst controlled versions of the (already...) delivered OSes, we could maybe setup a CI/CD like pipeline to re-create the according developer VM (for building a updated app with the exact old dependencies as on the formerly delivered old customer OS). So to say: VM just in time. Well I'll keep that in mind. That may help against the VM mess (see above). But building exact (older) OSes still requires a solution to distribute the entire OS incl. app remotely and savely. See also the other discussion branch on that if you like. Thanks
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
peter: > Peter Stuge wrote: > > Essentially you will be maintaining a private fork of gentoo.git, > > If this seems too heavy handed then you can just as well do the reverse: > > Maintain an overlay repo with the packages you care to control in the > state you care to have them, set that in the catalyst stage4.spec > portage_overlay and add unwanted package versions in gentoo.git to > the package.mask directory used by catalyst. > > This may sound complicated but it isn't bad at all. > > For total control also make your own profile, e.g. based on embedded, > but that's not per se neccessary, only if the standard profiles has too > much conflicts with what you want in @system. > > catalyst will rebuild @system according to spec file but with too much > difference that just becomes annoying and feels more trouble than a > controlled profile. > > This approach falls somewhere between your options (1) and (5). Wow, wasn't aware of catalyst at all. What a beast in terms of control. (FYI: I enjoyed the links on catalyst you sent me directly. Unfortunatelly I cannot answer you directly due to the default TLS guarantee kicked in by my provider: "TLS is required, but was not offered by host". That's usually to be fixed on the receiving side.) While being able to build exact environments with catalyst I wonder how it could help fixing the issue of my original post. To sum up: Whenever we need to deliver a updated app to customers whose OS is too old (but updating it is too risky), we could either a) keep evenly outdated dev build OSes around forever (oh no!), or b) post our newly built app in a container (leaving the lovely native world); and both ignore the fact that customers wish maintenance of the entire OS actually, too. So, ideally, there is c): In a hypothetic case we would prepare a entire OS incl. our app (maybe via catalyst?) which would require a bootloader-like mini-OS on the customer's side, to receive updates over the internet, switch the OS at boot time, and fallback. I was recently playing with systemd-boot and it's interesting try-boot feature. Thanks
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Many thanks for your detailed thoughs for sharing the rich experiences on this! See below: antarus: > On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. > > I wrote a very long reply, but I've removed most of it: I basically > have a few questions, and then some comments: > > I don't quite grasp your problem statement, so I will repeat what I > think it is and you can confirm / deny. > > - Your devs build using gentoo synced against some recent tree, they > have recent packages, and they build some software that you deploy to > prod. Yes. > - Your prod machines are running gentoo synced against some recent > tree, but not upgraded (maybe only glsa-check runs) and so they are > running 'old' packages because you are afraid to update them[0] Well, we did sync (without updading packages) in the past but today we even fear to sync against recent trees. Without going into details, as a rule of thumb, weekly or monthly sync + package updates work near to perfect. (It's cool to see what a good job emerge does on our own internal production systems.) Updating systems older than 12 months or so may, however, be a hugh task. And too risky for remote production systems of customers. > - Your software builds OK in dev, but when you deploy it in prod it > breaks, because prod is really old, and your developments are using > packages that are too new. Exactly. > My main feedback here is: > - Your "build" environment should be like prod. You said you didn't > want to build "developer VMs" but I am unsure why. For example I run > Ubuntu and I do all my gentoo development (admittedly very little > these days) >in a systemd-nspawn container, and I have a few shell scripts to > mount everything and set it up (so it has a tree snapshot, some git > repos, some writable space etc.) Okay, yes. That is way (1) I mentioned in my OP. It works indeed but has the mentioned drawbacks: VMs and maintenance pile up, and for each developer. And you don't know when there is the moment to create a new VM. But yes it seems to me one of the ways to go: *Before* creating a production system you need to freeze portage, create dev VMs, and prevent updates on the VMs, too. (Freezing aka not updating has many disadvantages, of course.) > - Your "prod" environment is too risky to upgrade, and you have > difficulty crafting builds that run in every prod environment. I think > this is fixable by making a build environment more like the prod > environment. > The challenge here is that if you have not done that (kept the > copies of ebuilds around, the distfiles, etc) it can be challenging to > "recreate" the existing older prod environments. > But if you do the above thing (where devs build in a container) > and you can make that container like the prod environments, then you > can enable devs to build for the prod environment (in a container on > their local machine) and get the outcome you want. Not sure I got your point here. But yes, it comes down to what was said above. > - Understand that not upgrading prod is like, to use a finance term, > picking up pennies in front of a steamroller. It's a great strategy, > but eventually you will actually *need* to upgrade something. Maybe > for a critical security issue, maybe for a feature. Having a build > environment that matches prod is good practice, you should do it, but > you should also really schedule maintenance for these prod nodes to > get them upgraded. (For physical machines, I've often seen businesses > just eat the risk and assume the machine will physically fail before > the steamroller comes, but this is less true with virtualized > environments that have longer real lifetimes.) Yes, haha, I agree. And yes, I totally ignored backporting security here, as well as the need that we might *require* a dependend package upgrade (e.g. to fix a known memory leak). I left that out for simlicity only. > > So, what we've thought of so far is: > > > > (1) Keeping outdated developer boxes around and compile there. We > > would freeze portage against accidental emerge sync by creating a > > git branch in /var/db/repos/gentoo. This feels hacky and requires a > > increating number of develper VMs. And sometimes we are hit by a > > silent incompatibility we were not aware of. > > In general when you build binaries for some target, you should build > on that target when possible. To me, this is the crux of your issue > (that you do not) and one of the main causes of you
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
sam: > > On 2 Jan 2023, at 12:48, m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. [...] > I'd really suggest just using stable in production and a mix > for developers so you can catch any problems beforehand. > > We try to be quite conservative about things like OpenSSL 3, > glibc updates, etc. Thanks, a misunderstanding then: I am talking about stable only. Whilst Gentoo may be conservative and all of you do an excellent job on keeping things smooth, incompatibilities of software being newly created on up-to-date developer systems (and then tried to be distributed to outdated production systems) may arise multiple times per year; *any* of the following alone may trigger a incompatibility. Just some random examples: (1) Most prominent, glibc updates. It has its sophisticated function versioning. You may or may not be hit. It is not depending on glibc's version but the internal function versions which are updated in a quite subtile way. (Function versioning is without doubt an impressive feature.) (2) libjpeg: In the past, libjpeg reached version 9 (like on Ubuntu today) but later was versioned 62 or 6.2 AFAIK. If you have an old Gentoo production system still on libjpeg-9, you have a hard time to update and distribute a new version of your app, as this version of libjpeg is not present in Gentoo anymore. (Don't get me wrong, there have probably been good reasons for this downgrade.) BTW: This little incompatibility is one of the reasons why it is hard to compile a app on Ubuntu for Gentoo. (3) An older one: libressl was removed. Well, we all remember the debate whether to keep it or not. (4) There is openssl-3 showing up on the horizon. I expect incompatibilities when distributing newly built software. (5) Portage EAPIs: If there is a new EAPI and you emerge --sync, then you need to update portage. This however, might require surprisingly many other updates. New python, setuptools and friends. I am not complaining here. Hey, we are on rolling release. Some of you may even know individual solutions to work around each of it. However, we just may get into trouble when distributing newly compiled apps (on new Gentoo systems) to older Gentoo systems. And we don't know in advance. I am looking for the best way to avoid that. Thanks
[gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi and happy new year. When we create apps on Gentoo they become easily incompatible for older Gentoo systems in production where unattended remote world updates are risky. This is due to new glibc, openssl-3 etc. So, what we've thought of so far is: (1) Keeping outdated developer boxes around and compile there. We would freeze portage against accidental emerge sync by creating a git branch in /var/db/repos/gentoo. This feels hacky and requires a increating number of develper VMs. And sometimes we are hit by a silent incompatibility we were not aware of. (2) Using Ubuntu LTS for production and Gentoo for development is hit by subtile libjpeg incompatibilites and such. (3) Distributing apps as VMs or docker: Even those tools advance and become incompatible, right? And not suitable when for smaller Arm devices. (4) Flatpak: No experience, does it work well? (5) Inventing a full fledged OTA Gentoo OS updater and distribute that together with the apps... Nah. Hm... Comments welcome. Thanks
Re: [gentoo-dev] Last rites: app-portage/flaggie
> > So I will deeply miss flaggie! Despite some issues it is a very > > helpful thing once you know it, e.g. for automated USE flag changes > > over ssh in a server farm. > > Hmm, that's a use case I didn't think of (and I didn't know anyone is > actually using flaggie non-interactively). I guess most of the time you > could get away with something like: > > echo app-foo/bar baz >> /etc/portage/package.use/local.conf > > etc. Yeah but... flaggie interprets all such files and it is able to keep them clean by just editing it at the right place (well, mostly) instead of having the same package being tweaked at mutiple places back and forth again. > To be honest, I'm thinking of starting from scratch like all cool > programmers do these days. Perhaps I should aim for a very simple tool > that doesn't try to represent all Portage configuration in AST > and instead just does plain manipulations. So perhaps they'll > be a flaggie2 after all. Oh, yes, please. "Just" something that does the monkey work of such flag manipulations. BTW: I'd really love to keep flaggie until there is your cool new flaggie2 and I'm very willing to ignore my bug report: https://bugs.gentoo.org/746542 ;-) Thanks
Re: [gentoo-dev] Last rites: app-portage/flaggie
ajak: > On Sun, Dec 25, 2022 at 06:02:11PM +0000, m1027 wrote: > > mgorny: > > > > > # Michał Górny (2022-12-25) > > > # make.conf writing is broken and package.use support incomplete. > > > # Last release in 2013. Attempted unsuccessfully fixing it in 2017. > > > # Use an editor instead. > > > # Removal on 2023-01-24. Bug #888423. > > > app-portage/flaggie > > > > So I will deeply miss flaggie! Despite some issues it is a very > > helpful thing once you know it, e.g. for automated USE flag changes > > over ssh in a server farm. > > If you're doing configuration management over ssh, you might want a > proper configuration management solution anyway, like ansible or > puppet. Yeah, I had a look on those some time ago. But how would you teach those tools to understand and manipulate USE flags properly? That's where flaggie is again required for AFAIK. Well, I never got a fan of ansible & co and instead have my bunch of bash scripts being sent/executed over ssh, opening tmux windows and such. Tested for up to 30 Gentoo boxes and that just WFM.
Re: [gentoo-dev] Last rites: app-portage/flaggie
mgorny: > # Michał Górny (2022-12-25) > # make.conf writing is broken and package.use support incomplete. > # Last release in 2013. Attempted unsuccessfully fixing it in 2017. > # Use an editor instead. > # Removal on 2023-01-24. Bug #888423. > app-portage/flaggie So I will deeply miss flaggie! Despite some issues it is a very helpful thing once you know it, e.g. for automated USE flag changes over ssh in a server farm. I am rather in C than in python so I cannot really help advancing flaggie. But I wonder whether there is a full specification for USE in make.conf as well as for package.use/* so there was a chance to think about a reimplementation somehow.
Re: [gentoo-dev] New project: binhost
dilfridge: > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, precompiled > packages for a subset of configurations, via central binary package hosting. > Currently we are still in the conceptual planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the wiki > page. > > Note that I see actually *building* the packages not as the central point of > the project (that could be e.g. a side effect of a tinderbox). I'm more > concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into production" > * ... > > Comments, ideas, flamebaits? :D What's with CFLAGS? I haven't been working with pre-built binary packages yet. But I imagine something flexible, between distant compiler and a server keeping the binaries. For my cases, from Raspberry Pis up to AMD Ryzens, I love to have this server to just receive the requests what and how to compile and keep exactly the packages which are requested so it can be distributed if requested again. Thanks
Re: [gentoo-dev] Make 'man' global USE flag from currently local
toralf: > On 2/13/21 12:16 PM, Alessandro Barbieri wrote: > > > > IMO manpages should be installed unconditionally and the useflag dropped > > I'd like to disagree - IIRC there're packages with a ridiculous large > and expensive dep tree just to build their man pages. Agreed. That's saying a man page fan. :-) There might be good resons for developers to not pre-build their man pages. At least, we cannot change that.
Re: [gentoo-dev] Packages up for grabs
mgorny: > [bv] www-apps/radicale I am actively using radicale on arm, arm64 and amd64 and thus feel like I should contribute. :-) As this was my debut to package maintainment, and I'd need at least some initial pointers on how to start, what's the least bothering way for everybody to ask for help and such. BTW: There is 2.x in portage but 3.x out, so there is not just a minor version bump to do. Thanks
Re: [gentoo-dev] [RFC] Discontinuing LibreSSL support?
mgorny: > On Tue, 2020-12-29 at 16:12 +0100, Toralf Förster wrote: > > On 12/29/20 2:57 PM, m1027 wrote: > > > - removing libressl, installing openssl, maybe wget then, followed > > > by the rest? > > remove is sufficient b/c emerge then immediately advices a > > @preserved-rebuild - at least that's the way it works here at the > > tinderbox (in the opposite direction FWIW) > > > > I'm not sure if you meant it but it reads as if you were talking about > removing the package. This is incorrect. > > You need to disable the USE flag and then --changed-use (or --newuse) > rebuild everything with the flag. If the depgraph is clean, emerge > should happily trigger the rebuild along with automatic replacement of > dev-libs/libressl with dev-libs/openssl. > > However, it's a good idea to run the same command with --fetchonly > first, to make sure that all distfiles are in place, in case wget got > broken in the process. It might not be the place to discuss emerge dependency details here, take it as some initial feedback on the transition from libressl to openssl. The general way to go seems indeed: - remove libressl from USE flags, also adjusting curl_ssl - initial emerge ... --fetchonly: to my surprise not always required - emerge -autDUN @world - finally the usual @preserved-rebuild - On some systems another @world update revealed again a lot - This also worked over ssh The systems I tried so far - 2x Gnome desktop systems, close to the USE defaults, went smoothly - 1x Raspberry Pi over ssh: still working, ;-) okay so far - 1x Developer system with some smaller issues The issues I had: - hostapd: when with +internal-tls, some build issue with libtommath; when with -internal-tls it required openssl -bindist; I did not investigate, just uninstalled hostapd yet - openssl+bind+openssh: conflict triggered to do +/-bindist for openssl; solution was -bindist everywhere (see other posts on bindist already) - old android-tools-6.0.1_p79: build issue mentioning ssl; not ivestigated further, just uninstalled Thanks
Re: [gentoo-dev] [RFC] Discontinuing LibreSSL support?
toralf: > On 12/29/20 2:57 PM, m1027 wrote: > > - removing libressl, installing openssl, maybe wget then, followed > >by the rest? > > remove is sufficient b/c emerge then immediately advices a > @preserved-rebuild - at least that's the way it works here at the > tinderbox (in the opposite direction FWIW) I gave it a shot and it did not work that way here. After uninstalling libressl and removing all according USE flags, @preserved-rebuild always tried to pull in libressl again. Maybe emerge -1 openssl, then emerge -auDUN @preserved-rebuild --nodeps helps.
Re: [gentoo-dev] [RFC] Discontinuing LibreSSL support?
> > On 29 Dec 2020, at 09:13, Marcel Schilling > > wrote: > > > > I just want to comment that I switched to LibreSSL on several > > Gentoo systems years ago and never had any major issues. I run > > both desktop and server systems with LibreSSL, based on X and > > Wayland. The only issues I ran into is a slight lag of the > > overlay behind the main tree so once in a while I had to mask a > > new version of some package for a week or so. Let me just come back on the different views here: @marcel: Exactly the same here. Smoothly running libressl on dozens of systems here, from embedded to ryzen servers, even on Gnome desktops. At least from the libressl *user's* perspective. sam: > TL;DR: [...libressl patches are...] just crippling functionality. @sam: From the perspective of libressl maintainers I have had a hard time reading this thread ;-) to learn that even security is supposed to be an issue with libressl today!? Aren't these crippling patches sometimes even helpful (see some apache patches) to crop unreliable extra features? I might be wrong here. Actually I'd prefer something 'boring' and stable on ssl over new features... Well, I cannot judge on the security issues in depth. From a short internet scan I don't see recent libressl issues but e.g. this one on openssl, https://www.openssl.org/news/vulnerabilities.html, only three weeks ago. Anyway, my personal conclusion on security: I've once switched to libressl because of the heartbleed issue. If security is better with openssl these days, I'd of course switch back. It might be worth having some warm explanations on the motivation in eselect NEWS, to help people out of the initial state of shock. > > So from a pure user perspective, thing change would mean a risky update > > to systems running stable for years with no gain whatsoever. Coming back on the technical way to switch back to openssl: Thanks to Gentoo, isn't the switch back more or less something predictable like - removing libressl USE / CURL flags - download everything before compiling (emerge -f ...) - removing libressl, installing openssl, maybe wget then, followed by the rest? It plead for something that actually *works* as many systems will need that change here. Thanks
Re: [gentoo-dev] [RFC] Discontinuing LibreSSL support?
I've been kindly asked by a gentoo dev to send my two pence in here: peter: > Michał Górny wrote: > > > LibreSSL users, does LibreSSL today have any benefit over OpenSSL? > > Yes, at least two: > > [...] > > B. It brings its own TLS API, a unique feature which by itself warrants > the package. Yeah, since openssl and libressl cannot be installed at the same time, I wonder what will be the future of libtls? To recall, it is a "a new TLS library, designed to make it easier to write foolproof applications" (see libressl.org). I've been using it for some time. It's great, and it is part of libressl. Another thing: Besides libressl there are boringssl and others. Even if still not the case (?), having virtual alternatives should in theory help keeping polished interfaces. If for whatever reason this not the case in practise, I believe dropping the alternatives should be worse. I cannot judge on the work the maintainers have to deal with compatibility issues between libressl and openssl, though. Let me know when I can help.
