Please review
https://issues.guix.gnu.org/45692
Re: Adding Substitute Mirrors page to installer
Hi zimoun, > > Any chance of this getting reviewed and merge within the next five years? > > I understand your frustration. Could you please point which patch number ? >From 41b174da1e38b71563405f1be48331fbe0e5700d Mon Sep 17 00:00:00 2001 From: raid5atemyhomework Date: Tue, 16 Mar 2021 23:45:37 +0800 Subject: [PATCH] gnu: Add substitute mirrors page to installer. * gnu/installer/services.scm (system-service) [snippet-type]: New field. (%system-services): Add substitute mirrors. (service-list-service?): New procedure. (modify-services-service?): New procedure. (system-services->configuration): Add support for services with `'modify-services` snippets. * gnu/installer/newt/services.scm (run-substitute-mirror-page): New procedure. (run-services-page): Call `run-substitute-mirror-page`. * gnu/services/base.scm (guix-shepherd-service)[start]: Accept second argument, a space-separated list of substitute URLs. * gnu/installer/final.scm (%user-modules): New variable. (read-operating-system): New procedure. (install-system): Read the installation configuration file and extract substitute URLs to pass to `guix-daemon` start action. * gnu/installer/tests.scm: Add new page in testing. --- gnu/installer/final.scm | 37 +++- gnu/installer/newt/services.scm | 26 +- gnu/installer/services.scm | 62 - gnu/installer/tests.scm | 12 +-- gnu/services/base.scm | 15 ++-- 5 files changed, 136 insertions(+), 16 deletions(-) diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm index fc0b7803fa..2324c960f2 100644 --- a/gnu/installer/final.scm +++ b/gnu/installer/final.scm @@ -22,9 +22,13 @@ #:use-module (gnu installer steps) #:use-module (gnu installer utils) #:use-module (gnu installer user) + #:use-module (gnu services) + #:use-module (gnu services base) #:use-module (gnu services herd) + #:use-module (gnu system) #:use-module (guix build syscalls) #:use-module (guix build utils) + #:use-module (guix ui) #:use-module (gnu build accounts) #:use-module (gnu build install) #:use-module (gnu build linux-container) @@ -38,6 +42,20 @@ #:use-module (ice-9 rdelim) #:export (install-system)) +;; XXX duplicated from guix/scripts/system.scm, but that pulls in +;; (guix store database), which requires guile-sqlite which is not +;; available in the installation environment. +(define %user-module + ;; Module in which the machine description file is loaded. + (make-user-module '((gnu system) + (gnu services) + (gnu system shadow + +(define (read-operating-system file) + "Read the operating-system declaration from FILE and return it." + (load* file %user-module)) +;; XXX + (define %seed (seed->random-state (logxor (getpid) (car (gettimeofday) @@ -174,6 +192,16 @@ or #f. Return #t on success and #f on failure." options (list (%installer-configuration-file) (%installer-target-dir + ;; Extract the substitute URLs of the user configuration. + (os (read-operating-system (%installer-configuration-file))) + (substitute-urls (and (operating-system? os) + (and=> (find +(lambda (service) + (eq? guix-service-type + (service-kind service))) +(operating-system-services os)) + (compose guix-configuration-substitute-urls + service-value (database-dir"/var/guix/db") (database-file (string-append database-dir "/db.sqlite")) (saved-database (string-append database-dir "/db.save")) @@ -206,8 +234,15 @@ or #f. Return #t on success and #f on failure." (lambda () ;; We need to drag the guix-daemon to the container MNT ;; namespace, so that it can operate on the cow-store. + ;; Also we need to change the substitute URLs to whatever + ;; the user selected during setup, so that the mirrors are + ;; used during install, not just after install. (stop-service 'guix-daemon) - (start-service 'guix-daemon (list (number->string (getpid + (start-service 'guix-daemon +`(,(number->string (getpid)) + ,@(if substitute-urls +`(,(string-join substitute-urls)) +'( (setvbuf (current-output-port) 'none) (setvbuf (current-error-port) 'none) diff
Re: Adding Substitute Mirrors page to installer
Any chance of this getting reviewed and merge within the next five years? Thanks raid5atemyhomework
Re: ZFS part of Guix? RFC?
Hello Denis, > While thinking about this very weird case of combining GPL and CDDL > code together, I wonder if the fact that we can't redistribute binaries > still makes it free software. AS I understood from early writings of GNU --- "Freedom" here is the freedom to modify how *the hardware you purchased, which is supposedly your own*, works. In principle, "binaries are enough", because you *can* modify executable binaries, and until Tivoization you could thus modify how the hardware you purchased and is supposedly owned by you works. However, early GNU writers (RMS I think?) noted that binaries are really awful and that you need source code in order to modify how your hardware work, unless you are willing to sacrifice a ridiculous portion of your life reverse-engineering executable binaries. As long as the source code is redistributable, users can modify the source code (without having to sacrifice too much of their limited lifetimes to the gods of extreme programming) and thereby modify how the hardware they purchased works. Thus, the essential freedom is preserved (users can fork ZFS and point their personal ZFS package definition at their fork), even if a legal snag prevents redistribution of binaries. I am well aware that *somebody* at Sun Microsystems screwed up by inventing the CDDL and putting the excellent ZFS under the CDDL and that is certainly a very cringey decision, but I want my hardware to work how I want it (i.e. without RAID5 write holes and memory buffer corruption bugs like in BTRFS "RAID5" mode) and not having ZFS on Guix is not helping that essential freedom. Thanks raid5atemyhomework
Re: [bug#50347] [RFC PATCH] lint: Warn about kernel modules with a suspect license
Hi list, I hope this topic gets attention. As far as I can tell, this is a blocker against 45692 getting merged. I am already annoyed enough with 45692 (it has been about half a year or so since submitted) and will not prioritize updating the patch until I can expect the CDDL-GPL incompatibility issue to be resolved somehow. Thank you to Maxime Devos for at least reviewing the final key patch to *start* ZFS support. I know a few people have asked me for help in getting ZFS working on *their* Guix systmes; I really hope they can speak up on public forums so that others know that ZFS support on Guix is desired by more than just me. Honestly, the only reason I use Guix is because this is the only FSF-recommended distribution with a recent ZFS available (I know how to code in Scheme, but only in anger; I prefer C++ or Haskell); but if Guix removes ZFS, I would rather install a non-FSF-recommended distribution than lose more of my homework to lousy RAID5 implementations. Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
BUMP
Re: Adding Substitute Mirrors page to installer
Hello Tobias, > > As an aside, I notice there is now a new "bayfront.guix.gnu.org" > > server. > > bayfront has been public sinc 2016..2017?, but it was recently exposed > through a second hostname, bordeaux.guix.gnu.org. That's the name that > should be used if added. > > > Not sure if I should modify this patch or not to add it or whatever. > > I don't see why not! It's the only Guix-hosted substitute server after > ci.guix.gnu.org, and has the advantage of not being a mere mirror. > If I add it, does this actually increase the chances the patch will get in or not? Note that the current Substitute Mirrors Page system allows to select only one set of substitutes. Mathieu was mentioning the ability to have a more complex Substitute Mirrors Page that allows to set up substitute servers in arbitrary order and so on, but I considered that it would be easier to implement a simple "just select one of these sets of substitutes" and hoped it would mean the patch gets in earlier rather than later. In any case, it looks to me that bordeaux is already in `%default-substitute-mirrors`, which this patch uses, so it should get included anyway as a fallback in case the SJTU mirror is not available or something. So maybe the patch is OK as-is? Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
bump. As an aside, I notice there is now a new "bayfront.guix.gnu.org" server. Not sure if I should modify this patch or not to add it or whatever. Thanks raid5atemyhomework > Hi Mathieu, > > I added the test as you recommended, however it looks to me that within the > test, reading the configuration file leads to an non-`operating-system?` > object, which seems strange. > > I was forced to modify the main code as below: > > - (substitute-urls (and (operating-system? os) > > > - (and=> (find > > > - (lambda (service) > > > -(eq? guix-service-type > > > - (service-kind service))) > > > - (operating-system-services os)) > > > -(compose > guix-configuration-substitute-urls > > > - service-value > > > > In previous version there was no`(and (operating-system? os) ...)` guard, as > we expect that the generated file, when `load`ed, will result in an > `operating-system`, so I wonder why the testing environment is somehow > different. Printing via `syslog "~A~%"` shows the `os` object as being some > `#`, not an `operating-system` object. > > Unfortunately I am not very familiar with any particular wrinkles regarding > Guile and its loading system, so I am unsure what is going wrong here... > > In any case I checked it manually by creating an installer image and then > installing into 2 different VMs, one with ci.guix.gnu.org and the other with > mirror.sjtu.edu.cn. > > Thanks > raid5atemyhomework > > From 41b174da1e38b71563405f1be48331fbe0e5700d Mon Sep 17 00:00:00 2001 > From: raid5atemyhomework raid5atemyhomew...@protonmail.com > Date: Tue, 16 Mar 2021 23:45:37 +0800 > Subject: [PATCH] gnu: Add substitute mirrors page to installer. > > - gnu/installer/services.scm (system-service) [snippet-type]: New field. > (%system-services): Add substitute mirrors. > (service-list-service?): New procedure. > (modify-services-service?): New procedure. > (system-services->configuration): Add support for services with > > > `'modify-services` snippets. > > - gnu/installer/newt/services.scm (run-substitute-mirror-page): New > procedure. > (run-services-page): Call `run-substitute-mirror-page`. > > - gnu/services/base.scm (guix-shepherd-service)[start]: Accept second > argument, a space-separated list of substitute URLs. > > - gnu/installer/final.scm (%user-modules): New variable. > (read-operating-system): New procedure. > (install-system): Read the installation configuration file and extract > substitute URLs to pass to `guix-daemon` start action. > > - gnu/installer/tests.scm: Add new page in testing. > > gnu/installer/final.scm | 37 +++- > gnu/installer/newt/services.scm | 26 +- > gnu/installer/services.scm | 62 - > gnu/installer/tests.scm | 12 +-- > gnu/services/base.scm | 15 ++-- > 5 files changed, 136 insertions(+), 16 deletions(-) > > diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm > index fc0b7803fa..2324c960f2 100644 > --- a/gnu/installer/final.scm > +++ b/gnu/installer/final.scm > @@ -22,9 +22,13 @@ > #:use-module (gnu installer steps) > #:use-module (gnu installer utils) > #:use-module (gnu installer user) > > - #:use-module (gnu services) > - #:use-module (gnu services base) > #:use-module (gnu services herd) > > - #:use-module (gnu system) > #:use-module (guix build syscalls) > #:use-module (guix build utils) > > - #:use-module (guix ui) > #:use-module (gnu build accounts) > #:use-module (gnu build install) > #:use-module (gnu build linux-container) > @@ -38,6 +42,20 @@ > #:use-module (ice-9 rdelim) > #:export (install-system)) > > +;; XXX duplicated from guix/scripts/system.scm, but that pulls in > +;; (guix store database), which requires guile-sqlite which is not > +;; available in the installation environment. > +(define %user-module > > - ;; Module in which the machine description file is loaded. > - (make-user-module '((gnu system) > -(gnu services) > > > -(gnu system shadow > > > - > > +(define (read-operating-system file) > > - "Read the operating-system declaration from FILE and return it." > - (load* file %
Re: Adding Substitute Mirrors page to installer
Hi Mathieu, I added the test as you recommended, **however** it looks to me that within the test, reading the configuration file leads to an non-`operating-system?` object, which seems strange. I was forced to modify the main code as below: + (substitute-urls (and (operating-system? os) + (and=> (find +(lambda (service) + (eq? guix-service-type + (service-kind service))) +(operating-system-services os)) + (compose guix-configuration-substitute-urls + service-value In previous version there was no`(and (operating-system? os) ...)` guard, as we expect that the generated file, when `load`ed, will result in an `operating-system`, so I wonder why the testing environment is somehow different. Printing via `syslog "~A~%"` shows the `os` object as being some `#`, not an `operating-system` object. Unfortunately I am not very familiar with any particular wrinkles regarding Guile and its loading system, so I am unsure what is going wrong here... In any case I checked it manually by creating an installer image and then installing into 2 different VMs, one with ci.guix.gnu.org and the other with mirror.sjtu.edu.cn. Thanks raid5atemyhomework >From 41b174da1e38b71563405f1be48331fbe0e5700d Mon Sep 17 00:00:00 2001 From: raid5atemyhomework Date: Tue, 16 Mar 2021 23:45:37 +0800 Subject: [PATCH] gnu: Add substitute mirrors page to installer. * gnu/installer/services.scm (system-service) [snippet-type]: New field. (%system-services): Add substitute mirrors. (service-list-service?): New procedure. (modify-services-service?): New procedure. (system-services->configuration): Add support for services with `'modify-services` snippets. * gnu/installer/newt/services.scm (run-substitute-mirror-page): New procedure. (run-services-page): Call `run-substitute-mirror-page`. * gnu/services/base.scm (guix-shepherd-service)[start]: Accept second argument, a space-separated list of substitute URLs. * gnu/installer/final.scm (%user-modules): New variable. (read-operating-system): New procedure. (install-system): Read the installation configuration file and extract substitute URLs to pass to `guix-daemon` start action. * gnu/installer/tests.scm: Add new page in testing. --- gnu/installer/final.scm | 37 +++- gnu/installer/newt/services.scm | 26 +- gnu/installer/services.scm | 62 - gnu/installer/tests.scm | 12 +-- gnu/services/base.scm | 15 ++-- 5 files changed, 136 insertions(+), 16 deletions(-) diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm index fc0b7803fa..2324c960f2 100644 --- a/gnu/installer/final.scm +++ b/gnu/installer/final.scm @@ -22,9 +22,13 @@ #:use-module (gnu installer steps) #:use-module (gnu installer utils) #:use-module (gnu installer user) + #:use-module (gnu services) + #:use-module (gnu services base) #:use-module (gnu services herd) + #:use-module (gnu system) #:use-module (guix build syscalls) #:use-module (guix build utils) + #:use-module (guix ui) #:use-module (gnu build accounts) #:use-module (gnu build install) #:use-module (gnu build linux-container) @@ -38,6 +42,20 @@ #:use-module (ice-9 rdelim) #:export (install-system)) +;; XXX duplicated from guix/scripts/system.scm, but that pulls in +;; (guix store database), which requires guile-sqlite which is not +;; available in the installation environment. +(define %user-module + ;; Module in which the machine description file is loaded. + (make-user-module '((gnu system) + (gnu services) + (gnu system shadow + +(define (read-operating-system file) + "Read the operating-system declaration from FILE and return it." + (load* file %user-module)) +;; XXX + (define %seed (seed->random-state (logxor (getpid) (car (gettimeofday) @@ -174,6 +192,16 @@ or #f. Return #t on success and #f on failure." options (list (%installer-configuration-file) (%installer-target-dir + ;; Extract the substitute URLs of the user configuration. + (os (read-operating-system (%installer-configuration-file))) + (substitute-urls (and (operating-system? os) + (and=> (find +(lambda (service) + (eq? guix-service-type + (service-kind service))) +(operating-system-services os)) +
Re: Adding Substitute Mirrors page to installer
Hello Maxime, > raid5atemyhomework schreef op ma 31-05-2021 om 09:42 [+]: > > > bump > > > > > bump > > To me it seems all questions have been addressed, > though I was not involved in the discussion. > So, maybe time to merge? > > See https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00294.html > for the patches. I am not really satisfied with the patch as-is, and I have this question: > For example, perhaps the result of a page is not an AST that is a field > of an `operating-system` form, but instead a procedure that accepts a > `(operating-system ...)` form and returns it. > Most existing pages would just append their keyed sub-form. > However the `mirrors` page would modify an existing `services` field in > the input `operating-system` form instead. Of course, implementing the above has some problems: * It is not a trivial change and it would take me a lot of time. * Nobody has responded to the above **prose** description of the idea, so I am very reluctant to invest in the time to implement the idea since nobody may end up reviewing the actual **code** implementation, which of course will be less easy to understand than the above prosody. I will not object if the existing patch is merged in, but I will not work on the above idea until somebody actually responds to it somehow. Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
ld modify an existing `services` field in > > the input `operating-system` form instead. > > > > > > > > +(define (run-substitute-mirror-page) > > > > > > > > - (let ((title (G_ "Substitute mirror"))) > > > > > > > > - (run-listbox-selection-page > > > > > > > > -#:title title > > > > > > > > > > > > -#:info-text (G_ "Choose a server to get substitutes from. > > > > > > > > > > > > - > > > > > > > > +Depending on your location, the official substitutes server can be > > > > slow; \ > > > > +in that case, using a mirror may be faster.") > > > > > > I wonder if it would make sense to select multiple mirrors, as fallback > > > if the preferred mirror is not up to date. We could also add the > > > possibility for the user to add a mirror manually. > > > In that case, the mirror page could look like the "User creation" page, > > > with an "Add" button opening a popup proposing to type a mirror URL or > > > to select one from an existing list. > > > WDYT? > > > > I think Less is More, and deploying an installer with a simple "select one > > mirror" pagenow will cover 90% of use-cases, and we can add the more > > complicated page later when there is more time. > > In particular, there is really only one public mirror of Guix that I know > > of, the SJTU mirror, so all the flexibility here is not very useful, at > > least not to me. > > Since "multiple mirrors" aren't even available yet anyway, why add the > > complication now? > > Maybe you can encourage more people to actually run mirrors if the > > installer has a visible "select mirrors" page, so that you can actually get > > more than just the SJTU mirror and "select multiple mirrors" is now a > > (good) problem to have to solve. > > For now I think a simple "select one mirror, we'll add ci.guix.gnu.org as a > > fallback" would be better. > > You can add a "select multiple mirrors in an order I want to specify" > > later, when there are multiple mirrors existing. > > Thanks > > raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
ng on your location, the official substitutes server can be slow; > > > \ > > > +in that case, using a mirror may be faster.") > > > > I wonder if it would make sense to select multiple mirrors, as fallback > > if the preferred mirror is not up to date. We could also add the > > possibility for the user to add a mirror manually. > > In that case, the mirror page could look like the "User creation" page, > > with an "Add" button opening a popup proposing to type a mirror URL or > > to select one from an existing list. > > WDYT? > > I think Less is More, and deploying an installer with a simple "select one > mirror" pagenow will cover 90% of use-cases, and we can add the more > complicated page later when there is more time. > > In particular, there is really only one public mirror of Guix that I know of, > the SJTU mirror, so all the flexibility here is not very useful, at least not > to me. > Since "multiple mirrors" aren't even available yet anyway, why add the > complication now? > > Maybe you can encourage more people to actually run mirrors if the installer > has a visible "select mirrors" page, so that you can actually get more than > just the SJTU mirror and "select multiple mirrors" is now a (good) problem to > have to solve. > > For now I think a simple "select one mirror, we'll add ci.guix.gnu.org as a > fallback" would be better. > You can add a "select multiple mirrors in an order I want to specify" later, > when there are multiple mirrors existing. > > Thanks > raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
a simple "select one mirror" page *now* will cover 90% of use-cases, and we can add the more complicated page later when there is more time. In particular, there is really only one public mirror of Guix that I know of, the SJTU mirror, so all the flexibility here is not very useful, at least not to me. Since "multiple mirrors" aren't even available yet anyway, why add the complication now? Maybe you can encourage more people to actually run mirrors if the installer has a visible "select mirrors" page, so that you can actually get more than just the SJTU mirror and "select multiple mirrors" is now a (good) problem to have to solve. For now I think a simple "select one mirror, we'll add ci.guix.gnu.org as a fallback" would be better. You can add a "select multiple mirrors in an order I want to specify" later, when there *are* multiple mirrors existing. Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
Bump Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, March 16, 2021 11:55 PM, raid5atemyhomework wrote: > Hi all, > > Below is the new patch version. > > In this version, the installer now also reads the generated > `operating-system` file to extract the `guix-configuration-substitute-urls`, > in order to pass it into the `start` action of `guix-daemon`. The `start` > action also now supports a second argument, the space-separated list of > substitute URLs. I'm wary of this technique as I feel it is unclean, but it > works and does not require significant changes to the existing software > architecture of the installer. > > Tested in this manner: > > - Created an installer image by `./pre-inst-env guix system image -t > iso9660 gnu/system/install.scm`. > - Started a new VM with the installer image and selected the SJTUG mirror. > - Confirmed that during installation the installer downloaded substitutes > from the SJTUG mirror. > - After installation completed on the VM, did a `guix pull` on the new VM > instance and confirmed it downloaded substitutes from the SJTUG mirror. > > I haven't tested for the use of the normal Berlin Cuirass, as that would > be ridiculously slow right now from my network, but I expect it would > continue to work. > > Thanks > raid5atemyhomework > > From 68a42cce2b4ae876cbbd1911aaa2a5bc8348bf15 Mon Sep 17 00:00:00 2001 > From: raid5atemyhomework raid5atemyhomew...@protonmail.com > > > Date: Tue, 16 Mar 2021 23:45:37 +0800 > Subject: [PATCH] gnu: Add substitute mirrors page to installer. > > - gnu/installer/services.scm (system-service) [snippet-type]: New field. > (%system-services): Add substitute mirrors. > (service-list-service?): New procedure. > (modify-services-service?): New procedure. > (system-services->configuration): Add support for services with > > > `'modify-services` snippets. > > - gnu/installer/newt/services.scm (run-substitute-mirror-page): New > procedure. > (run-services-page): Call `run-substitute-mirror-page`. > > - gnu/services/base.scm (guix-shepherd-service)[start]: Accept second > argument, a space-separated list of substitute URLs. > > - gnu/installer/final.scm (%user-modules): New variable. > (read-operating-system): New procedure. > (install-system): Read the installation configuration file and extract > substitute URLs to pass to `guix-daemon` start action. > > > gnu/installer/final.scm | 36 ++- > gnu/installer/newt/services.scm | 26 +- > gnu/installer/services.scm | 62 - > gnu/services/base.scm | 15 ++-- > 4 files changed, 125 insertions(+), 14 deletions(-) > > diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm > index fc0b7803fa..6eca3ec606 100644 > --- a/gnu/installer/final.scm > +++ b/gnu/installer/final.scm > @@ -22,9 +22,13 @@ > #:use-module (gnu installer steps) > #:use-module (gnu installer utils) > #:use-module (gnu installer user) > > - #:use-module (gnu services) > - #:use-module (gnu services base) > #:use-module (gnu services herd) > > - #:use-module (gnu system) > #:use-module (guix build syscalls) > #:use-module (guix build utils) > > - #:use-module (guix ui) > #:use-module (gnu build accounts) > #:use-module (gnu build install) > #:use-module (gnu build linux-container) > @@ -38,6 +42,20 @@ > #:use-module (ice-9 rdelim) > #:export (install-system)) > > +;; XXX duplicated from guix/scripts/system.scm, but that pulls in > +;; (guix store database), which requires guile-sqlite which is not > +;; available in the installation environment. > +(define %user-module > > - ;; Module in which the machine description file is loaded. > - (make-user-module '((gnu system) > -(gnu services) > > > -(gnu system shadow > > > - > > +(define (read-operating-system file) > > - "Read the operating-system declaration from FILE and return it." > - (load* file %user-module)) > +;; XXX > > - > > (define %seed > (seed->random-state > > (logxor (getpid) (car (gettimeofday) > > > @@ -174,6 +192,15 @@ or #f. Return #t on success and #f on failure." > options > (list (%installer-configuration-file) > (%installer-target-dir > > - ;; Extract the substitute URLs of the user configuration. > > > - (os (read-operating-system > (%installer-configuration-file))) > > >
Re: A Critique of Shepherd Design
> Loopback is handled by the ‘loopback’ shepherd service, which is > provided via ‘%base-services’. Perhaps you just need to have your > service depend on it? My service requires `tor`, which itself requires `loopback`, but it was still unable to access `127.0.0.1:9050` until I added a service that invokes `nm-online` and had my service require that. It works that way without problems, which leads me to conclude that loopback is *still* handled by NetworkManager somehow, despite what the Shepherd-level services claim. Unfortunately I cannot divulge what my service is; suffice it to say that naive code that just opens to `127.0.0.1:9050` failed when invoked before `nm-online` completes. Thanks raid5atemyhomework
Re: A Critique of Shepherd Design
Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, March 23, 2021 1:02 AM, Ludovic Courtès wrote: > Hi, > > raid5atemyhomework raid5atemyhomew...@protonmail.com skribis: > > > I'm not sure you can afford to keep it simple. > > It has limitations but it does the job—just like many other init systems > did the job before the advent of systemd. > > > Consider: https://issues.guix.gnu.org/47253 > > In that issue, the `networking` provision comes up potentially before the > > network is, in fact, up. This means that other daemons that require > > `networking` could potentially be started before the network connection is > > up. > > The ‘networking’ service is just here to say that some network interface > is up or will be up. It’s admittedly vague and weak, but it’s enough > for most daemons; they just need to be able to bind and listen to some > port. > > > One example of such a daemon is `transmission-daemon`. This daemon will > > bind itself to port 9091 so you can control it. Unfortunately, if it gets > > started while network is down, it will be unable to bind to 9091 (so you > > can't control it) but still keep running. On my system that means that on > > reboot I have to manually `sudo herd restart trannsmission-daemon`. > > Could you report a bug for this one? I don’t see why it’d fail to bind. Let me see if I can still get to the old syslogs where `transmission-daemon` claims it cannot bind, but it still keeps going anyway. I've pulled my `transmission-daemon` directly into my `configuration.scm` so I can edit its `requirement` to include a custom `networking-online` service that does `nm-online`. > > In another example, I have a custom daemon that I have set up to use > > the Tor proxy over 127.0.0.1:9050. It requires both `networking` and > > `tor`. When it starts after `networking` comes up but before the > > actual network does, it dies because it can't access the proxy at > > 127.0.0.1:9050 (apparently NetworkManager handles loopback as well). > > Loopback is handled by the ‘loopback’ shepherd service, which is > provided via ‘%base-services’. Perhaps you just need to have your > service depend on it? > > > Switching to a concurrent design for Shepherd --- any concurrent design --- > > is probably best done sooner rather than later, because it risks strongly > > affecting customized `configuration.scm`s like mine that have almost a half > > dozen custom Shepherd daemons. > > I suspect the main issue here is undeclared dependencies of some of the > Shepherd services you mention. > > I like the “sooner rather than later” bit, though: it sounds like you’re > about to send patches or announce some sponsorship program?… :-) Not particularly, but I *have* looked over Shepherd and here are some notes. Maybe I'll even send patches, but the reaction to the ZFS patches makes me just shrug; I'd need to devote more time than what I spent on ZFS, and the ZFS patches aren't getting into Guix, so why bother. If I get annoyed enough I'll just patch my own system and call it a day. My own system has `nm-online` and I don't expect to not have networking, so the `nm-online` delay is unlikely to be an issue, and I don't intend to mess with the `configuration.scm` anymore because it's just too brittle, I'll just host VMs instead and use a SystemD fully-free OS like Trisquel, I only need Guix for the ZFS (which Trisquel does not have, for some reason). Anyway... It seems to me that a good design to use would be that each `` should have its own process. Then the big loop in `modules/shepherd.scm` will then be "just" a completely event-based command dispatcher that forwards commands to the correct per-`` process. Now, one feature Shepherd has is that it can be set with `--socket-file=-`, which, if specified, causes GNU Shepherd to enable GNU readline and use the readline library to read `-herd`-like (?) commands. Unfortunately the `readline` interface is inherently blocking instead of event-based. The C interface of GNU readline has an alternative interface that is compatible with event-based (and I've used this in the past to create a toy chat program that would display messages from other users while you were typing your own) but it looks like this interface is not exposed. I checked `readline-port` as well, but the code I could find online suggests that this just uses the blocking `readline` interface, and would (?) be incompatible with the Guile `select`. (side note: the `SIGCHLD` problem could probably be fixed if Guile had `pselect` exposed, but apparently it's not exposed and I'm not prepared to dedicate even more time fiddling with the lack of syscalls in Guile. Maybe a signal-via-pipe technique would work as
Re: A Critique of Shepherd Design
Hello Ludo', > Hi, > > raid5atemyhomework raid5atemyhomew...@protonmail.com skribis: > > > Now, let us combine this with the second feature (really a bug): GNU > > shepherd is a simple, single-threaded Scheme program. That means that > > if the single thread enters an infinite loop (because of a Shepherd > > service description that entered an infinite loop), then Shepherd > > itself hangs. > > You’re right that it’s an issue; in practice, it’s okay because we pay > attention to the code we run there, but obviously, mistakes could lead > to the situation you describe. > > It’s a known problem and there are plans to address it, discussed on > this list a few times before. The Shepherd “recently” switched to > ‘signalfd’ for signal handling in the main loop, with an eye on making > the whole loop event-driven: > > https://issues.guix.gnu.org/41507 > > This will address this issue and unlock things like “socket activation”. > > That said, let’s not lie to ourselves: the Shepherd’s design is > simplistic. I think that’s okay though because there’s a way to address > the main issues while keeping it simple. I'm not sure you can afford to keep it simple. Consider: https://issues.guix.gnu.org/47253 In that issue, the `networking` provision comes up potentially *before* the network is, in fact, up. This means that other daemons that require `networking` could potentially be started before the network connection is up. One example of such a daemon is `transmission-daemon`. This daemon will bind itself to port 9091 so you can control it. Unfortunately, if it gets started while network is down, it will be unable to bind to 9091 (so you can't control it) but still keep running. On my system that means that on reboot I have to manually `sudo herd restart trannsmission-daemon`. In another example, I have a custom daemon that I have set up to use the Tor proxy over 127.0.0.1:9050. It requires both `networking` and `tor`. When it starts after `networking` comes up but before the actual network does, it dies because it can't access the proxy at 127.0.0.1:9050 (apparently NetworkManager handles loopback as well). Then shepherd respawns it, then it dies again (network still not up) enough times that it gets disabled. This means that on reboot I have to manually `sudo herd enable raid5atemyhomework-custom-daemon` and `sudo herd restart raid5atemyhomework-custom-daemon`. On SystemD-based systems, there's a `NetworkManager-network-online.service` which just calls `nm-online -s -q --timeout=30`. This delays network-requiring daemons until after the network is in fact actually up. However in https://issues.guix.gnu.org/47253#1 Mark points out this is undesirable in the Guix case since it could potentially stall the (single-threaded) bootup process for up to 30 seconds if the network is physically disconnected, a bad UX for desktop and laptop users (who might still want to run `transmission-daemon`, BTW) because it potentially blocks the initialization of X and make the computer unusable for such users for up to 30 seconds after boot. I note that I experienced such issues in some very old Ubuntu installations, as well. SystemD can afford to *always* have `nm-online -s -q --timeout=30` because it's concurrent. The `network-online.service` will block, but other services like X that don't ***need*** the network will continue booting. So the user can still get to a usable system even if the boot isn't complete because the network isn't up yet due to factors beyond the control of the operating system. Switching to a concurrent design for Shepherd --- *any* concurrent design --- is probably best done sooner rather than later, because it risks strongly affecting customized `configuration.scm`s like mine that have almost a half dozen custom Shepherd daemons. Thanks raid5atemyhomework
Re: A Critique of Shepherd Design
Good rmoning Mark, > Hi, > > raid5atemyhomework raid5atemyhomew...@protonmail.com writes: > > > GNU Shepherd is the `init` system used by GNU Guix. It features: > > > > - A rich full Scheme language to describe actions. > > - A simple core that is easy to maintain. > > > > However, in this critique, I contend that these features are bugs. > > The Shepherd language for describing actions on Shepherd daemons is a > > Turing-complete Guile language. Turing completeness runs afoul of the > > Principle of Least Power. In principle, all that actions have to do > > is invoke `exec`, `fork`, `kill`, and `waitpid` syscalls. > > These 4 calls are already enough to run "sleep 1000" and wait > for it to finish, or to rebuild your Guix system with an extra patch > added to glibc. I agree. But this mechanism is intended to avoid stupid mistakes like what I committed, not protect against an attacker who is capable of invoking `guix system reconfigure` on arbitrary Scheme code (and can easily wrap anything nefarious in any `unsafe-turing-complete` or `without-static-analysis` escape mechanism). Seatbelts, not steel walls. > > > Yet the language is a full Turing-complete language, including the > > major weakness of Turing-completeness: the inability to solve the > > halting problem. > > The fact that the halting problem is unsolved in the language means it > > is possible to trivially write an infinite loop in the language. In > > the context of an `init` system, the possibility of an infinite loop > > is dangerous, as it means the system may never complete bootup. > > Limiting ourselves to strictly total functions wouldn't help much here, > because for all practical purposes, computing 10^100 digits of Pi is > just as bad as an infinite loop. Indeed. Again, seatbelts, not steel walls. It's fairly difficult to commit a mistake that causes you to accidentally write a program that computes 10^100 digits of pi, not so difficult to have a brain fart and use `(- count 1)` instead of `(+ count 1)` because you were wondering idly whether an increment or a decrement loop would be more Scemey or if both are just as Schemey as the other. What I propose would protect against the latter (a much more likely mistake), as in-context the recursive loop would be flagged since the recursion would be flagged due to being a call to a function that is not a member of a whitelist. Hopefully getting recursive loops flagged would make the sysad writing `configuration.scm` look for the "proper" way to wait for an event to be true, and hopefully lead to them discovering the (hopefully extant) documentation on whatever domain-specific language we have for waiting for the event to be true instead of rolling their own. > That said, I certainly agree that Shepherd could use improvement, and > I'm glad that you've started this discussion. > > At a glance, your idea of having Shepherd do more within subprocesses > looks promising to me, although this is not my area of expertise. An issue here is that we sometimes pass data across Shepherd actions using environment variables, which do not cross process boundaries. Xref. the `set-http-proxy` of `guix-daemon`; the environment variable is used as a global namespace that is accessible from both the `set-http-proxy` and `start` actions. On the other hand, arguably the environment variable table is a global resource shared amongst multiple shepherd daemons. This technique in general may not scale well for large numbers of daemons; environment variable name conflicts may cause subtle problems later. I think it would be better if in addition to the "value" (typically the PID) each Shepherd service also had a `settings` (which can be used to contain anything that satisfies `(lambda (x) (equal? x (read (print x` so that it can be easily serialized across each subprocess launched by each action) that can be read and modified by each action. Then the `set-http-proxy` action would update this `settings` field for the shepherd service, then queue up a `restart` action. It could by convention be an association list. This would also persist the `http_proxy` setting, BTW --- currently if you `herd set-http-proxy guix-daemon ` and then `herd restart guix-daemon` later, the HTTP proxy is lost (since the environment variable is cleared after `set-http-proxy` restarts the `guix-daemon`). In short, this `set-http-proxy` example looks like a fairly brittle hack anyway, and maybe worth avoiding as a pattern. Then there's actions that invoke other actions. From a cursory glance at the Guix code it looks like only Ganeti and Guix-Daemon have actions that invoke actions, and they only invoke actions on their own Shepherd services. It seems to me safe for an action invoked
Re: A Critique of Shepherd Design
e to go import a bunch of things (that I have to go search for, and
I have to figure out Guile path settings for the bits that are provided by
Guix, etc.), and then afterwards I have to copy back the tested code into my
`configuration.scm`. This was enough of an annoyance to me that it discouraged
me from testing it, which could have caught the above bug.
Since `invoke` calls an external program, which runs isolated in its own
process (and can be `kill`ed in order to unstuck Shepherd), and can more easily
be independently tested outside the `configuration.scm`, it's substantially
safer than writing the same logic directly in the Shepherd language, so it gets
a pass.
Many other daemons that have some kind of "wait-for-daemon-startup" program
will often include tests for the "wait" program as well in its test suite, so
using `invoke` is substantially safer than writing bespoke code in Scheme that
performs the same action.
> > `mkdir-p` etc.
>
> I have vague plans to replace 'mkdir-p', 'mkdir-p/perms' etc. with some
> procedure
> (prepare-directory
> `("/var/stuff #:bits #o700 #:user ... #:group
> ("blabla" #:bits ...
> #:atomic-generate-contents ,(lambda (port) (display 'stuff port
> ...))
>
> ... automatically taking care of symlinks & detecting whether there are
> conflicts
> with different services.
>
> > Sub-forms (or the entire form for an action) can be wrapped in
> > `(unsafe-turing-complete ...)`
> > to skip this analysis for the sub-form,
>
> An escape hatch seems good to me, though I would rather call it
> `(without-static-analysis (totality) ...)'. The forms are not neccesarily
> ‘unsafe’,
> only potentially so. We could also define a checkers for other issues this
> way.
>
> (I'm thinking of potential TOCTTOU (time of check to time of use) problems
> involving
> symbolic links.)
>
> > but otherwise, by default, the specific subset must be used, and users have
> > to
> > explicitly put `(unsafe-turing-complete ...)` so they are aware that they
> > can
> > seriously break their system if they make a mistake here. Ideally, as much
> > of
> > the code for any Shepherd action should be outside an
> > ``unsafe-turing-complete`, and only parts of the code that really need the
> > full Guile language to implement should be rapped
> > in`unsafe-turing-complete`.
>
> I'm getting Rusty vibes here. Sounds sound to me.
>
> > (`lambda` and `let-syntax` and `let`, because they can rebind the meanings
> > of symbols,
> > would need to be in `unsafe-turing-complete` --- otherwise the analysis
> > routine would
> > require a full syntax expander as well)
>
> No. The analysis routine should not directly work on Scheme code, but rather
> on
> the 'tree-il' (or maybe on the CPS code, I dunno).
> Try (macro-expand '(let ((a 0)) a)) from a Guile REPL.
>
> > Turing-completeness is a weakness, not a strength, and restricting
> > languages to be
> > the Least Powerful necessary is better. The `unsafe-turing-complete` form
> > allows
> > an escape, but by default Shepherd code should be in the restricted
> > non-Turing-complete
> > subset, to reduce the scope for catastrophic mistakes.
>
> I'm assuming make-fork+exec-constructor/container would be defined in Scheme
> and be whitelisted in raid5atemyhomework-scheme?
Yes.
>
> - Section 5 -- Declarative
>
> Something you don't seem to have considered: defining services in a
> declarative
> language! Hypothetical example à la SystemD, written in something vaguely
> resembling Wisp (Wisp is an alternative read syntax for Scheme with less
> parentheses):
I prefer SRFI 110 myself.
> (I'm a bit rusty with the details on defining shepherd services
> in Guix:
>
> shepherd-service
> start `I-need-to-think-of-a-name-constructor #:binary #$(file-append
> package "/bin/stuff") #:arguments stuff #:umask suff #:user stuff #:group
> stuff ,@(if linux`(#:allowed-syscalls ',(x y z ...)))
> #:friendly-shutdown
> thunk
> invoke #$(file-append package "/bin/stop-stuff-please")
> #:polling-test-started?
> thunk
> if . file-exists? "/run/i-am-really-started"
> #true
>
> ;; tell I-need-to-think-of to wait 4 secs
> ;; before trying again
> seconds 4
> ;; how much time starting may take
> ;; until we consider things failed
> ;; & stop the process
> #:startup-timeout?
> thunk
> if spinning-disk?
> plenty
> little
> #:actions etcetera
> ...
>
> Well, that's technically still procedural, but this starts to look
> like a SystemD configuration file due to the many keyword arguments!
>
> (Here (thunk X ...) == (lambda () X ...), and
> I-need-to-think-of-a-name-constructor
> is a procedure with very many options, that
> ‘should be enough for everyone. If it doesn't support
> enough arguments,
> look in SystemD for inspiration.)
>
> That seems easier to analyse. Although it's bit kitchen-sinky,
> no kitchen is complete without a sink, so maybe that's ok ...
> (I assume the kitchen sink I-need-to-think-of-a-name-constructor
> would be defined in the guix source code.)
Well, yes but then we can start arguing that using SystemD would be
better, as it is more battle-tested and already exists and all the kitchen-sink
features are already there and a lot more people can hack its language than can
hack the specific dialect of scheme (augmented with Guixy things like
`make-forkexec-constructor`) used by GNU Shepherd.
Thanks
raid5atemyhomework
A Critique of Shepherd Design
e code that checked if the daemon initialization completed correctly. I think it is a common enough pattern that: * We have to spawn a daemon. * We have to wait for the daemon to be properly initialized (`#:post-fork`) * When shutting down the daemon, it's best to at least try to politely ask it to finish using whatever means the daemon has (`#:pre-kill`). * If the daemon doesn't exit soon after we politely asked it to, be less polite and start sending signals. So I think the above should cover a good number of necessities. Then, we can define a strict subset of Guile, containing a set of forms we know are total (i.e. we have a proof they will always halt for all inputs). Then any Shepherd actions, being Lisp code and therefore homoiconic, can be analyzed. Every list must have a `car` that is a symbol naming a syntax or procedure that is known to be safe --- `list`, `append`, `cons*`, `cons`, `string-append`, `string-join`, `quote` (which also prevents analysis of sub-lists), `and`, `or`, `begin`, thunk combinators, as well as the domain-specific `make-forkexec-constructor`, `make-kill-destructor`, `wait-for-condition`, `timed-action`, and probably some of the `(guix build utils)` like `invoke`, `invoke/quiet`, `mkdir-p` etc. Sub-forms (or the entire form for an action) can be wrapped in `(unsafe-turing-complete ...)` to skip this analysis for the sub-form, but otherwise, by default, the specific subset must be used, and users have to explicitly put `(unsafe-turing-complete ...)` so they are aware that they can seriously break their system if they make a mistake here. Ideally, as much of the code for any Shepherd action should be *outside* an ``unsafe-turing-complete`, and only parts of the code that really need the full Guile language to implement should be rapped in `unsafe-turing-complete`. (`lambda` and `let-syntax` and `let`, because they can rebind the meanings of symbols, would need to be in `unsafe-turing-complete` --- otherwise the analysis routine would require a full syntax expander as well) Turing-completeness is a weakness, not a strength, and restricting languages to be the Least Powerful necessary is better. The `unsafe-turing-complete` form allows an escape, but by default Shepherd code should be in the restricted non-Turing-complete subset, to reduce the scope for catastrophic mistakes. Thanks raid5atemyhomework
Making the Guix installer resilient against transient network issues
;; alive. ``` Notes: * `guix system build` only builds the *system*. It doesn't build the bootloader. I can't find a command that builds the bootloader; only `guix system init` or `guix system reconfigure` do that, but we need to differentiate between the failure "downloading from the substituter failed" (which might be fixable by just retrying) from "writing to the device being installed into failed". * In the above I use `guix build grub grub-efi` as a proxy for this, but it would be nice if there were some kind of `guix system build-bootloader` that would perform *building* of the script that installs the bootloader, but doesn't actually install the bootloader *yet*. * I don't know how best to ask the user if they want to retry the system building process. Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
Hi all, Below is the new patch version. In this version, the installer now also reads the generated `operating-system` file to extract the `guix-configuration-substitute-urls`, in order to pass it into the `start` action of `guix-daemon`. The `start` action also now supports a second argument, the space-separated list of substitute URLs. I'm wary of this technique as I feel it is unclean, but it works and does not require significant changes to the existing software architecture of the installer. Tested in this manner: * Created an installer image by `./pre-inst-env guix system image -t iso9660 gnu/system/install.scm`. * Started a new VM with the installer image and selected the SJTUG mirror. * Confirmed that during installation the installer downloaded substitutes from the SJTUG mirror. * After installation completed on the VM, did a `guix pull` on the new VM instance and confirmed it downloaded substitutes from the SJTUG mirror. I haven't tested for the use of the normal Berlin Cuirass, as that would be ridiculously slow right now from my network, but I expect it would continue to work. Thanks raid5atemyhomework >From 68a42cce2b4ae876cbbd1911aaa2a5bc8348bf15 Mon Sep 17 00:00:00 2001 From: raid5atemyhomework Date: Tue, 16 Mar 2021 23:45:37 +0800 Subject: [PATCH] gnu: Add substitute mirrors page to installer. * gnu/installer/services.scm (system-service) [snippet-type]: New field. (%system-services): Add substitute mirrors. (service-list-service?): New procedure. (modify-services-service?): New procedure. (system-services->configuration): Add support for services with `'modify-services` snippets. * gnu/installer/newt/services.scm (run-substitute-mirror-page): New procedure. (run-services-page): Call `run-substitute-mirror-page`. * gnu/services/base.scm (guix-shepherd-service)[start]: Accept second argument, a space-separated list of substitute URLs. * gnu/installer/final.scm (%user-modules): New variable. (read-operating-system): New procedure. (install-system): Read the installation configuration file and extract substitute URLs to pass to `guix-daemon` start action. --- gnu/installer/final.scm | 36 ++- gnu/installer/newt/services.scm | 26 +- gnu/installer/services.scm | 62 - gnu/services/base.scm | 15 ++-- 4 files changed, 125 insertions(+), 14 deletions(-) diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm index fc0b7803fa..6eca3ec606 100644 --- a/gnu/installer/final.scm +++ b/gnu/installer/final.scm @@ -22,9 +22,13 @@ #:use-module (gnu installer steps) #:use-module (gnu installer utils) #:use-module (gnu installer user) + #:use-module (gnu services) + #:use-module (gnu services base) #:use-module (gnu services herd) + #:use-module (gnu system) #:use-module (guix build syscalls) #:use-module (guix build utils) + #:use-module (guix ui) #:use-module (gnu build accounts) #:use-module (gnu build install) #:use-module (gnu build linux-container) @@ -38,6 +42,20 @@ #:use-module (ice-9 rdelim) #:export (install-system)) +;; XXX duplicated from guix/scripts/system.scm, but that pulls in +;; (guix store database), which requires guile-sqlite which is not +;; available in the installation environment. +(define %user-module + ;; Module in which the machine description file is loaded. + (make-user-module '((gnu system) + (gnu services) + (gnu system shadow + +(define (read-operating-system file) + "Read the operating-system declaration from FILE and return it." + (load* file %user-module)) +;; XXX + (define %seed (seed->random-state (logxor (getpid) (car (gettimeofday) @@ -174,6 +192,15 @@ or #f. Return #t on success and #f on failure." options (list (%installer-configuration-file) (%installer-target-dir + ;; Extract the substitute URLs of the user configuration. + (os (read-operating-system (%installer-configuration-file))) + (substitute-urls (and=> (find + (lambda (service) + (eq? guix-service-type + (service-kind service))) + (operating-system-services os)) + (compose guix-configuration-substitute-urls + service-value))) (database-dir"/var/guix/db") (database-file (string-append database-dir "/db.sqlite")) (saved-database (string-append database-dir "/db.save")) @@ -206,8 +233,15 @@ or #f. Return #t on success and #f on failure." (lambda () ;; We need to drag the guix-daemon to the container MNT
Re: Adding Substitute Mirrors page to installer
> The ability to also use the same mirror during install rather than after > it would be very nice. After all, the guix daemon has to be restarted during > installation in the meantime anyway, so on restart it should be possible to > switch the `substitute-urls`. However the complications are: > > - The `(gnu installer service)` module inherently assumes that services are > completely orthogonal to everything else being configured in the > installation. I'm not sure what the best way to extract the substitute mirror > selection would be. > - The installation image has to do a local `guix system reconfigure` of > itself so that its shepherd points the guix daemon to a new mirror, so that > the guix daemon restart in `install-system` of `(gnu installer final)` will > refer to a new mirror. Another way to do this would be to add another argument to the `start` action of `guix-daemon`, in much the same way the installer passes in its pid so that the guix daemon can access the copy-on-write store on the destination. This argument would override the `--substitute-urls`. So in `(gnu services base)` the `guix-shepherd-service` procedure would have something like: (define substitute-urls (match args ((_ substitute-urls . __) substitute-urls) (else #$(string-join substitute-urls #; (fork-exec-command/container #; ... #:pid (match args ((pid . _) pid) (else (getpid))) #; ...) The question now is how does the `install-system` procedure in `(gnu installer final)` determine what substitute URL to pass into `(start-service 'guix-daemon (list (number->string (getpid)) <>))`? * Change the architecture of the installer somehow so that a single page can both manipulate the `services` field and also manipulate how `install-system` is invoked, and add a new argument to `install-system`. * Or have `install-system` `read` in the `(%installer-configuration-file)`, look for the `operating-system` form, then delve in its `services` field for a `substitute-urls` field. This feels fairly brittle but does take advantage of the homoiconicity of Scheme. Thanks raid5atemyhomework
Re: Adding Substitute Mirrors page to installer
Hello, Below I have a patch that adds a page for substitute mirrors. Limitation is that the substitute mirror is only used after installation completes. During installation the guix daemon still loads from the Berlin server. Also, channel is still the default Guix channel (which is fairly slow as well from some places). Testing done: * Create an install image by `./pre-inst-env guix system image -t iso9660 gnu/system/install.scm` on a patched Guix. * Create a new VM and install using the created install image. * Select the SJTU mirror. * Complete installation (also notice that during install, the mirror is *not* used, which could be confusing to users). * On installation completion, reboot VM, then `guix pull` on root. * Check that `guix pull` gets substitutes from SJTU mirror. The ability to also use the same mirror *during* install rather than after it would be very nice. After all, the guix daemon has to be restarted during installation in the meantime anyway, so on restart it should be possible to switch the `substitute-urls`. However the complications are: * The `(gnu installer service)` module inherently assumes that services are completely orthogonal to everything else being configured in the installation. I'm not sure what the best way to extract the substitute mirror selection would be. * The installation image has to do a local `guix system reconfigure` of itself so that its shepherd points the guix daemon to a new mirror, so that the guix daemon restart in `install-system` of `(gnu installer final)` will refer to a new mirror. > I agree that we need a convenient way to add mirrors, it can be critical > to users who get low throughput from Berlin. Indeed. > > To that I'd add the option to add channels straight from the installer. > Not sure it belongs to a separate change set, maybe we can hit two birds > we one stone here. If you mean mirrors of the official Guix channel, this would be nice. However, channels are not described in the `operating-system` declaration. Thus, we need to create channel by extra mechanism in installer. This can probably be done by hooking somehow into `install-final` as well, as it creates the `/mnt` mountpoint for installing. If you mean other non-Guix channels, the only channels I know of that are not Guix cannot be named here, so --- are there any channels that *can* be named in official documentation about Guix? Thanks raid5atemyhomework >From af7e4d1336ed9010a31011d2fbae2a27fdaca237 Mon Sep 17 00:00:00 2001 From: raid5atemyhomework Date: Wed, 10 Mar 2021 09:21:42 + Subject: [PATCH] gnu: Add substitute mirrors page to installer. * gnu/installer/services.scm (system-service) [snippet-type]: New field. (%system-services): Add substitute mirrors. (service-list-service?): New procedure. (modify-services-service?): New procedure. (system-services->configuration): Add support for services with `'modify-services` snippets. * gnu/installer/newt/services.scm (run-substitute-mirror-page): New procedure. (run-services-page): Call `run-substitute-mirror-page`. --- gnu/installer/newt/services.scm | 26 +- gnu/installer/services.scm | 62 - 2 files changed, 78 insertions(+), 10 deletions(-) diff --git a/gnu/installer/newt/services.scm b/gnu/installer/newt/services.scm index 74f28e41ba..0fd5d3f2de 100644 --- a/gnu/installer/newt/services.scm +++ b/gnu/installer/newt/services.scm @@ -92,6 +92,29 @@ client may be enough for a server.") (condition ())) +(define (run-substitute-mirror-page) + (let ((title (G_ "Substitute mirror"))) +(run-listbox-selection-page + #:title title + #:info-text (G_ "Choose a server to get substitutes from. + +Depending on your location, the official substitutes server can be slow; \ +in that case, using a mirror may be faster.") + #:info-textbox-width 70 + #:listbox-height 8 + #:listbox-items (filter (lambda (service) +(eq? 'substitute-mirror + (system-service-type service))) + %system-services) + #:listbox-item->text (compose G_ system-service-name) + #:sort-listbox-items? #f + #:button-text (G_ "Exit") + #:button-callback-procedure + (lambda _ +(raise + (condition +())) + (define (run-services-page) (let ((desktop (run-desktop-environments-cbt-page))) ;; When the user did not select any desktop services, and thus didn't get @@ -100,4 +123,5 @@ client may be enough for a server.") (run-networking-cbt-page) (if (null? desktop) (list (run-network-management-page)) -'() +'()) +(list (run-substitute-mirror-page) diff --git a/gnu/installer/services.scm b/gnu/installer/services.scm index e
Re: Adding Substitute Mirrors page to installer
BUMP > Hi Guix Developers, > > I want to add a page to the installer to allow selection of substitute > mirrors. In particular, from my ISP, the SJTU mirror is significantly faster > (1MB/s->5MB/s) than directly from `ci.guix.gnu.org` (4kB/s->40kB/s), so I > think putting that option to the installer would be beneficial to others in > similar circumstances. > > However, the current`(gnu installer services)` assumes that snippets are > added to a list of services. To implement substitute mirrors, I need to > insert a `modify-services` form with a `(guix-service-type config => > (guix-configuration (inherit config) (substitute-urls ))`. > > What I want to propose is something like this: > > - Add a `snippet-type` field to ``, default `'service-list`. > - The field can instead be set to `'modify-services` to mean that the > snippet is to be in a `modify-services` form. > - In `system-services->configuration`, filter services according to > `snippet-type` and put the snippets in the appropriate parts of the `service` > form. > > - Add entries for the SJTU mirror (with fallback to the original > `ci.guix.gnu.org`) and the default unmirrored `ci.guix.gnu.org`, with > `system-service-type` of `'substitute-urls`. > > - Then in `(gnu installer newt services)`, add a page for > `'substitute-urls`. > > What do you think? > > Thanks > raid5atemyhomework >
Re: ZFS on Guix, again
BUMP > Hi Ludo, > > > > I agree with 宋文武 regarding ‘file-system-service-type’. > > > raid5atemyhomework raid5atemyhomew...@protonmail.com skribis: > > > > > > > However, for the case where the user expects the "typical" ZFS style of > > > > managing file systems, we need to mount all the ZFS file systems and > > > > ensure that they aer all already mounted by the time `file-systems` > > > > Shepherd service is started. This means we need to be able to extend > > > > the `requirement` of the `file-systems` Shepherd service. And we need > > > > to do that without putting any extra `/etc/fstab` entries since for > > > > "typical" ZFS style of managing file systems, they are required to not > > > > be put in `/etc/fstab`. > > > > > > Looks like this fstab issue is the main reason why you felt the need to > > > define an extra service type. Why is it important that ZFS not be > > > listed in /etc/fstab? > > > > Because on all non-Guix operating systems, they aren't listed > > in`/etc/fstab`: > > > > - > > https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gaztn/index.html > > So what do we do here? > > - Force all ZFS filesystems to be declared `mountpoint=legacy` and be > written as `file-system` declarations in the `operating-system` (which will > eventually reach `/etc/fstab`). > - This is undesirable since ZFS users expect that setting up mount > points for ZPOOL and ZFS datasets are just handled by the same commands that > create the ZPOOL and ZFS dataset. This is in contrast with other file systems > where the creation of the filesystem is a separate step from adding its mount > point. > - If a ZFS filesystem is created or destroyed (for example I might want > to create a temporary filesystem to `zfs send` to in order to implement > defragmentation, or to recompress data if I forgot to set `compression=on`) > then the user has to edit the configuration file and then `guix system > reconfigure` in order to make the changes stick. Most ZFS users just create > and destroy ZFS datasets as part of maintenance. > - If Guix goes this way, most ZFS users (including me) will not > consider ZFS support on Guix to be anywhere near "serviceable". > - Hack a `fstab?` field in `file-system` forms. > - Arguably bad design. > - Just split up the Shepherd service into a > `file-systems-target-service-type` and have `file-systems-service-type` > extend it, like I already proposed before. > > Also how about`linux-loadable-modules-service-type`? Is the proposed design > okay? Do we really want to name it `linux-loadable-modules-service-type` in > contrast to the current `operating-system` field `kernel-loadable-modules`? > > Thanks > raid5atemyhomework > > Thanks > raid5atemyhomework
Adding Substitute Mirrors page to installer
Hi Guix Developers, I want to add a page to the installer to allow selection of substitute mirrors. In particular, from my ISP, the SJTU mirror is significantly faster (1MB/s->5MB/s) than directly from `ci.guix.gnu.org` (4kB/s->40kB/s), so I think putting that option to the installer would be beneficial to others in similar circumstances. However, the current `(gnu installer services)` assumes that snippets are added to a list of services. To implement substitute mirrors, I need to insert a `modify-services` form with a `(guix-service-type config => (guix-configuration (inherit config) (substitute-urls ))`. What I want to propose is something like this: * Add a `snippet-type` field to ``, default `'service-list`. * The field can instead be set to `'modify-services` to mean that the snippet is to be in a `modify-services` form. * In `system-services->configuration`, filter services according to `snippet-type` and put the snippets in the appropriate parts of the `service` form. * Add entries for the SJTU mirror (with fallback to the original `ci.guix.gnu.org`) and the default unmirrored `ci.guix.gnu.org`, with `system-service-type` of `'substitute-urls`. * Then in `(gnu installer newt services)`, add a page for `'substitute-urls`. What do you think? Thanks raid5atemyhomework
Re: ZFS on Guix, again
Hi Ludo, > > I agree with 宋文武 regarding ‘file-system-service-type’. > > raid5atemyhomework raid5atemyhomew...@protonmail.com skribis: > > > > > However, for the case where the user expects the "typical" ZFS style of > > > managing file systems, we need to mount all the ZFS file systems and > > > ensure that they aer all already mounted by the time `file-systems` > > > Shepherd service is started. This means we need to be able to extend the > > > `requirement` of the `file-systems` Shepherd service. And we need to do > > > that without putting any extra `/etc/fstab` entries since for "typical" > > > ZFS style of managing file systems, they are required to not be put in > > > `/etc/fstab`. > > > > Looks like this fstab issue is the main reason why you felt the need to > > define an extra service type. Why is it important that ZFS not be > > listed in /etc/fstab? > > Because on all non-Guix operating systems, they aren't listed in`/etc/fstab`: > > - > https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gaztn/index.html So what do we do here? * Force all ZFS filesystems to be declared `mountpoint=legacy` and be written as `file-system` declarations in the `operating-system` (which will eventually reach `/etc/fstab`). * This is undesirable since ZFS users expect that setting up mount points for ZPOOL and ZFS datasets are just handled by the same commands that create the ZPOOL and ZFS dataset. This is in contrast with other file systems where the creation of the filesystem is a separate step from adding its mount point. * If a ZFS filesystem is created or destroyed (for example I might want to create a temporary filesystem to `zfs send` to in order to implement defragmentation, or to recompress data if I forgot to set `compression=on`) then the user has to edit the configuration file and then `guix system reconfigure` in order to make the changes stick. Most ZFS users just create and destroy ZFS datasets as part of maintenance. * If Guix goes this way, most ZFS users (including me) will not consider ZFS support on Guix to be anywhere near "serviceable". * Hack a `fstab?` field in `file-system` forms. * Arguably bad design. * Just split up the Shepherd service into a `file-systems-target-service-type` and have `file-systems-service-type` extend it, like I already proposed before. --- Also how about `linux-loadable-modules-service-type`? Is the proposed design okay? Do we really want to name it `linux-loadable-modules-service-type` in contrast to the current `operating-system` field `kernel-loadable-modules`? Thanks raid5atemyhomework Thanks raid5atemyhomework
Re: ZFS on Guix, again
Hi Ludo' > Hi, > > Sorry for the delay; this isn’t as simple as it looks! > > I agree with 宋文武 regarding ‘file-system-service-type’. > > raid5atemyhomework raid5atemyhomew...@protonmail.com skribis: > > > However, for the case where the user expects the "typical" ZFS style of > > managing file systems, we need to mount all the ZFS file systems and ensure > > that they aer all already mounted by the time `file-systems` Shepherd > > service is started. This means we need to be able to extend the > > `requirement` of the `file-systems` Shepherd service. And we need to do > > that without putting any extra `/etc/fstab` entries since for "typical" ZFS > > style of managing file systems, they are required to not be put in > > `/etc/fstab`. > > Looks like this fstab issue is the main reason why you felt the need to > define an extra service type. Why is it important that ZFS not be > listed in /etc/fstab? Because on all non-Guix operating systems, they aren't listed in `/etc/fstab`: * https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gaztn/index.html What ZFS users expect is that you just do something as simple as this: # zpool create mypool raidz2 /dev/disk/by-id/ata-Generic_M0D3L_53R14LN0 /dev/disk/by-id/ata-Generic_M0D3L_53R14LN1 /dev/disk/by-id/ata-Generic_M0D3L_53R14LN2 /dev/disk/by-id/ata-Generic_M0D3L_53R14LN3 log mirror /dev/disk/by-id/ata-Generic_55DM0D3L_53R14LN0 /dev/disk/by-id/ata-Generic_55DM0D3L_53R14LN1 /dev/disk/by-id/ata-Generic_55DM0D3L_53R14LN2 And what happens is: * The pool `mypool` is created containing a RAIDZ-2 of the 4 HDDs listed, with a separate log device consisting of a mirror of 3 SSDs. * A filesystem `mypool` is created on the pool `mypool`. * The `mypool` filesystem is mounted on `/mypool`. * On all subsequent bootups, the `mypool` filesystem is mounted on `/mypool`. In ZFS you are expected to have dozens of filesystems. If you have a new application, the general expectation is that you create a new filesystem for it. In general you might have one pool, or maybe two or three, but you host most of your data in multiple filesystems on that same pool. So for example you might want to create a filesystem for videos, which are sequentially accessed and tend to be fairly large, so setting `recordsize=1M` makes sense (good for sequential access, not so much for random, and good for very large files measurable in dozens of megabytes). # zfs create -o recordsize=1M -o mountpoint=/home/raid5atemyhomework/Videos mypool/videos The above command does: * The filesystem `videos` is created on the pool `mypool`. * The `mypool/videos` filesystem is mounted on `/home/raid5atemyhomework/Videos`. * On all subsequent bootups, the filesystem is mounted on `/home/raid5atemyhomework/Videos`. Now I might also want to run say a PostgreSQL service. * PostgreSQL allocates in page sizes of 8k, so `recordsize=8k` is best. * PostgreSQL uses a journal, which has a different access pattern from the rest of the data. Journals are written sequentially and read sequentially, while the database itself is accessed randomly. * The data should have `logbias=throughput` to optimize and reduce use of the ZIL SLOG, to avoid "log on a log" slowdown effects. * The journal itself should continue to use the default "latency". So I would do: # zfs create -o recordsize=8k -o logbias=throughput -o mountpoint=/postgresql mypool/postgresql # zfs create -o logbias=latency -o mountpoint=/postgresql/pg_wal mypool/postgresql/pg_wal That means creating two filesystems for a single application, one for the PostgreSQL data, the other for the PostgreSQL journal. What the above examples show is: * The habit for a ZFS user is to create many filesystems. On my own homelab I have two filesystems (one for documents and code, one for videos and pictures) for data I manage myself, and I have two other filesystems for two different applications I am running as well. * Each filesystem has different tuning properties. On a server you might have a dozen or so ZFS filesystems for various applications you need to run. There are also many other tuning parameters to tweak. If done by `/etc/fstab` it would lead to a fairly large file. The base logic here is that `/etc/fstab` has to be stored on disk anyway, and ZFS can just store the same information on the disks it is managing directly. Then ZFS supports nice tabulated output of properties via `zfs list`: # zfs list -o name,recordsize,logbias,atime,relatime NAMERECSIZE LOGBIAS ATIME RELATIME hddpool128K latency offon hddpool/bitcoin128K latency offon hddpool/common 128K latency offon hddpool/lightning 64K latency offon hddpool/media1M latency o
ZFS on Guix, again
f the `file-systems` Shepherd service. And we need to do that without putting any extra `/etc/fstab` entries since for "typical" ZFS style of managing file systems, they are required to ***not*** be put in `/etc/fstab`. * We can just create a separate `file-systems-target-service-type` that always accepts (list of) symbols that the `file-systems` Shepherd service will `requirement`. Then `file-systems-service-type` can just extend that service type. This is what I already originally did. * 宋文武 proposed to instead make `file-systems-service-type` accept a heteregonous list of either symbols or `` records. * Ludo' ***agreed*** with this but then says that mixing symbols and `` records in the same list is bad design. So... this is confusing. There are two alternatives: * Go with what I already proposed which I think is more general-purpose and cleaner (there is a separate service type that accepts symbols, and a separate service type that accepts `` records, and the latter just extends the former). * Don't make a separate service type, but now we need to add some kind of `fstab?` field to `file-system` so that the ZFS shepherd service that mounts ZFS file systems will not be included in the `/etc/fstab`. I think overall that having lots of tiny service types that are then combined together fits the functional design of Guix better. So I would strongly propose my original design rather than hacks on top of `file-system-service-type`. Thanks raid5atemyhomework
Re: ZFS on Guix
Hello Danny,
> I just wanted to say that I'm not ignoring your patch, I'm just not qualified
> to review it. I hope someone steps up to it--otherwise I can't really tell
> whether (mbegin %state-monad...) inside a random service procedure is a good
> idea.
>
> Then again, provenance-service-type does it and there it seems to be fine...
For ***this*** very specific case it is because of a random weirdness of
`system-service-type`.
Specifically, the *value* of that service-type is an association list of
filenames and their contained store values. However, an ***extension*** of
that service-type must be a monadic action that results in an association list
of filename-contents.
Here is relevant code in `gnu/services`:
```scheme
(define (system-derivation entries mextensions)
"Return as a monadic value the derivation of the 'system' directory
containing the given entries."
(mlet %store-monad ((extensions (mapm/accumulate-builds identity
mextensions)))
(lower-object
(file-union "system"
(append entries (concatenate extensions))
(define system-service-type
;; This is the ultimate service type, the root of the service DAG. The
;; service of this type is extended by monadic name/item pairs. These items
;; end up in the "system directory" as returned by
;; 'operating-system-derivation'.
(service-type (name 'system)
(extensions '())
(compose identity)
(extend system-derivation)
(description
"Build the operating system top-level directory, which in
turn refers to everything the operating system needs: its kernel, initrd,
system profile, boot script, and so on.")))
```
So *extensions* must be monads (due to `mapm/accumulate-builds` on the
`mextensions`) but the raw value must be a simple non-monadic assoc list.
The patch moves some generated files ("kernel" and "hurd") from the value of
the `system-service-type` to an extension of `system-service-type`, thus the
extra `mbegin %store-monad`. It needs to be `%store-monad` since that is the
monad used by the `system-derivation` function.
See:
```scheme
(define (kernel-builder-configuration->system-entry config)
"Return the kernel and hurd entries of the 'system' directory."
(mbegin %store-monad
#;...))
#;...
(define kernel-loadable-module-service-type
(service-type (name 'kernel-loadable-modules)
(extensions
(list (service-extension system-service-type
kernel-builder-configuration->system-entry))) ;; <-- OVER HERE
(compose concatenate)
(extend kernel-builder-configuration-add-modules)
(description
"Register packages containing kernel-loadable modules and adds
them+to the system.")))
```
So it is not just some "random service procedure", it is because that is the
interface exposed by `system-service-type` for its extensions, extensions of
`system-service-type` have to yield a monadic action. `mbegin` is one of the
simpler monadic actions. It is also in the "correct place" as best as I can
tell, since only service types in `gnu/services.scm` dare to extend
`system-service-type`.
`provenance-service-type` does this as well because it *also* extends
`system-service-type`. This is basically done here simply because that is what
`system-service-type` expects.
Thanks
raid5atemyhomework
Re: ZFS on Guix
> * the shepherd services defined in `configuration.scm` > seem one-shot services to me, so maybe add '(one-shot? #t)'. I was wary of making the `zfs-scan` one-shot, since there is a dependent service on top of it. Not to mention restarting `zfs-scan` could be useful if for example you were doing hotswapping of devices. Maybe. > * in the 'stop' of zfs-automount, the code changes the working > directory. As it isn't restored afterwards, this doesn't seem > kosher to me. Or maybe this isn't an issue at all. The code is modeled after `file-system-shepherd-service` procedure in `gnu/services/base.scm`, which includes a `(chdir "/")` as well with the comment "Make sure PID 1 doesn't keep TARGET busy.". Since this also does filesystem mounting/unmounting, I judged it best to imitate the existing filesystem mounting/unmounting service. Thanks raid5atemyhomework
Re: ZFS on Guix
Hi guix-developers and users, Here are some notes I made about how to get `/` on ZFS, maybe someone else can think about it. --- Most importantly, it seems for this style we need to consider first `/boot` ***not*** on ZFS, and have `/` on ZFS. I presume grub has some way to read ZFS pools in order to get at the `/boot` if `/boot` is on ZFS, since `/`-on-ZFS for Ubuntu is able to put even `/boot` on ZFS, however note that even there the `/boot` is on a different pool from the `/`. When this works (*cough*) it should be possible to create a ZFS pool (with `mountpoint=legacy`) from a ZFS-enabled boot of Guix, then create a `configuration.scm` that we would then `sudo guix system init` onto a temporary mountpoint. I would strongly suggest putting `/boot` elsewhere though. Here are a bunch of things we need for `/` on ZFS: * ZFS module installing into `initrd`. * ZFS module loading while in `initrd` before the pivot to the "real" `/`. * ZFS `import` of the pool containing `/` mount. ## `initrd` module installation We need a facility to add modules to the `initrd` that are not part of the kernel. Currently `raw-initrd`/`base-initrd` will only get `.ko` files from the given `linux` package. We need to modify the `initrd` interface to add say a `#:module-packages` list of packages whose `.ko` files we will also add to the `initrd`. This translates to modifying `flat-linux-module-directory` in `gnu/system/linux-initrd.scm` to additionally accept a list of packages all of whose `.ko` and/or `.ko.gz` files will be added to the module directory. Then somewhere over in `raw-initrd` we would: (define kodir (flat-linux-module-directory linux linux-modules module-packages)) We also need an extensible service type that will eventually lead to adding new entries in the `#:module-packages`. We need a "root" `initrd` extensible service type that will construct the `initrd` via the `operating-system-initrd` field. This service type will accept additional arguments to pass to the `initrd` function. Then the `initrd-kernel-loadable-module-service-type` would accept lists of package specifications, then provide a `#:module-packages` argument to the root `initrd` service type. Rough sketch: (define initrd-arguments-service-type (service-type #;...)) (define initrd-kernel-loadable-module-service-type (service-type (name 'initrd-kernel-loadable-module-service-type) (extensions (list (service-extension initrd-arguments-service-type (lambda (module-packages) (if (null? module-packages) '() (list #:module-packages module-packages)) (compose concatenate) (extend append) (default-value '( ## `initrd` module explicit loading Normally modules are loaded "as needed" but ZFS needs to be explicitly loaded. The `base-initrd` already accepts a list of `linux-modules` to load, we just need some way to hook into adding `#:linux-modules`. This probably means modifying how `operating-system-initrd-file` in `(gnu system)` works (which would probably be needed by the `initrd-arguments-service-type` anyway). We could try hooking into the currently-deprecated `extra-modules` instead, here's a sketch: (define initrd-kernel-module-loader-service-type (service-type (name 'initrd-kernel-module-loader-service-type) (extensions (list (service-extension initrd-arguments-service-type (lambda (modules) (if (null? modules) '() (list #:extra-modules modules)) (compose concatenate) (extend append) (default-value '( ## `initrd` additional pre-mount actions Before a `/` on ZFS can be mounted, ZFS has to be told to scan for the ZFS pool containing the `/`. We could add yet another argument to `raw-initrd`/`base-initrd`, `#:premount-actions`, a list of gexpressions. Then the `#:pre-mount` argument to `boot-system` would become something like: #:pre-mount (lambda () (and #$@device-mapping-commands #$@premount-actions)) As usual a service type can be created which extends `initrd-arguments-service-type`. For root-on-ZFS specifically we would need to know the root pool and execute something like this: #~(begin (invoke/quiet #$(file-append zfs/static "/sbin/zpool") "-a" "-N" #$root-pool)) Then, "normal" root specification would be done, with the root pool name given as the device name of the `/` mountpoint, and with the ZFS mountpoint set to `legacy`.
Re: ZFS on Guix
Hi Joshua, > raid5atemyhomework raid5atemyhomew...@protonmail.com writes: > > > The patchset currently dying on issues.guix.gnu.org would provide a nice > > simple single-step way to enable very basic ZFS support on your Guix > > system. Until it gets merged, however, you can still enable very very basic > > ZFS support on your Guix system by following the below minimal guide. > > If this patch does not get merged into guix, then you could always add > it to the cookbook. :) It includes changes to how Guix builds the system, and requires code changes to Guix itself, not just to the `configuration.scm`. The code changes enable: * `/home` on ZFS. * Allowing you to put any filesystem inside a `zvol` and declare it in `file-system`. * Assurance that by the time `file-systems` is started, even ZFS automounts have been mounted; important if you are starting a Shepherd service that has been configured to access filesystems (i.e. all the important ones). The modifications in the previous email work only in `configuration.scm` but do not allow the above. I think `/home` on ZFS is an important target to have (it's what I wanted for my server) at least, but requires changes to how other services in Guix System are built anyway. Thanks raid5atemyhomework
Re: ZFS on Guix
ol.cache` file, because I'm not really certain whether it's kosher in Guix to have a file maintained by ZFS in the `/etc` dir hierarchy. This has a number of consequences: * `zdb` doesn't work because it looks for `/etc/zfs/zpool.cache`. Good luck trying to figure out why your pool is slow. * You can't practically have more than a few dozen disks in your system, because the above uses `zpool import -a` which will cause ZFS to scan all the disks at bootup which would probably be slow if you have lots of disks. * You can't practically use `zvol`s to back other filesystems in such a way that you can put them in a `file-system` declaration. Though why use any file-system other than ZFS amirite. You can still use `zvol`s to e.g. back a virtual machine that you launch manually when the system boot is finished, or from a Shepherd service that explicitly lists `user-processes` as a requirement. Though hmmm the above doesn't use `zvol_wait` anywhere... sigh. * There's no ZED. No automatic replacement of failing drives with a hot spare. No monitoring. You can probably try launching it in its own Shepherd service, but you need to figure out how to populate `/etc/zfs/zed/` yourself. If you do, you probably will not be doing it from the `configuration.scm` file meaning it'll be hard to replicate the setup elsewhere. Thanks raid5atemyhomework
Re: ZFS on Guix
Hi guix-developers, > Now, my understanding is that `/etc/` directory is recreated at each `guix > system reconfigure`. Thus, if ZFS maintains information in `/etc/zfs/` then > on a reconfigure the information is lost. > > If so --- for Guix, what should I use instead? > Okay, a `guix system reconfigure` doesn't remove files in `/etc`, at best I **think** it just copies/overwrites data, but if the `etc` directory of the built system doesn't have a file, it will not be overwritten if something else writes to `/etc`. Is my understanding correct? On the other hand --- is it "properly Guix" for a system component to write to a file in the `/etc` directory? Where should I put this kind of not-quite-configuration cache file? Thanks raid5atemyhomework
Re: ZFS on Guix
Hi guix-developers, I just found out that ZFS on Linux maintains a file `/etc/zfs/zpool.cache` which contains information on ZPOOLs (i.e. ZFS-managed RAID arrays). I just didn't encounter this file before on Guix because the file is created if and only if the directory `/etc/zfs/` exists, and that directory is not created by the ZFS installation process on Guix (because I'm the one making the ZFS installation process on Guix right now and I didn't know about this directory). Now, my understanding is that `/etc/` directory is recreated at each `guix system reconfigure`. Thus, if ZFS maintains information in `/etc/zfs/` then on a reconfigure the information is lost. If so --- for Guix, what should I use instead? As it happens, ZFS uses this directory for at least these things: * `/etc/zfs/zpool.cache` - a binary file containing information about what ZPOOLs were created on this system. The `zpool` command updates this file! * `/etc/zfs/zpool.d/` - a directory of scripts that can be used to extend the `zpool` command; the ZFS release normally installs a bunch of files it has in those directories. * `/etc/zfs/vdev_id.conf` - a sysad-managed configuration file that is used to indicate the paths. * `/etc/zfs/zed.d/` - a directory of scripts that are executed when particular events are detected by the ZFS Event Daemon (which I didn't know about, and which should also be installed and started as a Shepherd service as well). Sysads are supposed to link or copy files from `/libexec/zfs/zed.d/` (which are provided by the installation) to enable/disable particular zedlets, and can add other events here. The ZFS release normally links some of the files in `/libexec/zfs/zed.d/`. * `/etc/zfs/zfs-functions` - a shell file containing functions that some bits of ZFS `init` scripts / `initramfs` scripts use, and maybe more. I think Guix can safely ignore this, though somebody (most likely me) will have to read it through to understand how ZFS actually implements some of its magical abilities. Of these, many are stuff that we might plausibly generate by adding more fields in `zfs-configuration`, though the ZED and its zedlet system would bring in the possibility of writing Guile GEXP to create scripts for particular events. However, I'm not so sure about `zpool.cache` file. For one, it doesn't contain text data so it's not easy to generate it ourselves. On the other hand I've got ZFS working without it, by just using `zpool import -a` to have ZFS scan for all devices. The problem with this technique is if Guix is used on a system with several dozen or hundred devices in various ZPOOLs; this would slow down boot while ZFS is checking all arrays and figuring out which device goes into which array, whereas a `zpool.cache` file would let ZFS know about all pools created or imported on the system and would not need to scan their labels and reassemble their arrays and so on. Another is that for a complex enough setup, a storage device might be connected to the hardware by hostile parties (for example, by network-addressable block devices, or USB) that contains a ZPOOL with a `mountpoint=/gnu/store` (or other sensitive directory) property, which, if auto-imported via `zpool import -a`, could let particular subdirectories of the filesystem to be subverted. How best do I handle this? Thanks raid5atemyhomework
Re: ZFS on Guix
Latest patchset: https://issues.guix.gnu.org/45692
Re: ZFS on Guix
Hi guix-developers,
Another issue here is that ZFS prefers to not be managed via
`/etc/fstab`/`mount`/`umount`. Instead, at startup ZFS magically imports ZFS
pools and mounts ZFS datasets in the correct place, as configured in the
`mountpoint` properties of the dataset. This magic is actually implemented by
executing `zpool import -a` at startup after ZFS module loading. Note that
`zpool import -a` has to be executed always so that ZFS can detect its pools,
though the automatic mounting can be suppressed with `zpool import -a -N`.
(`systemd`-based distros have a number of `systemd` services that handle ZFS
importation by use of a `/etc/zfs/zpool.cache` file that ZFS maintains, and
then does `zfs mount -a` to do automounting, but I'll leave that for later.)
Now, properly speaking, the shepherd `file-systems` service should not be
started until we have actually performed ZFS automounting. This means that
`file-systems` has to have as a requirement the ZFS shepherd service that
implements the automounting. And of course it should *not* require that if ZFS
isn't installed in the system.
So, I made another patch that makes the `file-systems` shepherd service have an
extensible set of requirements, like `user-processes` does. It's below.
Actual `file-system`s declared in the operating system then make themselves
requirements of `file-systems`. Then, a ZFS automounting shepherd service can
be installed by the `zfs-service-type` and added as a requirement of the
`file-systems` shepherd service.
This is important since one possible use of ZFS is to have it provide the
`/home` filesystem. And the `/home` filesystem has to be mounted before
`user-homes` shepherd service starts. So the ZFS automounting shepherd service
has to be a requirement of `file-systems`, which is enabled by the below.
Please review!
>From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Wed, 6 Jan 2021 08:15:54 +0800
Subject: [PATCH] gnu: Make file-systems target extensible by services.
* gnu/services/base.scm (file-system-shepherd-services): Move file-systems
shepherd service to ...
(file-systems-target-shepherd-services): ... new procedure here.
(file-systems-target-service-type): New variable.
(file-system-service-type): Extend file-systems-target service to add each
file-system as a requirement.
* gnu/system.scm (operating-system-default-essential-services): Instantiate
file-systems-target-service-type.
(hurd-default-essential-services): Instantiate file-systems-target-service-type.
---
gnu/services/base.scm | 37 +++--
gnu/system.scm| 2 ++
2 files changed, 29 insertions(+), 10 deletions(-)
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..13cfb6a8a2 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen
;;; Copyright © 2020 Florian Pelz
;;; Copyright © 2020 Brice Waegeneire
+;;; Copyright © 2021 raid5atemyhomework
;;;
;;; This file is part of GNU Guix.
;;;
@@ -67,6 +68,7 @@
#:export (fstab-service-type
root-file-system-service
file-system-service-type
+file-systems-target-service-type
swap-service
host-name-service
console-keymap-service
@@ -362,18 +364,29 @@ FILE-SYSTEM."
(gnu system file-systems)
,@%default-modules)))
+(define (file-systems-target-shepherd-services requirements)
+ (list
+(shepherd-service
+ (provision '(file-systems))
+ (requirement (cons* 'root-file-system 'user-file-systems requirements))
+ (documentation "Target for all the initially-mounted file systems")
+ (start #~(const #t))
+ (stop #~(const #t)
+(define file-systems-target-service-type
+ (service-type
+(name 'file-systems)
+(extensions (list (service-extension shepherd-root-service-type
+
file-systems-target-shepherd-services)))
+(compose concatenate)
+(extend append)
+;; Extensions can add new values to this list.
+(default-value '())
+(description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
(define (file-system-shepherd-services file-systems)
"Return the list of Shepherd services for FILE-SYSTEMS."
(let* ((file-systems (filter file-system-mount? file-systems)))
-(define sink
- (shepherd-service
- (provision '(file-systems))
- (requirement (cons* 'root-file-system 'user-file-systems
- (map file-system->shepherd-service-name
-file-systems)))
- (documentation "Target for all the initially-mounted file systems")
- (start #~(const #t))
- (stop #~(const #f
(defi
Re: ZFS on Guix
Thank you Carlo! I've now tested the new tests I added in `gnu/tests/linux-modules.scm`, and the existing tests as well, and they all pass. Hope to get some review on that patch! Thanks raid5atemyhomework
Re: ZFS on Guix
Hi Carlo,
Thanks, I modified patch as below.
I also can't get `guix system vm` to run in my hacking env, so I couldn't make
a VM for testing like you did. I got this:
```
Formatting '/gnu/store/imdlq0cay61d2cw3199g04z9bp16qx8j-qemu-image', fmt=qcow2
cluster_size=65536 compression_type=zlib size=73400320 lazy_refcounts=off
refcount_bits=16
Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied
```
Any tips on getting `guix system vm` working? I've never used VMs before. My
dev env is a foreign distro with Guix installed on top.
The above issue also means I haven't actually run the tests I made in
`gnu/tests/linux-modules.scm`, because it requires building a VM as well.
However, I *did* do a bunch of `guix system build` tests:
* With the same `configuration.scm` that uses the legacy
`(kernel-loadable-modules ...)` declaration in `operating-system`, I tried both
with and without the patch.
* Without patch and with patch created builds with a different hash, however,
they linked to the same directories. `diff -r` did not find any differences.
* With the patch, I made two variants of `configuration.scm`, one that used the
legacy `(kernel-loadable-modules...)` and another that used a service that
extended `kernel-laodable-module-service-type`.
* Builds resulted in the exact same system with exact same hash.
>From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Allow services to install kernel-loadable modules.
* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" and "hurd".
(operating-system-default-essential-services): Instantiate
kernel-loadable-module-service.
(hurd-default-essential-services): Instantiate
kernel-loadable-module-service.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(kernel-loadable-module-service-type): New variable.
(kernel-loadable-module-service): New procedure.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document kernel-loadable-module-service-type.
---
doc/guix.texi | 6 +++
gnu/services.scm| 70
gnu/system.scm | 37 +
gnu/tests/linux-modules.scm | 81 -
4 files changed, 157 insertions(+), 37 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system
reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..d7332a46b2 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès
;;; Copyright © 2016 Chris Marusich
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen
+;;; Copyright © 2021 raid5atemyhomework
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -75,6 +78,7 @@
service-back-edges
instantiate-missing-services
fold-services
+kernel-loadable-module-service
service-error?
missing-value-service-error?
@@ -106,6 +110,7 @@
profile-service-type
firmware-service-type
gc-root-service-type
+kernel-loadable-module-service-type
%boot-service
%activation-service
@@ -864,6 +869,71 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '(
+;; Configuration for the kernel builder.
+(define-record-type*
kernel-builder-configuration
+ make-kernel-builder-configuration
+ kernel-builder-configuration?
+ this-kernel-builder-configuration
+
+ (kernel kernel-builder-configuration-kernel (default #f))
+ (hurd kernel-builder-c
Re: ZFS on Guix
Hi Carlo and guix-developers,
I have this patch below for creating a new
`kernel-loadable-module-service-type`, which can be extended by another service
to add kernel-loadable modules provided by packages, hope for a review.
>From 984602faba1e18b9eb64e62970147aab653d997f Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Add 'kernel-loadable-module-service-type' for services
to extend with kernel-loadable modules.
---
doc/guix.texi| 6 +
gnu/services.scm | 68
gnu/system.scm | 36 +++--
3 files changed, 83 insertions(+), 27 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system
reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..2c0bbf9725 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -33,6 +33,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -75,6 +77,7 @@
service-back-edges
instantiate-missing-services
fold-services
+kernel-loadable-module-service
service-error?
missing-value-service-error?
@@ -106,6 +109,7 @@
profile-service-type
firmware-service-type
gc-root-service-type
+kernel-loadable-module-service-type
%boot-service
%activation-service
@@ -864,6 +868,70 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '(
+;; Configuration for the kernel builder.
+;; Only used by the KERNEL-LOADABLE-MODULE-SERVICE-TYPE.
+(define-record-type*
kernel-builder-configuration
+ make-kernel-builder-configuration
+ kernel-builder-configuration?
+ this-kernel-builder-configuration
+
+ (kernel kernel-builder-configuration-kernel (default #f))
+ (hurd kernel-builder-configuration-hurd (default #f))
+ (modules kernel-builder-configuration-modules (default '(
+
+(define (package-for-kernel target-kernel module-package)
+ "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+ (package
+(inherit module-package)
+(arguments
+ (substitute-keyword-arguments (package-arguments module-package)
+ ((#:linux kernel #f)
+target-kernel)
+
+(define (kernel-builder-configuration->system-entry config)
+ "Return the kernel and hurd entries of the 'system' directory of OS."
+ (mbegin %store-monad
+(let* ((kernel (kernel-builder-configuration-kernel config))
+ (hurd(kernel-builder-configuration-hurd config))
+ (modules (kernel-builder-configuration-modules config))
+ (kernel (if hurd
+kernel
+(profile
+ (content (packages->manifest
+ (cons kernel
+ (map (lambda (module)
+(if (package? module)
+(package-for-kernel kernel
module)
+module))
+ modules
+ (hooks (list linux-module-database))
+ (return `(("kernel" ,kernel)
+,@(if hurd `(("hurd" ,hurd)) '()))
+(define (kernel-builder-configuration-add-modules config modules)
+ "Constructs a kernel builder configuration that has its modules extended."
+ (kernel-builder-configuration
+(inherit config)
+(modules (append (kernel-builder-configuration-modules config) modules
+
+(define kernel-loadable-module-service-type
+ (service-type (name 'kernel-loadable-modules)
+(extensions
+ (list (service-extension system-service-type
+
kernel-builder-configuration->system-entry)))
+(compose concatenate)
+(extend kernel-builder-configuration-add-modules)
+(description
+
Re: ZFS on Guix
Hi guix-developers, In https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00053.html Carlo mentioned to instead use the `service` system to modify all parts needed in the operating system to get ZFS installed, and noted as well that the only thing missing in the `service` system is the ability to push kernel modules into the operating system. Here's a sketch of an attempt to add that feature so we can conveniently add ZFS on Guix by just adding a `(service zfs-service-type...)`. First, we need a new `kernel-loadable-module-service-type` in `gnu/services.scm`: ```scheme (define kernel-loadable-module-service-type ;; A service to add the kernel modules of a package. (service-type (name 'kernel-loadable-module) (extensions '()) (compose concatenate) (extend append) (description "Register kernel modules that will be placed in a loadable place where the kernel can find them.") (default-value '( ``` Absolutely no idea if a `'()` extensions makes sense but *shrug*. Then, in `gnu/system.scm`, we create this new function: ```scheme (define (operating-system-all-kernel-loadable-modules os) ;; Gathers the kernel-loadable modules in the KERNEL-LOADABLE-FIELD ;; field of the OS, as well as those registered by services. (let ((service (fold-services (cons (service kernel-loadable-module-service-type '()) (operating-system-user-services os)) #:target-type kernel-loadable-module-service-type))) (append (service-value service) (operating-system-kernel-loadable-modules ``` Does that make sense? Am I misusing `fold-services` and `service-value` here? Onward, we modify as well the function `operating-system-directory-base`: ```scheme (define* (operating-system-directory-base-entries os) "Return the basic entries of the 'system' directory of OS for use as the value of the SYSTEM-SERVICE-TYPE service." (let* ((locale (operating-system-locale-directory os)) (kernel (operating-system-kernel os)) (hurd(operating-system-hurd os)) (modules (operating-system-all-kernel-loadable-modules os)) ;... ``` The above is the reason why we need to `cons` a synthetic `kernel-loadable-module-service-type` service, the function `operating-system-directory-base-entries` is used do create the default for `operating-system-essential-services`, and the `operating-sstem-services` is just the concatenation of the user and essential services. I think. Would this work? Thanks raid5atemyhomework
Re: A new paradigm for modifying operating system declarations
Hi Carlo,
> In principle, I think this should all be handled by a service.
> Services have a number of extension points where they can impact
> the operating system being declared, by extending other services.
> For example, adding a package into the global profile is done by
> extending profile-service-type (which you can find in
> gnu/services.scm). Adding a shepherd service to manage a process
> is done by extending shepherd-root-service-type (in
> gnu/services/shepherd.scm).
>
> This is how many services work. As an example, sddm-service-type
> extends services to: (a) start a process on the running system,
> (b) put files in /etc, (c) install some pam services, (d) add an
> account on the system, and (e) install packages in the global
> profile.
>
> As far as I can tell, the only thing missing for a
> zfs-service-type to do what you want is that services can't
> currently add new kernel modules (although they can load them via
> kernel-module-loader-service-type). I may have missed a mechanism
> for this, though. If we added the ability to do this, then it
> should be possible to add zfs support by adding a single (service
> zfs-service-type) to your services list.
>
> The approach of using services in this way has some advantages
> which are outlined in a blog post from 2015[1]. For me the most
> compelling advantage is that an zfs-service-type is more
> restricted in what it can do, and must be more explicit. An
> install-zfs procedure has free-reign over the entire
> operating-system definition that it gets passed, which makes it
> harder to reason about the composition of such procedures.
Thank you for this, this is certainly something I would prefer to simplify
installing ZFS onto the operating system! I wasn't aware of the
`profile-service-type`.
I already have a `zfs-loader-service-type` here:
https://issues.guix.gnu.org/45643#1
These are the things that the service has to do:
* Get a kernel module compiled and added to the set of kernel modules that can
be loaded.
* From what I understand of what you describe, this is non-existent for now.
How difficult would this be? Where do I start in getting this implemented?
* Load the kernel module.
* Execute `zpool import -a -l` to import all ZFS pools, mount those that are
marked automount, and request for passwords if encrypted.
* This has to occur after kernel module loading `(requirements
'(kernel-module-loader))`.
* This has to occur before `file-systems`. In
https://issues.guix.gnu.org/45643#2 I made `file-systems` into a target similar
to `user-processes` (i.e. it has a `file-systems-service-type` that can be
extended with a list of `requirements`, just like `user-procesess`) so that the
`zpool import` service is a dependency of `file-systems`.
* This is needed in order to get `/home` on ZFS, since `user-homes` can
race with this and populate `/home` before the ZFS module is loaded and the
pools are imported, and ZFS will refuse to mount on top of a non-empty
directory. We need to make sure that ZFS gets to mount before `file-systems`
is started, since `file-systems` will also trigger `user-homes`.
* Add the compiled ZFS on the profile.
* Add ZFS module configuration into an `/etc/modprobe.d/zfs.conf`.
The above would not get us `/` on ZFS, since we would need to have the ZFS
kernel module inside the `initrd`, have the kernel load the module before
launch, probably import pools early, and then look for root pool and so on.
The problem is getting the ZFS configuration inside the `initrd` as well, in
addition to the userspace tools.
So, here's a sketch:
```scheme
(operating-system
(kernel linux-libre-5.4)
;...
(services
(cons* (service zfs-service-type
(zfs-configuration
; we need to compile for a specific kernel, the alternative here
; would be (operating-system this-operating-system) but the below
; is a good bit shorter, despite the DRY violation...
(kernel linux-libre-5.4)
(options
'(("zfs_arc_max" 500000)
;...
%desktop-services))
#;...)
```
Thanks
raid5atemyhomework
Re: A new paradigm for modifying operating system declarations
Hi Jan, > > Better but still don't like it. > Can't we put the os declaration into a variable and then pass it to a > procedure? > Say: > > > (define OS > > (operating-system > > (kernel linux-libre-5.4) > > ; ... other fields ... > > )) > > > > (install-zfs OS) > > (install-foo OS) > > (install-bar OS) > > (install-something OS) > > > > No, because `` objects are not mutable (or at least their formal interface doesn't expose any mutation). What we could do would be: ```scheme (define os (operating-system (kernel linux-libre-5.4) #;...)) (set! os (install-zfs os)) (set! os (install-foo os)) (set! os (install-bar os)) os ``` However, in many examples I've seen, the `configuration.scm` file looks like this: ```scheme (use-modules (gnu)) (use-package-modules #;...) (use-service-modules #;...) (operating-system (host-name "my-system") (timezone "Europe/Paris") (locale "en_US,utf-8") (kernel linux-libre-5.4) #;...) ``` What I want to avoid would be to have to nest the existing, usually screens-long, `operating-system` form. So compare: ```scheme (use-modules (gnu)) (use-package-modules #;...) (use-service-modules #;...) (decorate (install-zfs install-foo install-bar operating-system) (host-name "my-system") (timezone "Europe/Paris") (locale "en_US,utf-8") (kernel linux-libre-5.4) #;...) ``` versus: ```scheme (use-modules (gnu)) (use-package-modules #;...) (use-service-modules #;...) (define os (operating-system (host-name "my-system") (timezone "Europe/Paris") (locale "en_US,utf-8") (kernel linux-libre-5.4) #;...)) (set! os (install-zfs os)) (set! os (install-foo os)) (set! os (install-bar os)) os ``` I feel the former is better and requires less boilerplate. Now of course, I ***have*** seen examples as well where the `operating-system` is put in a `define` form as well, but those are rare and the default stuff that come with Guix tend not to use this, and we should consider that new Guix sysads might not be comfortable working with EMACS and prefer nano, so adding even just *one* additional layer of nestedness to a long `operating-system` form is not easy. Of course, such a sysad might then consider not indenting it correctly. Thanks raid5atemyhomework
Re: A new paradigm for modifying operating system declarations
Good morning Taylan,
> First, let me point out a more conventional alternative to what your
> 'decorate' macro does:
>
> (define (compose proc . rest)
> "Functional composition; e.g. ((compose x y) a) = (x (y a))."
> (if (null? rest)
> proc
> (let ((rest-proc (apply compose rest)))
> (lambda x
> (let-values ((x (apply rest-proc x)))
> (apply proc x))
>
> This allows for something like:
>
> ((compose install-foo install-bar install-zfs)
> (operating-system ...))
>
> Or perhaps cleaner:
>
> (define my-os-modifier (compose install-foo install-bar install-zfs))
>
> (my-os-modifier
> (operating-system ...))
>
> If you need custom modifications within, you can do:
>
> (define my-os-modifier
> (compose install-foo
> (lambda (os) ...)
> install-bar))
>
> It's more verbose, but doesn't "break" standard Scheme syntax as much.
> Function composition is conceptually pretty easy and probably more
> well-known than "decorators" (which I had never heard of, personally).
Yes, except function composition does not work on syntactic forms, which is
why, with `compose`, you have to separate the `operating-system` form instead
of being able to compose `operating-system` with the rest of the os
modifications.
The intent is that you have already an existing `operating-system` form with a
single layer of parenthesis. With `compose`, if you want to install ZFS and a
bunch of other complex OS modifications, you have to add a layer of
parenthesis. With `decorate`, you don't:
```scheme
((compose install-zfs install-foo)
(operating-system
(name "my-system") #;...))
;vs
(decorate (install-zfs
install-foo
operating-system)
(nmae "my-system") #;...)
```
>
> Fewer macros means the reader needs to keep fewer special rules in mind.
Fine, I'm doubling down then.
```scheme
(define-syntax compose-syntax
(syntax-rules ()
((compose-syntax (x ...))
(syntax-rules ::: ()
((form args :::)
(x ... args :::
((compose-syntax x)
(syntax-rules ::: ()
((form args :::)
(x args :::
((compose-syntax (x ...) y ...)
(syntax-rules ::: ()
((form args :::)
(let-syntax ((sub-syntax (compose-syntax y ...)))
(x ... (sub-syntax args :::))
((compose-syntax x y ...)
(syntax-rules ::: ()
((form args :::)
(let-syntax ((sub-syntax (compose-syntax y ...))
(x (sub-syntax args :::)
```
Then use it as follows:
```
(define-syntax my-operating-system
(compose-syntax
(install-zfs #:options '(("zfs_arc_max" 50)) #:os)
operating-system))
(my-operating-system
(name "my-system") #;...)
```
Again, the goal here is to keep the nestedness of your existing, very long
`operating-system` form, which your simple `compose` function fails to do,
because you can't compose syntax with `compose` and `operating-system` is a
syntax form.
>
> Secondly, I wonder if passing an OS declaration through various
> procedures that modify it is really the best approach in the first place.
>
> For build phases, we have the 'modify-phases' syntax. For services,
> there is 'modify-services'. Maybe there should be a 'modify-os' kind of
> syntax. (In other words, if we're going to invent new syntax, why not
> go all-out and create the most convenient syntax for the use-case.)
Because a *generic* form to modify the operating system is worthless --- you
can just define the operating-system directly.
What `install-zfs` does is that it installs the same kernel-specific package in
three different points:
* `kernel-loadable-modules`, because ZFS needs to get into the kernel somehow.
* `packages`, because the kernel module is useless if you don't have the
userland tools to interact with the kernel module.
* `services`, because ZFS is well-documented outside of Guix as automatically
mounting its filesystems at bootup, but that actually requires a bit of magic
in the `init` system, specifically you need to actually **load** the module,
then execute `zpool import -a -l` to have it scan for all filesystems and mount
those that need automounting.
Thus, an `install-zfs`, that is a *single* form that inserts the correct bits
in the correct ***three*** places, makes the experience of adding ZFS to your
`operating-system` easier because there's less scope for error in actually
adding the package. You just add a single `install-zfs`, not add three things
(plus an extra `(define my-zfs (make-zfs-package linux-libre-5.4))` before your
form).
Now, if this kind of simplifying form is useful for ZFS, I imagine that this
kind of simplifying form would also exist for other things you could install
into your operating system in the future. Thus, we need some way to take an
existing `` and pass it through a number of single
simplifying operating system transformations, which I don't think something
like `modify-os` would work well with.
Thanks
raid5atemyhomework
A new paradigm for modifying operating system declarations
file:
```scheme
(decorate (install-zfs
(install-ddcci #:options '("dyndbg" "delay=120") #:os)
operating-system)
(name "example-system")
#;...)
```
Obviously, it's called "decorate" since it's partly inspired by Python
decorators, which use a similar-looking pattern, with different syntax.
What are your opinions? Blech? Yummy? Is it worth exploring this paradigm
for adding particularly complex features to an operating system definition?
Thanks
raid5atemyhomework
Re: ZFS on Guix
Available here: https://issues.guix.gnu.org/45643
Please review! Hopefully it gets merged!
> Yes, I found that out after a little more digging.
>
> I have a preliminary patch here: https://issues.guix.gnu.org/45592
> However I will make a patchset based on this in order to support /home on ZFS
> and managing ZFS the "legacy" way by (file-system ...) declaration, please
> wait.
>
> > Hi,
> > the reason is that our "zfs" package uses
> > ("util-linux" ,util-linux "lib")
> > and then does
> > (substitute* "lib/libzfs/libzfs_mount.c"
> > (("/bin/mount") (string-append util-linux "/bin/mount"))
> > (("/bin/umount") (string-append util-linux "/bin/umount")))
> > .
> > That can't work.
> > zfs interna that will be patched by the Guix package are:
> > ./lib/libzfs/libzfs_mount.c
> > do_mount(const char *src, const char *mntpt, char *opts)
> > char *argv[9] = {
> > "/bin/mount",
> > "--no-canonicalize",
> > "-t", MNTTYPE_ZFS,
> > "-o", opts,
> > (char *)src,
> > (char *)mntpt,
> > (char *)NULL };
> > Easy fix would be to change our package to have both
> > ("util-linux:lib" ,util-linux "lib")
> > and
> > ("util-linux" ,util-linux)
> > .
Re: ZFS on Guix
Yes, I found that out after a little more digging.
I have a preliminary patch here: https://issues.guix.gnu.org/45592
However I will make a patchset based on this in order to support /home on ZFS
and managing ZFS the "legacy" way by (file-system ...) declaration, please wait.
> Hi,
>
> the reason is that our "zfs" package uses
>
> ("util-linux" ,util-linux "lib")
>
> and then does
>
> (substitute* "lib/libzfs/libzfs_mount.c"
> (("/bin/mount") (string-append util-linux "/bin/mount"))
> (("/bin/umount") (string-append util-linux "/bin/umount")))
>
> .
>
> That can't work.
>
> zfs interna that will be patched by the Guix package are:
>
> ./lib/libzfs/libzfs_mount.c
>
> do_mount(const char *src, const char *mntpt, char *opts)
>
> char *argv[9] = {
> "/bin/mount",
> "--no-canonicalize",
> "-t", MNTTYPE_ZFS,
> "-o", opts,
> (char *)src,
> (char *)mntpt,
> (char *)NULL };
>
> Easy fix would be to change our package to have both
>
> ("util-linux:lib" ,util-linux "lib")
>
> and
>
> ("util-linux" ,util-linux)
>
> .
Re: ZFS on Guix
Hi guix-developers,
I just did a an `strace -f zfs mount tank`, and got this interesting bit in a
child process:
execve("/gnu/store/a45p39mgqvfd8kjwibyr0q42klmw7gmf-util-linux-2.35.1-lib/bin/mount",
["/gnu/store/a45p39mgqvfd8kjwibyr0"..., "--no-canonicalize", "-t", "zfs",
"-o", "defaults,atime,strictatime,dev,e"..., "tank", "/tank"], 0x7ffee6ad7bf8
/* 29 vars */) = -1 ENOENT (No such file or directory)
The `/gnu/store` directory indeed does not contain a `/bin/mount` executable,
it only contains `include` `lib` and `share` dirs. So it looks to me that
maybe the `zfs` package needs some more modifications to get it working on
guix, since apparently ZFS userspace tools still go through `mount` anyway.
Trying to `mount` directly makes ZFS complain about requiring a "legacy"
mountpoint (there might be an option somewhere not shown by the `strace` that
suppress that complaint), and setting a legacy mountpoint makes `mount` error
with `filesystem 'tank' cannot be mounted at '/tank' due to canonicalization
error 2.` even though I give `--no-canonicalize`, though.
Thanks
raid5atemyhomework
ZFS on Guix
Hi guix-developers,
I'm trying to get ZFS working on a test VM running Guix System.
Attempting to do a simple `guix install zfs` does not work, as I reported in
https://issues.guix.gnu.org/45401 , ZFS 0.8.5 is only up to kernel 5.9 but
latest guix has kernel 5.10.
I created a procedure to modify the base `zfs` package and compile it for a
different Linux kernel:
```scheme
(define (make-zfs-package kernel)
(package
(inherit zfs)
(name (string-append "zfs-for-"
(package-name kernel)
"-"
(package-version kernel)
"-version"))
(arguments
(cons* #:linux kernel (package-arguments zfs)
```
Then in my system configuration `.scm`, I selected a specific Linux and had its
module added to the boot and had it loaded in a Shepherd service:
```scheme
(define system-kernel linux-libre-5.4)
(define system-zfs (make-zfs-package system-kernel))
;; ...
(operating-system
(kernel system-kernel)
(kernel-loadable-modules (list (list system-zfs "module")))
;; ...
(packages
(cons* system-zfs
; ...
%base-packages))
;; ...
(services
(cons* (service kernel-module-loader-service-type
'("zfs"))
; ...
%base-services))
;; ...
)
```
As I documented here: https://issues.guix.gnu.org/45592#1
Now ZFS is loaded at boot (`lsmod | grep zfs` shows ZFS loaded), and `zfs
version` and `zpool list` don't error out.
However, when I created some extra space in the QEMU image and created a
partition `vda3`, I get an error in `zpool create tank vda3`.
The error is `cannot mount 'tank': Input/output error`.
I've also tried adding a new device to the QEMU boot as well, also the same
occurs.
Looking at a partition editor, the partition does get formatted as zfs, it just
won't mount.
Has anyone ever managed to get ZFS working on Guix? Any tips? Is it enough to
`modprobe` it in a system service or does it have to somehow be added to the
Linux command line? Maybe it only works on even older Linux-libre versions
than 5.4?
Thanks
raid5atemyhomework
