Re: How did you handle making a GNU/Linux distribution?
Hi Sage, On 8/22/21 5:53 PM, Sage Gerard wrote: Thanks for the detailed answer! It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of Guile to bring GNU's benefits to more Scheme and Racket programmers. Has someone already tried something like that? I haven't tried Xiden yet, and I haven't done any concrete work toward this (I have been working on managing Racket packages with Guix), but Christine Lemmer-Webber had floated the idea at some point of trying to integrate Racket and Guile. IIRC, I think what she's had in mind was trying to make a Guile backend for Racket along the lines of the Chez Scheme backend (or the BC backend, or experimental backends like Pycket). As I said, I haven't actually tried any of this, but, as I've thought about what might be involved, there are two things that have struck me as downsides: 1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience Report)" (§6, p. 13) that, "If our task were to compile Racket to an existing target, then we would not have achieved such a high degree of compatibility. … we have taken the liberty of modifying Chez Scheme to make it an easier target for Racket." https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf Presumably a Racket-on-Guile project would face the same trade-off, where modifications to Guild, if Racket CS is a guide, could require hard work over many years, and lesser compatibility would make the result less useful. 2. As you probably know, Racket programs can't generally use Chez Scheme implemented libraries, because Chez Scheme effectively is the "unsafe" layer of the Racket VM. For example, not all Racket procedures are Chez Scheme procedures, and Racket's continuations wrap Chez Scheme's to implement delimited and composable control, threads, parameters, full continuation marks, etc. For Racket CS, this isn't a great loss (there aren't so many Chez-specific libraries, and portable libraries can run in Racket's R6RS language), but, for a hypothetical Racket-on-Guile, bidirectional interoperability would be a big attraction: imagine Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all working together. If I were going to work on this, I'd start by looking at having Racket and Guile coexist as siblings with interoperability through their FFIs level. Even better, eventually you could compile Guile to Racket linklets, so the two could coexist in the same primitive module system. There would probably always need to be something along the lines of require/typed to interoperate between the languages, since Guile has mutable pairs and an unusual approach to falsehood and nullity to let Scheme's #f and '() coexist with Emacs List's nil. (See https://www.gnu.org/software/guile/manual/html_node/Nil.html). I'm at the point where users are requesting a GNU/Linux distribution for Xiden, such that Racket is the primary language for day-to-day operation. I'm ignorant of the scope of work, and am unsure if I can do it alone. To me, at least, the scope of the work in creating a new GNU/Linux distribution seems daunting. My hope would be that bringing Racket and Guile closer together would let most if not all of the effort be shared. Hope some of this is useful, and best of luck! -Philip
Re: How did you handle making a GNU/Linux distribution?
Thanks for the detailed answer! It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of Guile to bring GNU's benefits to more Scheme and Racket programmers. Has someone already tried something like that? On 8/21/21 5:18 PM, Leo Famulari wrote: > On Sat, Aug 21, 2021 at 04:43:34PM +, Sage Gerard wrote: >> My name is Sage. I wrote a cross-platform Guix-like package manager >> called Xiden. It applies functional package management to the Racket >> ecosystem. It is also free software under the GPLv3. The source is >> available at zyrolasting/xiden on Github, pending migration to a new host. > Hi! Cool! > >> I'm at the point where users are requesting a GNU/Linux distribution for >> Xiden, such that Racket is the primary language for day-to-day >> operation. I'm ignorant of the scope of work, and am unsure if I can do >> it alone. My understanding is that you had to bootstrap your entire >> toolchain and address Ken Thompson's compiler hack from a different >> angle. Is that right? > At the time that Guix was founded, it was not considered possible to > bootstrap a GNU/Linux system from source; on which OS would you perform > the first compilation? How would you bootstrap that OS? Ad infinitum. > Stage0 and mes and friends are fixing that now. [0] > > So, some binary "seeds" were created, which are compiled binaries of > core tools such as GCC and binutils, and the system was based on those > seeds. And we gave instructions for reproducing these seeds; you could > use any commonly available distro as a host. Maybe that sounds icky in > 2021, since the situation has improved radically, but it's how every OS > was created since decades ago [1], except that most operating systems (and > even language ecosystems) did not keep track of their seeds or give > instructions for reproducing them. So Guix was better than all the rest > from its beginning :) > > https://guix.gnu.org/manual/en/html_node/Bootstrapping.html > > I was not involved at that stage of Guix. I'm sure others can give more > detailed answers. > >> I'd greatly appreciate learning how you all built Guix's GNU/Linux >> distribution so that I can prepare a realistic roadmap and recruit help >> where necessary. I'm bad at both of those things, but if there is any >> opportunity to collaborate on implementation details based on Xiden's >> progress, I am happy to give back. > [0] https://bootstrappable.org/ > https://bootstrappable.org/projects/mes.html > https://www.gnu.org/software/mes/ > Also visit the #bootstrappable channel on Libera.chat > > [1] It would be interesting to know the last truly bootstrapped OS
Using a swapfile on btrfs for hibernation
Hello Guix!
Here is a feedback on enabling hibernation using a swapfile stored on btrfs
filesystem. It wasn 't straight forward as it's not documented and I had to
read the code do make it work.
Note that even tho swpafile on btrfs filesystem are supported, therea are
some limitations⁵:
- the filesystem must be an a single device, ie. no btrfs RAID,
- the swapfile shouldn't be copy-on-write,
- the swapfile shouldn't be compressed.
Let's start with the btrfs specific part first. The following instructions
was compiled from tutorials¹ ² ³ for other Linux distributions. First we
gather the tools we'll need for the job, especially btrfs_map_physical⁴to
find the offset from the start of the btrfs partition where the swapfile is
located, this tool isn't packaged (I don't know if it should) so we'll
build it in the old fashion way.
--8<---cut here---start->8---
guix environment --ad-hoc gcc-toolchain curl gawk e2fsprogs btrfs-progs
curl -OL
https://github.com/osandov/osandov-linux/raw/master/scripts/btrfs_map_physical.c
gcc -O2 -o ./btrfs_map_physical ./btrfs_map_physical.c
--8<---cut here---end--->8---
Next we create our swapfile in its own subvolume named “swap” on the btrfs
filesystem of device “/dev/sda2” (mounted at “/mnt/rootfs”) and we make
sure our new swapfile isn't COWable nor compressed. To be able to hibernate
in all cironstancese the swapfile need to be at least as big as the total
RAM available.
--8<---cut here---start->8---
swappath=/mnt/rootfs/swap
swapfile="$swappath/swapfile"
cd "$path"
btrfs subvolume create swap
mount -o subvol=swap /dev/sda2 ./swap
truncate -s 0 "$swapfile"
chattr +C "$swapfile"
btrfs property set "$swapfile" compression none
swap_size="$(grep MemTotal /proc/meminfo | awk '{print $2 * 1024}')"
fallocate --length "$swap_size" "$swapfile"
chmod 600 "$swapfile"
mkswap "$swapfile"
--8<---cut here---end--->8---
As we have a working swapfile, we'll genereate the kernel arguments to pass
to the kernel. The less obvious being the offset of the swapfile from the
start of the filesystem, to do so we are using the previously compiled
“btrfs_map_physical” binary and few shell utilities.
--8<---cut here---start->8---
swapon "$swapfile"
swap_physical_offset=$(btrfs_map_physical "$swapfile" | sed -n "2p" | awk
"{print \$NF}")
swap_offset=$(echo "${swap_physical_offset} / $(getconf PAGESIZE)" | bc)
swap_uuid=$(findmnt -no UUID -T "$swapfile")
resume_args="resume=${swap_uuid} resume_offset=${swap_offset}"
echo "${resume_args}"
--8<---cut here---end--->8---
What's left is just to tweak our “operating-system” to make use of ourr new
swapfile.
--8<---cut here---start->8---
(operating-system
(file-systems
(cons* ...
(file-system
(mount-point "/")
(device
(uuid "12345678-1234-1234-1234-123456789abc" 'btrfs))
(type "btrfs")
(options "compress=zstd,subvol=guix-system"))
(file-system
(mount-point "/swap")
(device
(uuid "12345678-1234-1234-1234-123456789abc" 'btrfs))
(type "btrfs")
;; Needed to access the swapfile? (not sure)
(needed-for-boot? #t)
(options "subvol=swap"))
%base-file-systems))
(swap-devices (list "/swap/swapfile"))
(kernel-arguments
(cons* "resume="12345678-1234-1234-1234-123456789abc"
"resume_offset=16400"
%default-kernel-arguments))
...)
--8<---cut here---end--->8---
Now we can reconfigure our system and reboot it. At that point we are able
to use “loginctl hibernate” or “loginctl hybrid-sleep” to save our current
system state to the swapfile and recover it at the next reboot.
As a keen reader may have noticed, enabling hibernation in Guix in this
case has a quirk. The most the most inconvenient is the Guix's “resume”
kernel argument format being different from Linux's⁶ « {/dev/ |
PARTUUID= | : | } » and other distributions allowing «
UUID= »; in Guix the uuid isn't prefixed with “UUID=” or “PARTUUID=”
and this isn't documented anywhere, you are forced to read the code.
Also, having swap related configuration spread between “swap-devices” and
“kernel-arguments” fields make me think having specific records for that
first field would make sense. And that would make the “resume“ format issue
irrelevant. WDYT about adding the following records to be used in
“swap-devices” record in addition to the current types?
--8<---cut here---start->8---
(swap-file
(file "/swap/swapfile")
(offset 16400)
(device (uuid "12345678-1234-1234-1234-123456789abc"))
(resume? #t) ; default #f
(priority 21)) ; swapflags argument passed to
Re: core-updates weather reports!
Hello, The situation is getting better here: --8<---cut here---start->8--- https://ci.guix.gnu.org 81.1% substitutes available (16,148 out of 19,921) at least 112,494.5 MiB of nars (compressed) 17,592,186,161,392.5 MiB on disk (uncompressed) 0.006 seconds per request (29.4 seconds in total) 162.1 requests per second --8<---cut here---end--->8--- I came up with a small script to pick randomly a failing package on the core-updates-frozen branch: --8<---cut here---start->8--- (use-modules (guix ci) (srfi srfi-1)) (define url "https://ci.guix.gnu.org;) (define system "x86_64-linux") (define evals (latest-evaluations "https://ci.guix.gnu.org; 5 #:spec "core-updates-frozen")) (define (last-eval evals) (first (filter-map (lambda (eval) (and (evaluation-complete? eval) (evaluation-id eval))) evals))) (define eval (last-eval evals)) (define failed-builds (filter-map (lambda (job) (and (eq? (job-status job) 'failed) (job-name job))) (evaluation-jobs url eval))) (define %seed (seed->random-state (logxor (getpid) (car (gettimeofday) (define failed-for-system (filter-map (lambda (job) (and (string-suffix? system job) job)) failed-builds)) (format #t "~a~%" (list-ref failed-for-system (random (length failed-for-system) %seed))) --8<---cut here---end--->8--- For instance: --8<---cut here---start->8--- mathieu@meije ~/guix [env]$ guix repl ~/tmp/fix.scm rust-test-case.x86_64-linux mathieu@meije ~/guix [env]$ guix repl ~/tmp/fix.scm rust-tungstenite.x86_64-linux --8<---cut here---end--->8--- That's a fun little game :) Mathieu
Re: core-updates weather reports!
Hey, > Okay, are you confident in the #50040 patch? It looks like there are a > lot of spurious failures from #48468. I just discovered that many spurious failures were also caused by short Nginx timeout values. Added a comment about that on #50040. I'm also currently experimenting directly on Berlin on that topic. Mathieu
