Ludovic Courtès writes: Hi!
> Janneke Nieuwenhuizen <jann...@gnu.org> skribis: > >> Similar to the Makefile.am change, this breaks-up packages into 26 chunks >> when building on 32bit. Also force garbage collection. >> >> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner >> define. >> Use it when building on a "i586" or "i686" cpu. > > .go files of a Guix built with this new (guix self) would not be found: > > newfstatat(AT_FDCWD, > "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", > {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0 Hmm, how did I miss this? > In fact, guix/*.go is entirely missing, it seems: Right...that makes sense now that I look at the code again. Wait...I probably even experienced this breakage after running `guix pull' on the Hurd but failed to notice its cause, the missing guix/*.go, and ascribed it to "something" being broken on Hurd. So the good news is that we'll most probably have guix pull work on the Hurd after fixing this! --8<---------------cut here---------------start------------->8--- (cute partition <> files) (lambda (regex) (cute string-match regex <>)) (cute string-append "^gnu/packages/" <>) --8<---------------cut here---------------end--------------->8--- it still has the "gnu/packages" directory name hardcoded here. (I even believe this was kind-of intentional, meaning not to split-up builds of other directories...). The result is, of course, that partition always produces an empty result for any '("guix/foo.scm") files! It is fixed by doing --8<---------------cut here---------------start------------->8--- (cute string-append "^" directory "/" <>) --8<---------------cut here---------------end--------------->8--- i.e., also splitting guix/ 26 ways. > I’m reverting for now. Thank you for looking into and taking care of this mess I made, sorry! > Note that (guix self) is very sensitive, so we should test it > thoroughly. Using the attached patch, I ran --8<---------------cut here---------------start------------->8--- $ guix pull --branch=master --url=$PWD [..] 08:11:02 janneke@drakenpad:~/src/guix/master $ guix describe Git checkout: repository: /home/janneke/src/guix/master/ branch: master commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 08:11:04 janneke@drakenpad:~/src/guix/master $ guix build hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- > Here I used ‘make as-derivation’ and ran the resulting > ‘guix’ command. Ah, nice that guix works too for me too now --8<---------------cut here---------------start------------->8--- $ make as-derivation [..] /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06 08:13:15 janneke@drakenpad:~/src/guix/master [env] $ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix build hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- Greetings, Janneke
>From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001 Message-ID: <a639a64d590067d3fe928be3ecc9d2d4cc2e8df3.1692770000.git.jann...@gnu.org> From: Janneke Nieuwenhuizen <jann...@gnu.org> Date: Thu, 22 Jun 2023 08:30:25 +0200 Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of directories into 26 chunks. --- guix/self.scm | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..e5a85eaad6 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès <l...@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbe...@riseup.net> +;;; Copyright © 2023 Janneke Nieuwenhuizen <jann...@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files (force-output)) (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + ;; Split gnu/packages in 26 chunks to avoid OOM errors + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^" directory "/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + ;; Hide compilation warnings. + (parameterize ((current-warning-port (%make-void-port "w"))) + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea -- 2.41.0
-- Janneke Nieuwenhuizen <jann...@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com