bug#43818: Use of local-file in icecat-source definition breaks REPL

[Maxim Cournoyer]

Hello,

Maxim Cournoyer  writes:

> CC: Mark H Weaver, one of the Icecat maintainer.
>
> Hello,
>
> The problem is that local-file doesn't work in Geiser.  This breaks
> working at the REPL:
>
> Enter `,help' for help.
> scheme@(guile-user)> ,m (gnu packages linux)
> While executing meta-command:
> ERROR:
>   1. :
>   format: "~a: patch not found\n"
>   arguments: ("icecat-use-older-reveal-hidden-html.patch")

Thanks for Christopher Baines on #guix who pointed that local-file
seemed to behave correctly at the REPL and Geiser (indeed!).  It made me
try to reproduce it in a --pure environment and I couldn't.

After a couple hours digging, I found that this snippet in my modified
.dir-locals file:

--8<---cut here---start->8---
(with-eval-after-load 'geiser-guile
  (let ((root-dir (locate-dominating-file
   default-directory ".dir-locals.el")))
(setq geiser-guile-load-path
  (cons root-dir
(delete root-dir geiser-guile-load-path)
--8<---cut here---end--->8---

Was the culprit, especially the use of with-eval-after-load (which
turned out to be unnecessary).

This snippet allows to switch between git worktrees and always have the
geiser-guile-load-path (which configures both the Guile %load-path and
the %load-compiled-path) set correctly.

Here's the diff of my (fixed) .dir-locals in my Guix git checkout:

--8<---cut here---start->8---
~/src/guix$ git diff .dir-locals.el
diff --git a/.dir-locals.el b/.dir-locals.el
index 19f15b3e1a..0869fbaa20 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,7 +8,17 @@
  ;; For use with 'bug-reference-prog-mode'.
  (bug-reference-url-format . "http://bugs.gnu.org/%s;)
  (bug-reference-bug-regexp
-  . "")))
+  . "")
+
+ ;; Emacs-Guix
+ (eval . (setq guix-directory
+   (locate-dominating-file default-directory 
".dir-locals.el")))
+ ;; Geiser
+ (eval . (let ((root-dir (locate-dominating-file
+  default-directory ".dir-locals.el")))
+   (setq geiser-guile-load-path
+ (cons root-dir
+   (delete root-dir geiser-guile-load-path)))
  (c-mode  . ((c-file-style . "gnu")))
  (scheme-mode
   .
--8<---cut here---end--->8---

I am closing this issue, sorry for the noise!

Maxim

bug#44101: Unable to use /dev/disk/by-id/ symlinks with u-boot and guix system reconfigure

[Vagrant Cascadian]

On 2020-10-23, Mathieu Othacehe wrote:
>>> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>>> ERROR:
>>>   1. /o-filename: "/dev/disk/by-id/mmc-SDU64_0xbaf3002e"
>>
>> Thanks for testing. This is probably a regression in
>> "write-file-on-device" that fails to open this kind of files. I'll try
>> to reproduce it locally.
>
> This is because the O_EXCL flag was passed, which causes open to fail if
> the block device is mounted.
>
> The revised patch attached should fix it.

Yes, this patch fixes the issue for me.

Thanks!

live well,
  vagrant

> From 780327ebb0db74ca4cc43d26ba7cf945d64c7d30 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe 
> Date: Wed, 21 Oct 2020 10:42:50 +0200
> Subject: [PATCH] system: reconfigure: Use the disk-installer if provided.
>
> ---
>  gnu/build/bootloader.scm|  4 ++--
>  gnu/tests/reconfigure.scm   |  4 +++-
>  guix/scripts/system/reconfigure.scm | 12 +---
>  3 files changed, 14 insertions(+), 6 deletions(-)
>
> diff --git a/gnu/build/bootloader.scm b/gnu/build/bootloader.scm
> index 5ec839f902..06a2c3ab76 100644
> --- a/gnu/build/bootloader.scm
> +++ b/gnu/build/bootloader.scm
> @@ -39,9 +39,9 @@
>(let ((bv (get-bytevector-n input size)))
>  (call-with-port
>   (open-file-output-port device
> -(file-options no-truncate no-create)
> +(file-options no-truncate no-fail)
>  (buffer-mode block)
> -(native-transcoder))
> +(make-transcoder (latin-1-codec)))
>   (lambda (output)
> (seek output offset SEEK_SET)
> (put-bytevector output bv)))
> diff --git a/gnu/tests/reconfigure.scm b/gnu/tests/reconfigure.scm
> index 928a210a94..52beeef447 100644
> --- a/gnu/tests/reconfigure.scm
> +++ b/gnu/tests/reconfigure.scm
> @@ -260,7 +260,9 @@ bootloader's configuration file."
>   ;; test suite, the bootloader installer script is omitted. 
> 'grub-install'
>   ;; would attempt to write directly to the virtual disk if the
>   ;; installation script were run.
> - (test (install-bootloader-program #f #f bootcfg bootcfg-file #f "/")
> + (test
> +  (install-bootloader-program #f #f #f bootcfg bootcfg-file #f "/")
> +
>  
>  (define %test-switch-to-system
>(system-test
> diff --git a/guix/scripts/system/reconfigure.scm 
> b/guix/scripts/system/reconfigure.scm
> index d89caf80fc..b1982b20d2 100644
> --- a/guix/scripts/system/reconfigure.scm
> +++ b/guix/scripts/system/reconfigure.scm
> @@ -204,7 +204,8 @@ services as defined by OS."
>  ;;; Bootloader configuration.
>  ;;;
>  
> -(define (install-bootloader-program installer bootloader-package bootcfg
> +(define (install-bootloader-program installer disk-installer
> +bootloader-package bootcfg
>  bootcfg-file device target)
>"Return an executable store item that, upon being evaluated, will install
>  BOOTCFG to BOOTCFG-FILE, a target file name, on DEVICE, a file system device,
> @@ -246,10 +247,12 @@ BOOTLOADER-PACKAGE."
>   ;; a broken installation.
>   (switch-symlinks new-gc-root #$bootcfg)
>   (install-boot-config #$bootcfg #$bootcfg-file #$target)
> - (when #$installer
> + (when (or #$installer #$disk-installer)
> (catch #t
>   (lambda ()
> -   (#$installer #$bootloader-package #$device #$target))
> +   (if #$installer
> +   (#$installer #$bootloader-package #$device #$target)
> +   (#$disk-installer #$bootloader-package 0 #$device)))
>   (lambda args
> (delete-file new-gc-root)
> (match args
> @@ -272,11 +275,14 @@ additional configurations specified by MENU-ENTRIES can 
> be selected."
>(let* ((bootloader (bootloader-configuration-bootloader configuration))
>   (installer (and run-installer?
>   (bootloader-installer bootloader)))
> + (disk-installer (and run-installer?
> +  (bootloader-disk-image-installer bootloader)))
>   (package (bootloader-package bootloader))
>   (device (bootloader-configuration-target configuration))
>   (bootcfg-file (bootloader-configuration-file bootloader)))
>  (eval #~(parameterize ((current-warning-port (%make-void-port "w")))
>(primitive-load #$(install-bootloader-program installer
> +disk-installer
>  package
>  bootcfg
>  bootcfg-file
> -- 
> 2.28.0


signature.asc
Description: PGP 

bug#44187: whishlist: time-machine --channel falls back to SWH

[zimoun]

Dear,

Let’s describe the use case.  Consider that:

  guix time-machine -C channels -- install foo

is provided in some documentation, say scientific paper.  Where the
channels.scm file is completly described:

--8<---cut here---start->8---
(list (channel
(name 'kikoo)
(url "https://example.org/that-great.git;)
(commit
  "353bdae32f72b720c7ddd706576ccc40e2b43f95")))
--8<---cut here---end--->8---

In the future, if https://example.org/that-great.git disappears, then
build/install the package ’foo’ is becoming difficult, nor impossible.

However, let’s consider that the repo ’that-great’ had been saved in SWH
(say manually); since it is a regular Git repo.  Guix should be able to
fallback to it transparently.


Obviously, another whislist is to have something to ease the save
request of the channel on SWH.  Maybe this latter could be part of the
several-times discussed “guix channel” subcommand. :-)


All the best,
simon

bug#44175: [optimization] Grafting is too slow

[Ludovic Courtès]

Maxim Cournoyer  skribis:

> Lars-Dominik Braun  writes:
>
>> Hi Maxim,
>>
>>> Judging from the above, it seems this issue has been resolved.
>> grafting is still a performance issue imo. Compare for example:
>>
>> $ time guix environment --ad-hoc  --search-paths r-learnr
>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
>> 210% cpu 2,844 total
>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
>> 0,08s system 164% cpu 1,277 total
>
> I'm opening a new issue to track optimizing the grafting code, since
> it's independent of environments (grafts are applied anytime a
> derivation is built, AFAICT).  Grafting is inherently IO-bound,

What is slow above is not grafting itself: it’s determining what to
graft that takes CPU time.

I had reopened the initial bug at ;
should we close this one?

Thanks,
Ludo’.

bug#44184: Building icecat does not always respect the number of cores

[divoplade]

Dear guix,

I have configured the guix daemon to use 1 core by default, so that
unattended upgrades and CI jobs will not fill my system memory and
crush my computer.

However, I noticed that in a precise time frame of compiling icecat,
the number of cores were not respected and all my cores were compiling.
The processes were named rustc, but it went fast so I could not gather
more info. 

This is not really annoying, because rust compilation does not take
that much memory, but I still have the feeling that it is a bug.

Best regards,

divoplade

bug#44179: can't build proot-static

[luhux]

If it can be fixed, I can run my packaged Emacs environment on my Android phone.

I executed this command on my x86_64-linux:

guix pack -s aarch64-linux -RR busybox

but it fail:

The following derivation will be built:
   /gnu/store/aax96hkd865z2932xlbfrhqrd7zwcyra-proot-static-5.1.0.drv
building /gnu/store/aax96hkd865z2932xlbfrhqrd7zwcyra-proot-static-5.1.0.drv...
WARNING: (guile-user): imported module (guix build utils) overrides core 
binding `delete'
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
environment variable `PATH' set to 
`/gnu/store/k6lhgdbxkix3fk1bw71bymsyv0ibi411-which-2.21/bin:/gnu/store/wnssqwknkb5kdprn47zf2m0s7faar57q-util-linux-2.35.1/bin:/gnu/store/wnssqwknkb5kdprn47zf2m0s7faar57q-util-linux-2.35.1/sbin:/gnu/store/a60cldyjgqjcxnaxgi7rmk2sgsnbf3dv-coreutils-minimal-8.30/bin:/gnu/store/68lv4ix40lwrp0v99cgxa9ji55h9ay27-make-4.2.1/bin:/gnu/store/ggfdxvdgbrjvg2cx3mm7yv5j1y433b01-tar-1.32/bin:/gnu/store/ink6zpk9adb2k2w6a0qif3d45vazl7dv-gzip-1.10/bin:/gnu/store/n8dfjyw11airbnfzz192p9bdbgf31kvj-bzip2-1.0.8/bin:/gnu/store/ggq21is2vyq1jkkqrs90brmwayc4rnmy-xz-5.2.4/bin:/gnu/store/qlxznrl2zzivcln9zgyzw2zshqkjirfs-file-5.38/bin:/gnu/store/yl67anl7279qzrd3kxzj97541jzr2k3d-diffutils-3.7/bin:/gnu/store/w5n1p8kj9zn4zmjhr0bjbknakxij67w2-patch-2.7.6/bin:/gnu/store/x03dmf3xhi3il3mx383zpl4zic38iz1p-findutils-4.7.0/bin:/gnu/store/j80hcnd9p1hh0yb5j1r0gn4vd329a78n-gawk-5.0.1/bin:/gnu/store/wk5vdnms96n2iqfpqyg9gl5lhg8vdc32-sed-4.8/bin:/gnu/store/f3qp8l7f6kmfrvwnyix35kax2lv7lmnb-grep-3.4/bin:/gnu/store/wiv74mq31s0vppzm99qwnjkd8xa3albw-coreutils-8.32/bin:/gnu/store/j7fqszmnydxix8l35d7l3b7dj44qjrw4-make-4.3/bin:/gnu/store/7sfbiqh21h90bc6zi8br1xh60m6qgdd5-bash-minimal-5.0.16/bin:/gnu/store/5m8wvcahy3dgpyhxlq5aq85zbvh5jqfz-ld-wrapper-0/bin:/gnu/store/5371r5kyj50l6176lg6jbfzgfgb2xh14-binutils-2.34/bin:/gnu/store/ap7hgyv4rjqmhg4a6cb6cypsh3g1f5q4-gcc-7.5.0/bin:/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/bin:/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/sbin'
environment variable `BASH_LOADABLES_PATH' unset
environment variable `C_INCLUDE_PATH' set to 
`/gnu/store/68lv4ix40lwrp0v99cgxa9ji55h9ay27-make-4.2.1/include:/gnu/store/7qra5x15yxnys5w53w5ihg5ibg11sdi0-talloc-static-2.3.1/include:/gnu/store/n8dfjyw11airbnfzz192p9bdbgf31kvj-bzip2-1.0.8/include:/gnu/store/ggq21is2vyq1jkkqrs90brmwayc4rnmy-xz-5.2.4/include:/gnu/store/qlxznrl2zzivcln9zgyzw2zshqkjirfs-file-5.38/include:/gnu/store/j80hcnd9p1hh0yb5j1r0gn4vd329a78n-gawk-5.0.1/include:/gnu/store/j7fqszmnydxix8l35d7l3b7dj44qjrw4-make-4.3/include:/gnu/store/5371r5kyj50l6176lg6jbfzgfgb2xh14-binutils-2.34/include:/gnu/store/ap7hgyv4rjqmhg4a6cb6cypsh3g1f5q4-gcc-7.5.0/include:/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/include:/gnu/store/nkpdg3jwllrsh7wbkzgflkcpyjn5wp0q-linux-libre-headers-5.4.20/include'
environment variable `CPLUS_INCLUDE_PATH' set to 
`/gnu/store/68lv4ix40lwrp0v99cgxa9ji55h9ay27-make-4.2.1/include:/gnu/store/7qra5x15yxnys5w53w5ihg5ibg11sdi0-talloc-static-2.3.1/include:/gnu/store/n8dfjyw11airbnfzz192p9bdbgf31kvj-bzip2-1.0.8/include:/gnu/store/ggq21is2vyq1jkkqrs90brmwayc4rnmy-xz-5.2.4/include:/gnu/store/qlxznrl2zzivcln9zgyzw2zshqkjirfs-file-5.38/include:/gnu/store/j80hcnd9p1hh0yb5j1r0gn4vd329a78n-gawk-5.0.1/include:/gnu/store/j7fqszmnydxix8l35d7l3b7dj44qjrw4-make-4.3/include:/gnu/store/5371r5kyj50l6176lg6jbfzgfgb2xh14-binutils-2.34/include:/gnu/store/ap7hgyv4rjqmhg4a6cb6cypsh3g1f5q4-gcc-7.5.0/include/c++:/gnu/store/ap7hgyv4rjqmhg4a6cb6cypsh3g1f5q4-gcc-7.5.0/include:/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/include:/gnu/store/nkpdg3jwllrsh7wbkzgflkcpyjn5wp0q-linux-libre-headers-5.4.20/include'
environment variable `LIBRARY_PATH' set to 
`/gnu/store/7qra5x15yxnys5w53w5ihg5ibg11sdi0-talloc-static-2.3.1/lib:/gnu/store/n8dfjyw11airbnfzz192p9bdbgf31kvj-bzip2-1.0.8/lib:/gnu/store/ggq21is2vyq1jkkqrs90brmwayc4rnmy-xz-5.2.4/lib:/gnu/store/qlxznrl2zzivcln9zgyzw2zshqkjirfs-file-5.38/lib:/gnu/store/j80hcnd9p1hh0yb5j1r0gn4vd329a78n-gawk-5.0.1/lib:/gnu/store/5371r5kyj50l6176lg6jbfzgfgb2xh14-binutils-2.34/lib:/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib:/gnu/store/z7hanmdmdalqh1v0y7z8ilinfhyfh91d-glibc-2.31-static/lib:/gnu/store/4xjcx6g4i229hzzqpqak6vw9bvqh88gi-glibc-utf8-locales-2.31/lib'
environment variable `GUIX_LOCPATH' set to 
`/gnu/store/4xjcx6g4i229hzzqpqak6vw9bvqh88gi-glibc-utf8-locales-2.31/lib/locale'
phase `set-paths' succeeded after 0.1 seconds
starting phase `install-locale'
using 'en_US.utf8' locale for category "LC_ALL"
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
proot-5.1.0-checkout/
proot-5.1.0-checkout/.travis.yml
proot-5.1.0-checkout/AUTHORS
proot-5.1.0-checkout/COPYING
proot-5.1.0-checkout/README.rst
proot-5.1.0-checkout/doc/
proot-5.1.0-checkout/doc/GNUmakefile
proot-5.1.0-checkout/doc/articles/
proot-5.1.0-checkout/doc/articles/extending_qemu-fig1.svg

bug#44101: Unable to use /dev/disk/by-id/ symlinks with u-boot and guix system reconfigure

[Vagrant Cascadian]

On 2020-10-23, Mathieu Othacehe wrote:
>> I don't quite understand why that would be the issue here; guix system
>> reconfigure works fine when /dev/mmcblkN is specified target in the
>> system config.scm, just not when the target is /dev/disk/by-id/...
>
> I don't think it works fine with /dev/mmcblkN. I think the bootloader
> configuration file is installed in /boot/extlinux/extlinux.conf when
> using /dev/mmcblkN or /dev/disk/, but the bootloader itself is actually
> never installed.

That does not match the behavior I observed; it definitely successfully
installs the bootloader when passing the raw device rather than a
symlink. Which I can only think of a small number of conditions that
would trigger that sort of behavior...

I'm fairly confident in this because I've been experimenting with two
incompatible u-boot bootloaders on the same boot media (one for
pinebook, and one for pinebook pro) and a successful installation
overwrites the bootloader, resulting in the expected machine booting,
and the other machine not booting. With a symlink it doesn't install to
the raw device at all...

That said, I'll move on to testing your next patch. :)


live well,
  vagrant


signature.asc
Description: PGP signature

bug#43893: [PATCH v3] maint: update-guix-package: Prevent accidentally breaking guix pull.

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer  skribis:

> The original problem was about the updated Guix package containing a
> faulty hash (due to being computed from a uncontrolled checkout that
> could be dirty).  The other concern about preventing the use of a not
> yet published commit was added based on earlier feedback.
>
>> above.  The trick at “Re-add SOURCE” is here precisely to prevent
>> attempts to re-download, as the comment explains, and this is what has
>> been causing these troubles.
>>
>> The advantages (to me) would have been simpler code, no shelling out to
>> ‘git’, and no fiddling with files under $PWD.
>
> Less code would also mean poorer diagnostics:
>
> time ./pre-inst-env guix build guix -S
> The following derivation will be built:
>
> /gnu/store/zhfchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv
> building 
> /gnu/store/zhfchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to 
> `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
> Initialized empty Git repository in 
> /gnu/store/02da8jb3wzzi3bqvrl214gdg0kkxmaf8-guix-1.1.0-31.07c13ae-checkout/.git/
> error: Server does not allow request for unadvertised object 
> 07c13aeb5abb1a5bc3cabffb9b2212993a0d5a0e
> Failed to do a shallow fetch; retrying a full fetch...
> From https://git.savannah.gnu.org/git/guix
>  * [new branch]  core-updates-> origin/core-updates
> [...]
>  * [new tag]   v1.1.0rc2   -> v1.1.0rc2
> fatal: reference is not a tree: 07c13aeb5abb1a5bc3cabffb9b2212993a0d5a0e
> git-fetch: 
> '/gnu/store/i5b1vv7qc6l2gi4xwa9mqzjy3shvgk30-git-minimal-2.28.0/bin/git 
> checkout 07c13aeb5abb1a5bc3cabffb9b2212993a0d5a0e' failed with exit code 128
> Trying content-addressed mirror at berlin.guix.gnu.org...
> Trying content-addressed mirror at berlin.guix.gnu.org...
> Trying to download from Software Heritage...
> builder for 
> `/gnu/store/zhfchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv'
>  failed to produce output path 
> `/gnu/store/02da8jb3wzzi3bqvrl214gdg0kkxmaf8-guix-1.1.0-31.07c13ae-checkout'
> build of 
> /gnu/store/zhfchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv
>  failed
> View build log at 
> '/var/log/guix/drvs/zh/fchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv.bz2'.
> guix build: error: build of 
> `/gnu/store/zhfchz831mncd2kyqmh5d2q0r2rpg57s-guix-1.1.0-31.07c13ae-checkout.drv'
>  failed
>
> real1m42.825s
> user0m2.191s
> sys 0m0.189s
>
> This took nearly 2 minutes, fetching the full Guix git repo just to tell
> me that *something* is not right.
>
> Currently, we have:
>
> time make update-guix-package
> git rev-parse HEAD
> 4893a1394e2eb8b97995b491f2f37ed85513a20f
> ./pre-inst-env 
> "/gnu/store/i7z4pfa0c22q0qkxyl7fy2nlp3w658yg-profile/bin/guile"   
>\
>./build-aux/update-guix-package.scm  \
>"`git rev-parse HEAD`"
> error: Commit 4893a1394e2eb8b97995b491f2f37ed85513a20f is not pushed 
> upstream.  Aborting.
> make: *** [Makefile:6507: update-guix-package] Error 1

I agree that the better diagnostic is nice.  Though it’s a script that’s
essentially for a handful of people, who can certainly cope with the
ugly error.

Anyway, I think we didn’t analyze the initial situation well enough
(myself included, by not commenting early and accurately).  I’m also not
fond of the added complexity and the risk of surprises when we make the
release, but OTOH, it’s no big deal in the big picture!

>> BTW, in ‘make release’ does ‘make update-guix-package’ and expects it to
>> work with a not-pushed-yet commit.  So it’s a case where we need
>> GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes.
>
> Ah, good point.  I'd like to fix this,

It’s mostly about setting this variable at the right place in
Makefile.am.

> but running 'make release', it fails on:
>
> make[3]: *** No rule to make target 'po/doc/guix-manual.pot', needed by
> 'distdir-am'.  Stop.
>
> What did I miss?

Probably you need ./bootstrap to generate the POT files.

Thanks,
Ludo’.

bug#41702: `guix environment` performance issues

[Ludovic Courtès]

Hi,

Lars-Dominik Braun  skribis:

> grafting is still a performance issue imo. Compare for example:

Agreed.  The fix in 58bb833365db4e8934a386497d5b00a063cfd27d is
incomplete: we’re still potentially doing things several times.

> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
> 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
> 0,08s system 164% cpu 1,277 total

Here’s what I get over a local socket:

--8<---cut here---start->8---
ludo@ribbon ~/src/guix$ time guix environment --ad-hoc  --search-paths r-learnr 
--no-grafts
export 
PATH="/gnu/store/wkhd1fnmg1q30g21bfn5jprz24m041w6-profile/bin${PATH:+:}$PATH"

real0m1.790s
user0m1.928s
sys 0m0.102s
ludo@ribbon ~/src/guix$ time guix environment --ad-hoc  --search-paths r-learnr
export 
PATH="/gnu/store/pwzrpzdbli2561f2yq70kljywn93h5w6-profile/bin${PATH:+:}$PATH"

real0m3.820s
user0m4.954s
sys 0m0.105s
--8<---cut here---end--->8---

Ludo’.

bug#41702: `guix environment` performance issues

[Lars-Dominik Braun]

Hi Maxim,

> Judging from the above, it seems this issue has been resolved.
grafting is still a performance issue imo. Compare for example:

$ time guix environment --ad-hoc  --search-paths r-learnr
guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% 
cpu 2,844 total
$ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s 
system 164% cpu 1,277 total

Cheers,
Lars



signature.asc
Description: PGP signature

bug#44175: [optimization] Grafting is too slow

[Maxim Cournoyer]

Hello Lars,

Lars-Dominik Braun  writes:

> Hi Maxim,
>
>> Judging from the above, it seems this issue has been resolved.
> grafting is still a performance issue imo. Compare for example:
>
> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
> 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
> 0,08s system 164% cpu 1,277 total

I'm opening a new issue to track optimizing the grafting code, since
it's independent of environments (grafts are applied anytime a
derivation is built, AFAICT).  Grafting is inherently IO-bound, since it
must scan all the built artifacts of a grafted package (from its
dependency that was 'replaced' all the way up), if I understood
correctly.  "info (guix) Security Updates" gives some information.

Perhaps we could benchmark how fast our code currently is for grafting,
and compare it with the fastest sed-like utility out there, as a
starting point, to have an idea of how much there is to gain from
optimization.

Maxim

bug#43565: cuirass: Fibers scheduling blocked.

[Ludovic Courtès]

Good afternoon fearless hacker!

Mathieu Othacehe  skribis:

>> ‘process-build-log’ in Cuirass uses ‘read-line/non-blocking’ to read a
>> line from the log port of ‘build-derivations&’.  If that really is
>> non-blocking (and I think it is), then we should be fine?
>>
>> We should attach GDB to Cuirass next time to see what’s blocking.
>
> Cuirass is currently hanging probably due to the same issue. I saved a
> GDB core dump in /home/mathieu/core.76483.

For those following along at home, we have 60 threads in there.

A couple of threads are blocked in ‘clock_nanosleep’, which I considered
fishy at first:

--8<---cut here---start->8---
(gdb) bt
#0  0x7fe26752f7a1 in __GI___clock_nanosleep (clock_id=-612010, flags=0, 
req=0x7fdf6b40d140, rem=0x7fdf6b40d140)
at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1  0x7fe267a0166d in ffi_call_unix64 ()
   from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#2  0x7fe2679ffac0 in ffi_call_int () from 
/gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#3  0x7fe267af5f2e in scm_i_foreign_call (cif_scm=, 
pointer_scm=, 
errno_ret=errno_ret@entry=0x7fe25a8e86cc, argv=0x7fe25b955df0) at 
foreign.c:1073
#4  0x7fe267b64a84 in foreign_call (thread=0x7fe26741e480, cif=, pointer=)
at vm.c:1282
#5  0x7fe2505253e0 in ?? ()
#6  0x7fe26741e480 in ?? ()
#7  0x7fe267bd7620 in ?? () from 
/gnu/store/0w76khfspfy8qmcpjya41chj3bgfcy0k-guile-3.0.4/lib/libguile-3.0.so.1
#8  0x7fe26741e480 in ?? ()
#9  0x7fe267b1043b in scm_jit_enter_mcode (thread=0x7fe26741e480, 
thread@entry=0x7fe2505253b0, 
mcode=0x7fe25052627c "L\215\243\210") at jit.c:5852
#10 0x7fe267b6bc24 in vm_regular_engine (thread=0x7fe2505253b0) at 
vm-engine.c:415
#11 0x7fe267b6c5b5 in scm_call_n (proc=proc@entry=#, 
argv=argv@entry=0x0, 
nargs=nargs@entry=0) at vm.c:1608
#12 0x7fe267ae8ae9 in scm_call_0 (proc=proc@entry=#) 
at eval.c:490
#13 0x7fe267adb138 in scm_call_with_unblocked_asyncs (proc=#) at async.c:406
--8<---cut here---end--->8---

This can only come from (fibers posix-clocks) via
‘with-interrupts’—probably OK.

Then there’s a couple of threads block in ‘pthread_cond_wait’, but
that’s presumably also Fibers internals.

Then there’s a whole bunch of threads stuck in ‘read’:

--8<---cut here---start->8---
(gdb) bt
#0  0x7fe267a180a4 in __libc_read (fd=80, buf=buf@entry=0x7fe22b0bb8f0, 
nbytes=nbytes@entry=8)
at ../sysdeps/unix/sysv/linux/read.c:26
#1  0x7fe267af69c7 in fport_read (port=, dst=, start=, count=8)
at fports.c:597
#2  0x7fe267b30542 in trampoline_to_c_read (port=# 7fe22b7b9880>, 
dst="#" = {...}, start=0, count=8) at ports.c:266
#3  0x7fe2580cb5fe in ?? ()
#4  0x7fe267431d80 in ?? ()
#5  0x7fe267bd7620 in ?? () from 
/gnu/store/0w76khfspfy8qmcpjya41chj3bgfcy0k-guile-3.0.4/lib/libguile-3.0.so.1
#6  0x7fe267431d80 in ?? ()
#7  0x7fe267b1043b in scm_jit_enter_mcode (thread=0x7fe267431d80, 
thread@entry=0x7fe2580cb5d0, 
mcode=0x7fe229340690 "H\203\350(I\211\314I)\304I\203\374\060\017\205T\003") 
at jit.c:5852
#8  0x7fe267b6b8e9 in vm_regular_engine (thread=0x7fe2580cb5d0) at 
vm-engine.c:360
#9  0x7fe267b6c5b5 in scm_call_n (proc=proc@entry=#, 
argv=argv@entry=0x0, 
nargs=nargs@entry=0) at vm.c:1608
#10 0x7fe267ae8ae9 in scm_call_0 (proc=proc@entry=#) 
at eval.c:490
#11 0x7fe267adb138 in scm_call_with_unblocked_asyncs (proc=#) at async.c:406
--8<---cut here---end--->8---

‘trampoline_to_c_read’ is known as ‘port-read’ in Scheme, so I think the
call above comes from ‘read-bytes’ in (ice-9 suspendable-ports).

Normally, this file descriptor is O_NONBLOCK, and thus ‘fport_read’
immediately returns EAGAIN, so ‘trampoline_to_c_read’ returns #false.

But does Cuirass create file descriptors as O_NONBLOCK?  This has to be
done explicitly, Fibers won’t do it for us.  As it turns out, the answer
is no, in at least one important case: the connection to the daemon
(untested patch below).

While GC is running, Cuirass typically sends ‘build-derivations’ RPCs
and they block until the GC lock is released.  That can lead to the
situation above: a bunch of threads blocked in ‘read’ from their daemon
socket, waiting for the RPC reply.  OTOH, ‘build-derivations’ RPCs are
made from a fresh thread created by ‘build-derivations&’.

There are probably other situations where the daemon replies slowly.
For instance, ‘fetch-input’ can remain stuck until GC is over.

WDYT?

Thanks for investigating!

Ludo’.

diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm
index 5a0c826..6db43c4 100644
--- a/src/cuirass/base.scm
+++ b/src/cuirass/base.scm
@@ -36,6 +36,9 @@
   #:use-module ((guix config) #:select (%state-directory))
   #:use-module (git)
   #:use-module (ice-9 binary-ports)
+  #:use-module 

bug#25527: PostgreSQL retains references to ld-wrapper and coreutils

[zimoun]

Hi,

On Fri, 23 Oct 2020 at 12:05, Ludovic Courtès  wrote:

> I think you’re mistaken:  :-)

Indeed!  Thank you for the explanation.

The manual says (emphasis by me):

References are a subset of the inputs of the derivation; this
subset is AUTOMATICALLY COMPUTED by the build daemon by scanning
all the files in the outputs.

Therefore, how is it computed?  Or what is the entry point to understand
how the build daemon does?

The output is the folder $(guix build postgresql)/, right?  So it means
one (or more) file refers to ld-wrapper and coreutils.  How can I find
which one?  Other said, what are the tools I could use?


All the best,
simon

bug#25527: PostgreSQL retains references to ld-wrapper and coreutils

[Ludovic Courtès]

Hi,

zimoun  skribis:

> The command “guix graph postgresql -t bag | dot -Tpdf > /tmp/psql.pdf”
> shows that this ’ld-wrapper-0“ comes from ‘util-linux’.  Therefore, is
> it affordable to remove this dependency?

I think you’re mistaken:  :-)

--8<---cut here---start->8---
$ guix graph -t references --path postgresql ld-wrapper
/gnu/store/2w144h8kp0pzngi5zziay3pz0hxqii6i-postgresql-11.6
/gnu/store/mpa04aq8lblbcviyxywxcsb1zbi0mf39-ld-wrapper-0
$ guix graph -t references --path $(guix build util-linux | grep '[0-9]$') 
ld-wrapper
guix graph: error: no path from 
'/gnu/store/j3vz21b9xgf244z1ivllj61i8975qx6r-util-linux-2.35.1' to 
'/gnu/store/mpa04aq8lblbcviyxywxcsb1zbi0mf39-ld-wrapper-0'
--8<---cut here---end--->8---

Ludo’.

bug#44101: Unable to use /dev/disk/by-id/ symlinks with u-boot and guix system reconfigure

[Mathieu Othacehe]

Hello,

>> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>> ERROR:
>>   1. /o-filename: "/dev/disk/by-id/mmc-SDU64_0xbaf3002e"
>
> Thanks for testing. This is probably a regression in
> "write-file-on-device" that fails to open this kind of files. I'll try
> to reproduce it locally.

This is because the O_EXCL flag was passed, which causes open to fail if
the block device is mounted.

The revised patch attached should fix it.

Thanks,

Mathieu
>From 780327ebb0db74ca4cc43d26ba7cf945d64c7d30 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe 
Date: Wed, 21 Oct 2020 10:42:50 +0200
Subject: [PATCH] system: reconfigure: Use the disk-installer if provided.

---
 gnu/build/bootloader.scm|  4 ++--
 gnu/tests/reconfigure.scm   |  4 +++-
 guix/scripts/system/reconfigure.scm | 12 +---
 3 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/gnu/build/bootloader.scm b/gnu/build/bootloader.scm
index 5ec839f902..06a2c3ab76 100644
--- a/gnu/build/bootloader.scm
+++ b/gnu/build/bootloader.scm
@@ -39,9 +39,9 @@
   (let ((bv (get-bytevector-n input size)))
 (call-with-port
  (open-file-output-port device
-(file-options no-truncate no-create)
+(file-options no-truncate no-fail)
 (buffer-mode block)
-(native-transcoder))
+(make-transcoder (latin-1-codec)))
  (lambda (output)
(seek output offset SEEK_SET)
(put-bytevector output bv)))
diff --git a/gnu/tests/reconfigure.scm b/gnu/tests/reconfigure.scm
index 928a210a94..52beeef447 100644
--- a/gnu/tests/reconfigure.scm
+++ b/gnu/tests/reconfigure.scm
@@ -260,7 +260,9 @@ bootloader's configuration file."
  ;; test suite, the bootloader installer script is omitted. 'grub-install'
  ;; would attempt to write directly to the virtual disk if the
  ;; installation script were run.
- (test (install-bootloader-program #f #f bootcfg bootcfg-file #f "/")
+ (test
+  (install-bootloader-program #f #f #f bootcfg bootcfg-file #f "/")
+
 
 (define %test-switch-to-system
   (system-test
diff --git a/guix/scripts/system/reconfigure.scm b/guix/scripts/system/reconfigure.scm
index d89caf80fc..b1982b20d2 100644
--- a/guix/scripts/system/reconfigure.scm
+++ b/guix/scripts/system/reconfigure.scm
@@ -204,7 +204,8 @@ services as defined by OS."
 ;;; Bootloader configuration.
 ;;;
 
-(define (install-bootloader-program installer bootloader-package bootcfg
+(define (install-bootloader-program installer disk-installer
+bootloader-package bootcfg
 bootcfg-file device target)
   "Return an executable store item that, upon being evaluated, will install
 BOOTCFG to BOOTCFG-FILE, a target file name, on DEVICE, a file system device,
@@ -246,10 +247,12 @@ BOOTLOADER-PACKAGE."
  ;; a broken installation.
  (switch-symlinks new-gc-root #$bootcfg)
  (install-boot-config #$bootcfg #$bootcfg-file #$target)
- (when #$installer
+ (when (or #$installer #$disk-installer)
(catch #t
  (lambda ()
-   (#$installer #$bootloader-package #$device #$target))
+   (if #$installer
+   (#$installer #$bootloader-package #$device #$target)
+   (#$disk-installer #$bootloader-package 0 #$device)))
  (lambda args
(delete-file new-gc-root)
(match args
@@ -272,11 +275,14 @@ additional configurations specified by MENU-ENTRIES can be selected."
   (let* ((bootloader (bootloader-configuration-bootloader configuration))
  (installer (and run-installer?
  (bootloader-installer bootloader)))
+ (disk-installer (and run-installer?
+  (bootloader-disk-image-installer bootloader)))
  (package (bootloader-package bootloader))
  (device (bootloader-configuration-target configuration))
  (bootcfg-file (bootloader-configuration-file bootloader)))
 (eval #~(parameterize ((current-warning-port (%make-void-port "w")))
   (primitive-load #$(install-bootloader-program installer
+disk-installer
 package
 bootcfg
 bootcfg-file
-- 
2.28.0

bug#44101: Unable to use /dev/disk/by-id/ symlinks with u-boot and guix system reconfigure

[Mathieu Othacehe]

Hello,

> I don't quite understand why that would be the issue here; guix system
> reconfigure works fine when /dev/mmcblkN is specified target in the
> system config.scm, just not when the target is /dev/disk/by-id/...

I don't think it works fine with /dev/mmcblkN. I think the bootloader
configuration file is installed in /boot/extlinux/extlinux.conf when
using /dev/mmcblkN or /dev/disk/, but the bootloader itself is actually
never installed.

> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
> ERROR:
>   1. /o-filename: "/dev/disk/by-id/mmc-SDU64_0xbaf3002e"

Thanks for testing. This is probably a regression in
"write-file-on-device" that fails to open this kind of files. I'll try
to reproduce it locally.

Thanks,

Mathieu