Re: [fr] Moment de convivialité Guix@Paris en janvier… 2024
Bonjour Guix, Quoting Tanguy LE CARROUR (2024-01-11 18:10:19) > Une partie de l'infrastructure d'Easter-eggs est sous le coup d'une attaque > DOS. Nous avons perdu une partie de notre réseau et, comme nous hébergeons > l'April, elle aussi. > […] > Je resterai quand même sur site pour accueillir, celles et ceux qui n'auraient > pas eu ce message ! Merci à ceux qui ont bravé le froid et le DDoS pour venir discuter de Guix et de langages-à-parenthèses™ hier soir ! Et encore désolé pour les conditions précaires. À très bientôt pour la prochaine édition. -- Tanguy
When is check-system run?
Hello, I would like to inquire regarding system tests. There is check-system make target, however it seems like at least some of them are broken. Can someone check whether that is case even on their machine? Do they all pass for you? I already have patches fixing 4 of them, and debugging 5th, but I would like to ask when exactly are the those tests run and where are the failures reported to? Since the breakage sneaked into the master, I assume they are not run as part of the regular patches? Are there any measures the project is planning to take to prevent this from happening again? Thank you and have a nice day, Tomas Volf -- There are only two hard things in Computer Science: cache invalidation, naming things and off-by-one errors. signature.asc Description: PGP signature
Re: Performance of computing cross derivations
On Thu, Jan 11, 2024 at 01:26:35PM +, Christopher Baines wrote: > > Ludovic Courtès writes: > > > Christopher Baines skribis: > > > >> I think you're right, while I send some other changes in #68266, I think > >> it's this change around make-rust-sysroot that has pretty much all the > >> effects on performance. > >> > >> I think the tens of thousands of duplicated packages from cross-base > >> that I was looking at are almost entirely coming from > >> make-rust-sysroot. As Ludo mentions in [1], maybe this has something to > >> do with use of cross- procedures in native-inputs, although I'm not sure > >> that moving those calls out of native-inputs is a correct thing to do. > >> > >> I don't know what the correct approach here is, but I think something > >> needs doing here to address the performance regression. > > > > I probably missed it in the thread: what commit caused the regression, > > and how can I test any changes? I’m willing to help but I missed some > > of the context. > > It's not a pure performance regression, more that in it's current form, > rust cross derivations are very expensive to compute. It's been this way > since cross-compiling was enabled in [1]. > > 1: > https://git.savannah.gnu.org/cgit/guix.git/patch/?id=e604972d9c697302691aeb22e9c50c933a1a3c72 > > I've been looking at data service slowness in processing revisions over > the last few weeks, and I think it's mostly down to this. Looking at the > revision prior to the change [2], computing all the derivations took > around 3 hours, which is ages, but still quick compared to the nearly 9 > hours it took after this change [3]. > > 2: https://data.guix.gnu.org/revision/58bbb38c5bd2e42aab9e9408d8c9d8da3409f178 > 3: https://data.guix.gnu.org/revision/c9e1a72cc27925484635ae01bc4de28bf232689d > > Obviously having more derivations is good and that usually means more > work for the data service, but in this case it seems like things can be > sped up quite a bit. > > For testing locally, I've been computing all the derivations for > i586-pc-gnu, but Efraim also posted a concise command to look at > computing some cross derivations for a subset of rust packages [4]. > > 4: https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00053.html list-all-cargo-build-system-packages is actually a script I have locally that I should probably put in the etc/teams/rust folder. I've attached it in case anyone wants to try it out, or see the speed-up of computing the cross-derivations. -- Efraim Flashner רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted guile -c '(use-modules (gnu packages)(guix packages)(guix build-system)) (display (fold-packages (lambda (package lst) (if (eq? (build-system-name (package-build-system package)) (quote cargo)) (cons package lst) lst)) (list)))' | tr ' ' '\n' | grep \@ signature.asc Description: PGP signature
Re: [fr] Moment de convivialité Guix@Paris en janvier… 2024
Attention, Une partie de l'infrastructure d'Easter-eggs est sous le coup d'une attaque DOS. Nous avons perdu une partie de notre réseau et, comme nous hébergeons l'April, elle aussi. Je vais donc être obligé d'annuler le Guix@Paris de ce soir. 😟 Désolé pour la notification de dernière minute. Je resterai quand même sur site pour accueillir, celles et ceux qui n'auraient pas eu ce message ! Je vous tiendrai au courant de la tenue de la prochaine édition. Librement, Tanguy
Commit Access: Sharlatan Hellseher
Hi Guix! I am happy to have been granted commit access and I am ready to help review pending issues and prepare queued packages for GNU packages in astronomy. I would like to concentrate on the packages covered by the Go, Lisp, Python, and Science teams. I would like to thank the Guix team for allowing me to become a committer member. I am looking forward to continuing our collaboration. If anyone has a good patch review workflow using Emacs, Gnus, and Magit, I would appreciate it ;-) Regards, Sharlatan Hellseher (Oleg) signature.asc Description: PGP signature
An update on ‘core-updates’
Hello Guix! Several of us have been fiddling with the ‘core-updates’ branch for a while. I think there’s now consensus that the branch is really dedicated to core packages and (guix build …) modules, as embodied in the new ‘core-packages’ team¹. We’ve updated GCC 11.x, glibc, binutils, and various packages from (gnu packages base). Notable exceptions are Coreutils, Findutils, sed, and tar; I tried but that’s a bit more work, notably because their variants in commencement.scm would no longer build because their build scripts use sed patterns not supported by Gash-Utils. Long story short: I’d like us to freeze and merge the branch ASAP, notably because the glibc graft on ‘master’ leads to a bad user experience. I’m happy with the current state of the branch and wouldn’t mind postponing remaining upgrades for the next cycle. Thoughts? Remaining work includes: checking that cross-compilation targets still work after the recent Binutils updates, checking i586-gnu (GNU/Hurd) and other platforms, and possibly addressing the Gawk non-determinism issue². Currently package subsets are built here: https://ci.guix.gnu.org/jobset/core-updates https://guix.bordeaux.inria.fr/jobset/guix-core-updates I don’t think I can commit to coordinating the stabilization effort though as I’m busy with other things this month. Would anyone like to take the lead on this? Happy updating! Ludo’. ¹ https://issues.guix.gnu.org/67880 ² https://issues.guix.gnu.org/68378
Re: Performance of computing cross derivations
Ludovic Courtès writes: > Christopher Baines skribis: > >> I think you're right, while I send some other changes in #68266, I think >> it's this change around make-rust-sysroot that has pretty much all the >> effects on performance. >> >> I think the tens of thousands of duplicated packages from cross-base >> that I was looking at are almost entirely coming from >> make-rust-sysroot. As Ludo mentions in [1], maybe this has something to >> do with use of cross- procedures in native-inputs, although I'm not sure >> that moving those calls out of native-inputs is a correct thing to do. >> >> I don't know what the correct approach here is, but I think something >> needs doing here to address the performance regression. > > I probably missed it in the thread: what commit caused the regression, > and how can I test any changes? I’m willing to help but I missed some > of the context. It's not a pure performance regression, more that in it's current form, rust cross derivations are very expensive to compute. It's been this way since cross-compiling was enabled in [1]. 1: https://git.savannah.gnu.org/cgit/guix.git/patch/?id=e604972d9c697302691aeb22e9c50c933a1a3c72 I've been looking at data service slowness in processing revisions over the last few weeks, and I think it's mostly down to this. Looking at the revision prior to the change [2], computing all the derivations took around 3 hours, which is ages, but still quick compared to the nearly 9 hours it took after this change [3]. 2: https://data.guix.gnu.org/revision/58bbb38c5bd2e42aab9e9408d8c9d8da3409f178 3: https://data.guix.gnu.org/revision/c9e1a72cc27925484635ae01bc4de28bf232689d Obviously having more derivations is good and that usually means more work for the data service, but in this case it seems like things can be sped up quite a bit. For testing locally, I've been computing all the derivations for i586-pc-gnu, but Efraim also posted a concise command to look at computing some cross derivations for a subset of rust packages [4]. 4: https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00053.html signature.asc Description: PGP signature
Re: GNU Shepherd 0.10.3 released
On 2024-01-11 13:41:39 +0100, Ludovic Courtès wrote: > Tomas Volf <~@wolfsden.cz> skribis: > > > Otherwise the shepherd would be stuck on shutdown waiting for process named > > > > [mt76-tx phy0] > > > > to terminate with messages along the lines of: > > > > shepherd[1]: waiting for process termination (processes left: (1 678)) > > > > It is a kernel thread as far as I can tell (based on > > https://stackoverflow.com/a/12231039): > > > > $ cd /proc/678 > > $ cat cmdline > > $ readlink exe; echo $? > > 1 > > > > Removing the module mt7921e stops the thread, so shepherd does not wait for > > it. > > Ooooh. > > Then I’m afraid this bug isn’t fixed yet because that code (“waiting for > process termination”) is currently in Guix, not in Shepherd. > > However, ‘processes’, which is what is used here and which is defined in > (guix build syscalls), already checks for kernel threads, though it > does it differently than what I implemented in shepherd: > > (define (kernel? pid) > "Return #t if PID designates a \"kernel thread\" rather than a normal > user-land process." > (let ((stat (call-with-input-file (format #f "/proc/~a/stat" pid) > (compose string-tokenize read-string > ;; See proc.txt in Linux's documentation for the list of fields. > (match stat > ((pid tcomm state ppid pgrp sid tty_nr tty_pgrp flags min_flt > cmin_flt maj_flt cmaj_flt utime stime cutime cstime > priority nice num_thread it_real_value start_time > vsize rss rsslim > (= string->number start_code) (= string->number end_code) _ ...) > ;; Got this obscure trick from sysvinit's 'killall5' program. > (and (zero? start_code) (zero? end_code)) > > It would be great if you could check whether this approach works for > you. Ah, that code indeed returns #f for the pid in question: scheme@(guix-user)> ((@@ (guix build syscalls) kernel?) 688) $1 = #f The stat file: $ cat /proc/688/stat 688 (mt76-tx phy0) S 2 0 0 0 -1 2129984 0 0 0 0 0 0 0 0 -2 0 1 0 964 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 5 1 1 0 0 0 0 0 0 0 0 0 0 0 So the start_code is not zero (I would guess it is -1). I have no idea what that means though. Tomas -- There are only two hard things in Computer Science: cache invalidation, naming things and off-by-one errors. signature.asc Description: PGP signature
Re: GNU Shepherd 0.10.3 released
Hello, Tomas Volf <~@wolfsden.cz> skribis: > Otherwise the shepherd would be stuck on shutdown waiting for process named > > [mt76-tx phy0] > > to terminate with messages along the lines of: > > shepherd[1]: waiting for process termination (processes left: (1 678)) > > It is a kernel thread as far as I can tell (based on > https://stackoverflow.com/a/12231039): > > $ cd /proc/678 > $ cat cmdline > $ readlink exe; echo $? > 1 > > Removing the module mt7921e stops the thread, so shepherd does not wait for > it. Ooooh. Then I’m afraid this bug isn’t fixed yet because that code (“waiting for process termination”) is currently in Guix, not in Shepherd. However, ‘processes’, which is what is used here and which is defined in (guix build syscalls), already checks for kernel threads, though it does it differently than what I implemented in shepherd: (define (kernel? pid) "Return #t if PID designates a \"kernel thread\" rather than a normal user-land process." (let ((stat (call-with-input-file (format #f "/proc/~a/stat" pid) (compose string-tokenize read-string ;; See proc.txt in Linux's documentation for the list of fields. (match stat ((pid tcomm state ppid pgrp sid tty_nr tty_pgrp flags min_flt cmin_flt maj_flt cmaj_flt utime stime cutime cstime priority nice num_thread it_real_value start_time vsize rss rsslim (= string->number start_code) (= string->number end_code) _ ...) ;; Got this obscure trick from sysvinit's 'killall5' program. (and (zero? start_code) (zero? end_code)) It would be great if you could check whether this approach works for you. (I had completely forgotten about this code. Funny thing is this one was inspired by sysvinit, whereas the one in Shepherd was inspired by systemd. A sign of times!) Ludo’.
Re: Performance of computing cross derivations
Hi, Christopher Baines skribis: > I think you're right, while I send some other changes in #68266, I think > it's this change around make-rust-sysroot that has pretty much all the > effects on performance. > > I think the tens of thousands of duplicated packages from cross-base > that I was looking at are almost entirely coming from > make-rust-sysroot. As Ludo mentions in [1], maybe this has something to > do with use of cross- procedures in native-inputs, although I'm not sure > that moving those calls out of native-inputs is a correct thing to do. > > I don't know what the correct approach here is, but I think something > needs doing here to address the performance regression. I probably missed it in the thread: what commit caused the regression, and how can I test any changes? I’m willing to help but I missed some of the context. Thanks, Ludo’.
Re: [swh-devel] Call for public review - SWH Nix/GNU Guix stack
Hi Benoit and all!
(Cc: guix-devel rather than gnu-system-discuss.)
Benoit Chauvet skribis:
> Regarding the Nix/GNU Guix stack, Software Heritage will soon be ready to
> support the
> ingestion of specific versioned files, tarballs, git, hg, svn source code
> listed in their respective manifests [1] (as origins). The new lister (and
> extra loaders, namely
> {Content|Directory|GitCheckout|SvnExport|HgCheckout}Loader) have been
> deployed in our staging infrastructure [2].
Excellent! I believe this addresses a problem we recently reported
regarding tarballs published with our own content-addressed URLs, which
look like:
https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas
My understanding is that so far these URLs were ignored by the
lister/loader because they didn’t end in *.tar.*.⁰
> The initial NixGuix loader (currently in production) lists and loads
> origins from a manifest, ignoring the specific origins mentioned above. The
> new stack will be able to ingest those origins. It will also optionally
> associate, if present, a NAR hash (specific intrinsic identifier to Nix and
> Guix) to what’s called an ExtID (SWH side).
> Regarding the SWH API reading side of the ExtID though is a work to be done.
>
> On staging, we have currently ingested origins that were listed from the
> GNU Guix manifest [3].
>
> We have already improved the implementations after discussing multiple
> limitations encountered along the way with the Guix community [4].
I’m sure Simon Tournier (Cc’d) already discussed with others at SWH how
crucial it is for us to be able to query content by nar hash.
Essentially, it would fill the gap that currently prevents us from
retrieving Subversion checkouts from SWH¹ and more generally complicates
retrieval of anything not referenced by a Git hash. So obviously, we’re
looking forward to that ExtID interface for SWH.
Thanks for sharing this status update, these are all exciting news and
perspectives!
Ludo’.
⁰ https://issues.guix.gnu.org/39885#15-lineno60
¹ https://issues.guix.gnu.org/43442#13-lineno37
Re: using shepherd's (shepherd service repl) in guix system
On Tue 09 Jan 2024 23:36, Ludovic Courtès writes: > scheme@(shepherd-user)> (+ 2 3) > $6 = 5 > scheme@(shepherd-user)> (lookup-service 'repl) > ;;; socket:56:1: warning: possibly unbound variable `lookup-service' > ice-9/boot-9.scm:1685:16: In procedure raise-exception: > Unbound variable: lookup-service > > Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. > scheme@(shepherd-user) [1]> While reading expression: > Attempt to suspend fiber within continuation barrier > scheme@(shepherd-user) [1]> While reading expression: > Attempt to suspend fiber within continuation barrier > > > The problem here is that the exception is raised C code (in > libguile/modules.c, then indirectly calling ‘scm_error’), which makes it > a “continuation barrier” (meaning that it prevents Fibers from switching > contexts among fibers in the shepherd process, hence the error above.) Interesting. I guess the short-term fix is to only enter the debugger if suspendable-continuation? is true for the fibers prompt, and otherwise just print a backtrace. Andy
