Re: Profiling of man-db database generation with zlib vs zstd
Hello, Maxim Cournoyer writes: > Hi Ludovic! > > Ludovic Courtès writes: > >> Hi, >> >> Maxim Cournoyer skribis: >> >>> scheme@(guix man-db)> ,profile (define a (mandb-entries >>> "/gnu/store/jgc63dxvpd8zq0p8al71x07a02qj8i1b-man-pages-5.13/share/man")) >>> % cumulative self >>> time seconds seconds procedure >>> 20.95 1.98 1.75 gdbm.scm:122:11 >>> 20.95 1.75 1.75 string-tokenize >>> 19.37 3.61 1.62 set-procedure-property! >>> 6.72 0.56 0.56 ice-9/eval.scm:604:6 >>> 4.35 0.36 0.36 %read-line >>> 4.35 0.36 0.36 anon #xa8e0b0 >>> 2.77 0.23 0.23 apply-smob/1 >>> 1.58 11.51 0.13 ice-9/eval.scm:292:11 >> >> Could it be that man-db.scm is being interpreted, hence eval.scm and >> ‘set-procedure-property!’ here? (Running ‘make’ doesn’t compile it.) OK, I've now compiled guix/man-db.scm with -O3, generated a profile with ~5300 manuals (guix shell --no-grafts --pure man-db perl@5.14 man-pages libx11:doc gnutls-dane:doc my-tcl) on core-updates with my local changes to compress man pages with zstd and profiled: --8<---cut here---start->8--- scheme@(guix man-db)> ,profile (define a (mandb-entries "/gnu/store/jp1kjkz5m116r960gvjk1sj4b0fkb0ip-profile/share/man")) % cumulative self time seconds seconds procedure 69.37 16.19 16.15 %read-line 14.14 3.31 3.29 string-tokenize 5.76 1.34 1.34 gdbm.scm:122:11 2.44 22.75 0.57 guix/man-db.scm:170:4 1.92 1.71 0.45 make-bytevector 1.57 0.37 0.37 anon #x209d110 0.52 0.14 0.12 open-file 0.35 16.96 0.08 ice-9/rdelim.scm:193:0:read-line 0.35 0.08 0.08 stringpointer 0.26 0.06 0.06 assv-ref 0.26 0.06 0.06 readdir 0.26 0.06 0.06 equal? 0.26 0.06 0.06 get-bytevector-n! 0.17 0.67 0.04 zstd.scm:234:2:read! 0.17 0.04 0.04 string-append 0.17 0.04 0.04 close-port 0.17 0.04 0.04 bytevector-u64-native-set! 0.17 0.04 0.04 stat 0.17 0.04 0.04 guix/man-db.scm:139:19 0.09 1.83 0.02 zstd.scm:216:0:make-zstd-input-port 0.09 0.33 0.02 ice-9/ftw.scm:445:2:loop 0.09 0.12 0.02 system/foreign.scm:150:0:write-c-struct 0.09 0.04 0.02 system/foreign.scm:167:0:read-c-struct 0.09 0.02 0.02 pointer->bytevector 0.09 0.02 0.02 regexp-exec 0.09 0.02 0.02 string 0.09 0.02 0.02 sizeof 0.09 0.02 0.02 basename 0.09 0.02 0.02 string-contains 0.09 0.02 0.02 lstat 0.09 0.02 0.02 force 0.09 0.02 0.02 anon #x209d0d8 0.00 34249.92 0.00 ice-9/boot-9.scm:220:5:map1 0.00 23.28 0.00 :3:9 0.00 17.17 0.00 ice-9/ports.scm:429:0:call-with-port 0.00 7.46 0.00 zstd.scm:273:0:call-with-zstd-input-port 0.00 1.34 0.00 anon #x208c5f0 0.00 1.34 0.00 %after-gc-thunk 0.00 0.73 0.00 guix/man-db.scm:129:0:read-synopsis 0.00 0.67 0.00 %read-line 0.00 0.28 0.00 guix/man-db.scm:205:0:mandb-entries 0.00 0.16 0.00 srfi/srfi-1.scm:452:2:fold 0.00 0.16 0.00 anon #x208c57c 0.00 0.16 0.00 guix/build/utils.scm:487:0:find-files 0.00 0.14 0.00 guix/man-db.scm:155:0:man-page->entry 0.00 0.12 0.00 system/foreign.scm:182:0:make-c-struct 0.00 0.08 0.00 sort 0.00 0.04 0.00 filter 0.00 0.04 0.00 close-port 0.00 0.04 0.00 ice-9/boot-9.scm:1971:6 0.00 0.04 0.00 zstd.scm:208:4 0.00 0.04 0.00 guix/build/utils.scm:503:28 0.00 0.04 0.00 string-map 0.00 0.02 0.00 ice-9/boot-9.scm:1689:4:with-exception-handler 0.00 0.02 0.00 system/foreign.scm:187:0:parse-c-struct 0.00 0.02 0.00 guix/build/utils.scm:484:4 --- Sample count: 1146 Total time: 23.282459186 seconds (7.318256931 seconds in GC) --8<---cut here---end--->8--- It still shows that parsing the files is much more expensive than decompressing them. This is also true of zlib-compressed manuals; here's the same experiment on master: --8<---cut here---start->8--- $ guix shell --no-grafts --pure man-db perl@5.14 man-pages libx11:doc gnutls-dane:doc tcl [env]$ echo $GUIX_ENVIRONMENT && exit /gnu/store/qqd7d22wf9d220prkm682yypybpr7df4-profile $ guix shell -D guix guile-gdbm-ffi guile-zstd [env]$ guild compile -O3 guix/man-db.scm `/home/maxim/.cache/guile/ccache/3.0-LE-8-4.6/home/maxim/src/guix-master/guix/man-db.scm.go' [env]$ ./pre-inst-env guix repl scheme@(guix-user)> ,m (guix man-db) scheme@(guix man-db)> ,profile (define a (mandb-entries "/gnu/store/qqd7d22wf9d220prkm682yy
Re: Hardened toolchain
Mar 26, 2022, 19:33 by kias...@tutanota.com:
> Hi Simon,
>
> Mar 25, 2022, 22:54 by zimon.touto...@gmail.com:
>
>> Hi,
>>
>> On Fri, 25 Mar 2022 at 20:39, kias...@tutanota.com wrote:
>>
>>> the middle of guix build -f hardened.scm
>>> building /gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv...
>>> Backtrace:
>>> In ice-9/eval.scm:
>>> 217:50 19 (lp (# ?))
>>> 217:50 18 (lp (# ?))
>>> 217:50 17 (lp (# ?))
>>> 217:50 16 (lp (# ?))
>>> 217:50 15 (lp (# ?))
>>> 217:50 14 (lp (# ?))
>>> 217:50 13 (lp (# ?))
>>> 217:50 12 (lp (# ?))
>>> 217:50 11 (lp (# ?))
>>> 217:50 10 (lp (# ?))
>>> 217:50 9 (lp (# ?))
>>> 217:50 8 (lp (# ?))
>>> 217:50 7 (lp (# ?))
>>> 217:50 6 (lp (# ?))
>>> 217:50 5 (lp (# ?))
>>> 217:50 4 (lp (# ?))
>>> 217:33 3 (lp (# ?))
>>> 159:9 2 (_ #(#(# #f) #f))
>>> 159:9 1 (_ #(#(# #f) #f))
>>> In unknown file:
>>> 0 (string-append "LDFLAGS=" "-Wl,-rpath=" #f "/lib " "-W?" ?)
>>>
>>> ERROR: In procedure string-append:
>>> In procedure string-append: Wrong type (expecting string): #f
>>> builder for `/gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv'
>>> failed with exit code 1
>>> build of /gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv failed
>>> View build log at
>>> '/var/log/guix/drvs/1n/lrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv.gz'.
>>> guix build: error: build of
>>> `/gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv' failed
>>> the middle of guix build -f hardened.scm
>>>
Here's a smaller example that has the same error:
===the file===
(use-modules (gnu)
(guix)
(guix packages))
(use-package-modules gcc base commencement)
(package-with-c-toolchain gcc `(("toolchain" ,(make-gcc-toolchain gcc
===the file===
===try to build it===
In unknown file:
0 (string-append "LDFLAGS=" "-Wl,-rpath=" #f "/lib " "-W?" ?)
ERROR: In procedure string-append:
In procedure string-append: Wrong type (expecting string): #f
===try to build it===
The gcc package already exists! Why can't I build gcc with itself?
>> You are creating a cycle, no? It is not a DAG and so the transformation
>> fails, no?
>>
> Oh I didn't notice that. The example makes sense too.
>
>> For instance, this:
>>
>> --8<---cut here---start->8---
>> (use-modules (guix packages)
>> (gnu packages gcc)
>> (gnu packages base))
>>
>> (define make-gcc-toolchain
>> (@@ (gnu packages commencement) make-gcc-toolchain))
>>
>> (define gcc-bis
>> (package
>> (inherit gcc)
>> (version (string-append (package-version gcc) "-bis"
>>
>> (define gcc-toolchain-bis
>> (make-gcc-toolchain gcc-bis glibc))
>>
>> (define (package-with-c-toolchain-bis package)
>> (package-with-c-toolchain
>> package `(("toolchain" ,gcc-toolchain-bis
>>
>>
>> (package-with-c-toolchain-bis gcc-bis)
>> --8<---cut here---end--->8---
>>
>> fails with the same message. There is bootstrapping issue: the binary
>> of gcc-bis is required to compile the source of gcc-bis; where does come
>> from such binary of gcc-bis?
>>
>>
>> Considering your use case, you need:
>>
>> - gcc considered as binary seed
>>
>> - use this binary gcc with the hardened options to compile the source
>> of GCC; resulting to the binary gcc-hardened-1
>>
>> - use this binary gcc-hardened-2 with the hardened options to recompile
>> the source of GCC; resulting to the binary gcc-hardened-2
>>
>> - if checksum(gcc-hardened-1) == checksum(gcc-hardened-2)
>> then use this binary to define a new toolchain
>> else reach the fixed point
>>
>> fixed point: use this binary gcc-hardened-{n-1} to compile the source of
>> GCC and output the binary gcc-hardened-{n}; compare the checksum of
>> the binary {n-1} and {n} and repeat until equality is reached.
>>
> Just so I understand, in other (imperative) words:
>
> gcc-hardened-1 = gcc-hardened built with regular gcc
> gcc-hardened-2 = gcc-hardened built with gcc-hardened-1
> n = 1
> while checksum(gcc-hardened-{n}) != checksum(gcc-hardened-{n+1}):
> gcc-hardened-{n+1} = gcc-hardened built with gcc-hardened-{n}
> n++
> define the new toolchain with gcc-hardened-{n+1}
>
>
>> Guix is not auto-magically resolving the fixed-point, i.e., it does not
>> unroll the cycle by magic. :-) You have to do it manually or write code
>> for automatise the process; described above.
>>
> Thanks, are there any examples in the code base that would be a good
> reference?
>
>>
>> Hope that helps.
>>
>> Cheers,
>> simon
>>
>
>
Re: XDG_CONFIG_DIR and dotfiles handling in Guix Home
Hi Andrew, Andrew Tropin skribis: > Another downside of home-files-service-type is that for historical > reasons it requires to use a path without leading dot and adds it > automatically during activation, which makes it impossible to create a > directory or file without leading dot with it, for example ~/wow. This > behavior is non-intuitive, so in two weeks we will make a breaking > change to fix it. Sounds like a great plan to me; looking forward to that. Thank you! Ludo’.
Re: The Shepherd on Fibers
Hi Maxim! Maxim Cournoyer skribis: >> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency. > > Very exciting work! I can't wait to have my 5 min boot reduced (mostly > busy-waiting on the child-hurd VM to start) ;-). Same here! The childhurd service will need some help as it’s currently written in a way that would block. > Interesting; in a (unmerged) patch I had made to mcron, I had extended > the select loop in a way that it'd detect available output and use a > suspendable port to accomplish this; perhaps Fibers would be a > cleaner/more straightforward way to accomplish this. Also, I was > particularly happy about how flexible the timestamp/metadata prefixed to > each line could be configured by a user; you may want to steal ideas > from it! [0] > > [0] https://lists.gnu.org/archive/html/bug-mcron/2021-08/msg8.html Oh nice! I hope that patch will eventually make it into mcron (or we could use our own version!). For the timestamp format there’s currently a parameter, which is not exposed, and defaults to syslog style. > I see. I was puzzle by this configure.ac check: > > +dnl Check for extra dependencies. > +GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)]) > +if test "x$have_fibers" != "xyes"; then > + AC_MSG_ERROR([Fibers is missing; please install it.]) > +fi > + > +dnl Make sure Fibers does not create POSIX threads: since shepherd > +dnl forks, it must be single-threaded. > +AC_CACHE_CHECK([whether Fibers might create POSIX threads], > + [ac_cv_fibers_creates_pthreads], > + [GUILE_CHECK([retval], > +[(use-modules (fibers)) > + (set! (@ (ice-9 threads) call-with-new-thread) > + (lambda _ (throw 'new-thread!))) > + (run-fibers (lambda () (spawn-fiber (lambda () 1))) > + #:parallelism 1 #:hz 0)]) > + if test "$retval" = 0; then > + ac_cv_fibers_creates_pthreads="no" > + else > + ac_cv_fibers_creates_pthreads="yes" > + fi]) > +if test "x$ac_cv_fibers_creates_pthreads" = "xyes"; then > + AC_MSG_ERROR([Fibers creates POSIX threads behind our back; aborting.]) > +fi > > > In which scenario would Fibers create unwanted POSIX threads? It shouldn’t happen with #:parallelism 1 #:hz 0. Here I’m just being super defensive: it’s PID 1 after all, and if future Fibers versions break that assumption for some reason, I’d rather detect it early. > 1. In fae59fb * build: Capture the source and object directories of > Fibers. > > Why must we even capture the Fibers source/compiled objects directories? We don’t have to, but if we don’t do it here, then the ‘shepherd’ package definition will have to use ‘wrap-program’ to do it for us. Since there’s just one dependency, I thought we might as well do it upstream. > 2. nitpick; in "service: 'read-pid-file' no longer blocks": > > -;; Copyright (C) 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 > Ludovic Courtès > +;; Copyright (C) 2013-2022 Ludovic Courtès > > While I agree this reads better, I've recently stumbled in info > '(maintain) Copyright Notices' on: > > You can use a range (‘2008-2010’) instead of listing individual years >(‘2008, 2009, 2010’) if and only if: 1) every year in the range, >inclusive, really is a “copyrightable” year that would be listed >individually; _and_ 2) you make an explicit statement in a ‘README’ file >about this usage. > > So you'd want to add an explicit statement in the README about it. Yes, will do (it’s typical FSF pedantry; I remember the time it took before this anecdotal-looking paragraph would end up in that document, as if software freedom was at stake :-)). > 3. I was a bit skeptical about the ability to configure the encoding of > the log file; in which situation would switching it to something else > than UTF-8 be useful? I agree; the log files produced by shepherd are always UTF-8. What’s configurable is the encoding of the data produced by daemons. It’s probably rarely useful, but if you have a daemon that writes, say, ISO-8859-1 strings, you can say so. That way, the output of that daemon is properly decoded as ISO-8859-1; the log file produced by shepherd remains UTF-8 no matter what. Thanks for taking a look! Since that initial message I added support for inetd-style service startup. Among the services I use, it could be use for sshd (OpenSSH), bitlbee, and dicod. More on that later… Ludo’.
Re: Hardened toolchain
Hi Simon,
Mar 25, 2022, 22:54 by zimon.touto...@gmail.com:
> Hi,
>
> On Fri, 25 Mar 2022 at 20:39, kias...@tutanota.com wrote:
>
>> the middle of guix build -f hardened.scm
>> building /gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv...
>> Backtrace:
>> In ice-9/eval.scm:
>> 217:50 19 (lp (# ?))
>> 217:50 18 (lp (# ?))
>> 217:50 17 (lp (# ?))
>> 217:50 16 (lp (# ?))
>> 217:50 15 (lp (# ?))
>> 217:50 14 (lp (# ?))
>> 217:50 13 (lp (# ?))
>> 217:50 12 (lp (# ?))
>> 217:50 11 (lp (# ?))
>> 217:50 10 (lp (# ?))
>> 217:50 9 (lp (# ?))
>> 217:50 8 (lp (# ?))
>> 217:50 7 (lp (# ?))
>> 217:50 6 (lp (# ?))
>> 217:50 5 (lp (# ?))
>> 217:50 4 (lp (# ?))
>> 217:33 3 (lp (# ?))
>> 159:9 2 (_ #(#(# #f) #f))
>> 159:9 1 (_ #(#(# #f) #f))
>> In unknown file:
>> 0 (string-append "LDFLAGS=" "-Wl,-rpath=" #f "/lib " "-W?" ?)
>>
>> ERROR: In procedure string-append:
>> In procedure string-append: Wrong type (expecting string): #f
>> builder for `/gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv'
>> failed with exit code 1
>> build of /gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv failed
>> View build log at
>> '/var/log/guix/drvs/1n/lrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv.gz'.
>> guix build: error: build of
>> `/gnu/store/1nlrgg5ryl486haw0kdqnbp4wa17lhwh-gcc-10.3.0.drv' failed
>> the middle of guix build -f hardened.scm
>>
>
> You are creating a cycle, no? It is not a DAG and so the transformation
> fails, no?
>
Oh I didn't notice that. The example makes sense too.
> For instance, this:
>
> --8<---cut here---start->8---
> (use-modules (guix packages)
> (gnu packages gcc)
> (gnu packages base))
>
> (define make-gcc-toolchain
> (@@ (gnu packages commencement) make-gcc-toolchain))
>
> (define gcc-bis
> (package
> (inherit gcc)
> (version (string-append (package-version gcc) "-bis"
>
> (define gcc-toolchain-bis
> (make-gcc-toolchain gcc-bis glibc))
>
> (define (package-with-c-toolchain-bis package)
> (package-with-c-toolchain
> package `(("toolchain" ,gcc-toolchain-bis
>
>
> (package-with-c-toolchain-bis gcc-bis)
> --8<---cut here---end--->8---
>
> fails with the same message. There is bootstrapping issue: the binary
> of gcc-bis is required to compile the source of gcc-bis; where does come
> from such binary of gcc-bis?
>
>
> Considering your use case, you need:
>
> - gcc considered as binary seed
>
> - use this binary gcc with the hardened options to compile the source
> of GCC; resulting to the binary gcc-hardened-1
>
> - use this binary gcc-hardened-2 with the hardened options to recompile
> the source of GCC; resulting to the binary gcc-hardened-2
>
> - if checksum(gcc-hardened-1) == checksum(gcc-hardened-2)
> then use this binary to define a new toolchain
> else reach the fixed point
>
> fixed point: use this binary gcc-hardened-{n-1} to compile the source of
> GCC and output the binary gcc-hardened-{n}; compare the checksum of
> the binary {n-1} and {n} and repeat until equality is reached.
>
Just so I understand, in other (imperative) words:
gcc-hardened-1 = gcc-hardened built with regular gcc
gcc-hardened-2 = gcc-hardened built with gcc-hardened-1
n = 1
while checksum(gcc-hardened-{n}) != checksum(gcc-hardened-{n+1}):
gcc-hardened-{n+1} = gcc-hardened built with gcc-hardened-{n}
n++
define the new toolchain with gcc-hardened-{n+1}
> Guix is not auto-magically resolving the fixed-point, i.e., it does not
> unroll the cycle by magic. :-) You have to do it manually or write code
> for automatise the process; described above.
>
Thanks, are there any examples in the code base that would be a good reference?
>
> Hope that helps.
>
> Cheers,
> simon
>
Re: The Shepherd on Fibers
Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]: > Thoughts? To avoid accidentally using threads, perhaps Shepherd could do (set! call-with-new-thread (lambda () (error "multi-threading is not supported in Shepherd"))) ? Greetings, Maxime. signature.asc Description: This is a digitally signed message part
A lexical use-modules?
Hi, [ CC'ing guix-devel@ because this functionality could be useful in Guix package definitions ] Currently, use-modules cannot be meaningfully used inside procedures, unless '(current-module)' is always the module in which the procedure is defined. I wondered if some kind of 'lexical use-modules' was possible, with sufficient macroology and module reflection, and it looks like it is: (use-modules (srfi srfi-1)) (define-syntax use-module/lexical ;; todo: integrate into (use-modules ...)? (lambda (s) (syntax-case s () ((_ foo) (let* ((module-name (syntax->datum #'foo)) (interface (resolve-interface module-name))) (define (binding->import name variable) (define name-syntax (datum->syntax s name)) #`(define-syntax #,name-syntax (identifier-syntax (@ foo #,(datum->syntax #'irrelevant name) #`(begin #,@(module-map binding->import interface))) (define &exception 'top-level) (let () (use-module/lexical (ice-9 exceptions)) (pk 'inner &exception raise-continuable)) ;;; (inner # #) (pk 'outer &exception) ;;; (outer top-level) (pk 'unbound-variable raise-continuable) ;; a backtrace! Limitation: things like (define (foo) (use-module/lexical (platform-specific-constants)) (if on-linux? CONSTANT_ONLY_DEFINED_ON_LINUX CONSTANT_ONLY_DEFINED_ON_HURD)) won't work in cross-compilation contexts. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
pelzflorian (Florian Pelz) schreef op za 26-03-2022 om 17:52 [+0100]: > Oh yes, I had not tried that. Thank you Maxime. Still with unchanged > original shepherd package, I get exactly the same > shepherd-avahi-daemon.go.drv.gz build failure iff I set (shepherd > new-shepherd). Therefore I ask how Ludo or perhaps you tested it. > Perhaps you have a proper new-shepherd package definition. I have not tested the wip-fibers variant of Shepherd with Guix. However, looking at scm->go (the procudure resonsible for compiling) in (gnu services shepherd): (with-extensions (list shepherd) [...]) , I'm wondering if the 'propagated-inputs' of 'shepherd' are respected here. Looking at (guix gexp), I don't see any mention of propagation anywhere, so I guess not (unverified). Maybe that's why (fibers) could not be found? For completeness, you could try inheriting from 'shepherd' and using git-reference/git-checkout instead of the copy-build-system (package (inherit shepherd) (source (git-checkout (url "/home/florian/...") (commit ... which would be closer to how shepherd is usually compiled. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
On Sat, Mar 26, 2022 at 07:27:00PM +0800, Zhu Zihao wrote: > > IIUC, After fork and execute the program, shepherd will wait for the pid > file to appear. This is a block operation. > > Shepherd on Fiber make this operation non-blocking (because the pid file > is created by Linux kernel, Shepherd just need to wait for it), every > service activation runs on its own fiber. after fork+exec, shepherd > suspend current fiber, andspawn another fiber to start another service > immediately. > > But finally all fibers are co-operatively scheduled on 1 thread. Thank you Zhu for explaining! I had not understood waiting for the PID file is the issue solved by fibers. Regards, Florian
Re: The Shepherd on Fibers
On Sat, Mar 26, 2022 at 12:59:20PM +0100, Maxime Devos wrote: > Grafting is not necessary. See (guix)Shepherd Services: > > >;; Use own Shepherd package. >(essential-services > (modify-services (operating-system-default-essential-services > this-operating-system) > (shepherd-root-service-type config => (shepherd-configuration > (inherit config) > (shepherd my- > shepherd)) > Oh yes, I had not tried that. Thank you Maxime. Still with unchanged original shepherd package, I get exactly the same shepherd-avahi-daemon.go.drv.gz build failure iff I set (shepherd new-shepherd). Therefore I ask how Ludo or perhaps you tested it. Perhaps you have a proper new-shepherd package definition. Regards, Florian
Re: Packaging rust-analyzer is not necessary.
Hi Paul And Maxime, > Even if you didn't succeed at updating _all_ dependencies, if you > have patches for some of them, please send them. It will help people > in the future with updating rust-analyzer or other rust packages. I had a patchset (here: https://issues.guix.gnu.org/46162) adding rust-analyzer and the rest of the other tools that come part of the rust tree. I think it would not be too much work to add if added as outputs/companion packages to rust itself. > For many people, a vaguely recent-ish version would be sufficient. At > least, that's the case for C, GCC and Clang. > ... > It might be possible to do "cargo xtask install --server", but many > advantages of Guix would be lost. That worked for me. I was using guix' rust tooling for my job. I much prefer using guix over rustup/cargo. I just had to patch rust a lot and my patches haven't made it in (yet?). > Indeed, e.g. it would be nice to figure out how to eliminate #:skip- > build?, replace #:cargo-inputs by regular inputs, figure out how to > stop having to package multiple versions of the same package. My gut feeling is still that the functional package managers need to collaborate with the rust/cargo team. Rust itself just does not lend itself well to the model. It is a shame since we share many of the same goals (reproducibility and reliability come to the top of mind). Kindly, John
Re: Packaging rust-analyzer is not necessary.
Paul Alesius schreef op vr 25-03-2022 om 09:05 [+0100]: > I've tried to produce a patch for the latest version, and it has > probably hundreds of dependencies that need to be updated. Even if you didn't succeed at updating _all_ dependencies, if you have patches for some of them, please send them. It will help people in the future with updating rust-analyzer or other rust packages. And is it necessary to update _all_ dependencies? For comparison, graphical applications usually compile just fine even if gtk+ or glibc are somewhat old. > In addition to that, rust-analyzer is under heavy development and > there is a release every month. Many people will want to use the > nightly version too. For many people, a vaguely recent-ish version would be sufficient. At least, that's the case for C, GCC and Clang. > As long as there is rust and rust-cargo in Guix, then rust-analyzer > can be easily compiled and installed from git with "cargo xtask > install --server" using the rust-cargo system. > > I'd suggest that a Guix package for rust-analyzer is not needed, > especially due to the excessive time required to update its package > definition and all of the vendored dependency crates, What is the point of a distribution if it not distributing packages? Removing a package for being ~two months old and pointing users to Rust's equivalent of "curl ... | bash -" instead seems a disservice to users to me. If I just do ‘cargo xtask install --server", how do I know if it isn't bundling anything or containing malware? Modifying the source code seems also non-trivial, compared to Guix where one can do things like --with-patch. What about statelessness, reproducibility, time- machine, the SWH fallback and sharing substitutes on the local network? It might be possible to do "cargo xtask install --server", but many advantages of Guix would be lost. > and focus should instead be on rust and rust-cargo. Indeed, e.g. it would be nice to figure out how to eliminate #:skip- build?, replace #:cargo-inputs by regular inputs, figure out how to stop having to package multiple versions of the same package. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
PipeWire as a PulseAudio replacement (was Re: The Shepherd on Fibers)
Hello Brendan, Brendan Tildesley writes: > I would like to replace pulseaudio with pipewire as the default in > %desktop-services, the only hurdle is how to launch the user daemons in all > the > different desktop configurations one might use. Other distros use systemd's > socket activation to magically launch pipewire. > > Otherwise XDG autostarts or some kind of guix home service could > launch it? The main issue for PipeWire is that it really doesn't support running a single system-wide daemon as we're doing with PulseAudio, so you would need to launch it through XDG autostart, guix home or something similar like you said! One thing that I think is blocking right now is that PipeWire (or actually WirePlumber, I don't remember) would need to see some environment variables set by other user services eg. session D-Bus or even the compositor, which isn't possible yet with Shepherd. Best, -- Josselin Poiret
Packaging rust-analyzer is not necessary.
Rust analyzer is a language server for the Rust programming language. I've tried to produce a patch for the latest version, and it has probably hundreds of dependencies that need to be updated. In addition to that, rust-analyzer is under heavy development and there is a release every month. Many people will want to use the nightly version too. As long as there is rust and rust-cargo in Guix, then rust-analyzer can be easily compiled and installed from git with "cargo xtask install --server" using the rust-cargo system. I'd suggest that a Guix package for rust-analyzer is not needed, especially due to the excessive time required to update its package definition and all of the vendored dependency crates, and focus should instead be on rust and rust-cargo. With regards, - Paul
Re: The Shepherd on Fibers
Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]: Thoughts? From service.scm: > + (define (sleep* n) > +;; In general we want to use (@ (fibers) sleep) to yield to the > scheduler. > +;; However, this code might be non-suspendable--e.g., if the user calls > +;; the 'start' method right from their config file, which is loaded with > +;; 'primitive-load', which is a continuation barrier. Thus, this variant > +;; checks whether it can suspend and picks the right 'sleep'. > +(if (yield-current-task) > +(begin > + (set! sleep* (@ (fibers) sleep)) > + (sleep n)) > +(begin > + (set! sleep* (@ (guile) sleep)) > + ((@ (guile) sleep) n Instead of working around this limitation of Fiber's 'sleep', why not modify Fiber's sleep to detect if it is from a suspendable context or not, and depending on that, use Guile's sleep/usleep or the perform-operation+sleep-operation? Additionally, the set! can be avoided here in favour of a loop variable: (define start ...) (define (sleep n) ;; In general [...] (if (yield-current-task) (begin ((@ (fibers) sleep) n) ((@ (fibers) sleep) n)) (begin ((@ (guile) sleep) n) ((@ (guile) sleep) n (let loop ((sleep sleep)) (define (try-again) ... (loop (sleep 1))) (catch ...)) FWIW, the delay can be decreased here, if Guile's sleep is replaced by an appropriate use of 'usleep'. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]: > Thoughts? What if 'exec-command' in 'fork+exec-command' fails? Is it still the case that the exception bubbles up, eventually causing the socket file to be deleted? Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]: > [...] > Thoughts? To help translators, I would use positional arguments in + (l10n "Accepted connection on ~a from ~:[~a~;~*local process~].") Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]: > Fibers is used in a single-threaded fashion, which is the main > constraint for shepherd since it forks. That also means that fibers > cannot be preempted, so it’s fully cooperative scheduling. When hz!=0, guile-fibers uses with-interrupts/thread-cputime which uses threads. However, when (provided? 'threads) is false, it uses 'with-interrupts/sigprof', which uses signal handlers and SIGPROF instead. So with some small tweaks to guile-fibers, it should be feasible to have singly-threaded, preemptive scheduling. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
pelzflorian (Florian Pelz) schreef op za 26-03-2022 om 12:06 [+0100]: > * Lastly I grafted > > (define-public shepherd > (package > (name "shepherd") > (replacement new-shepherd) > … Grafting is not necessary. See (guix)Shepherd Services: ;; Use own Shepherd package. (essential-services (modify-services (operating-system-default-essential-services this-operating-system) (shepherd-root-service-type config => (shepherd-configuration (inherit config) (shepherd my- shepherd)) signature.asc Description: This is a digitally signed message part
Re: The Shepherd on Fibers
IIUC, After fork and execute the program, shepherd will wait for the pid file to appear. This is a block operation. Shepherd on Fiber make this operation non-blocking (because the pid file is created by Linux kernel, Shepherd just need to wait for it), every service activation runs on its own fiber. after fork+exec, shepherd suspend current fiber, andspawn another fiber to start another service immediately. But finally all fibers are co-operatively scheduled on 1 thread. "pelzflorian (Florian Pelz)" writes: > On Sat, Mar 26, 2022 at 07:09:12PM +0800, Zhu Zihao wrote: >> > So shepherd service authors still cannot write blocking code but need >> > to yield? >> >> IMO, service should not block service, if they have something important >> to do before running a program, why not fork first and do it in the >> child-process (maybe wrap the actual startup program in Guile script)? > > In other words, all stays the same as without fibers? > > Regards, > Florian -- Retrieve my PGP public key: gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F Zihao signature.asc Description: PGP signature
Re: The Shepherd on Fibers
On Sat, Mar 26, 2022 at 07:09:12PM +0800, Zhu Zihao wrote: > > So shepherd service authors still cannot write blocking code but need > > to yield? > > IMO, service should not block service, if they have something important > to do before running a program, why not fork first and do it in the > child-process (maybe wrap the actual startup program in Guile script)? In other words, all stays the same as without fibers? Regards, Florian
Re: The Shepherd on Fibers
Zhu Zihao writes: > [[PGP Signed Part:Undecided]] >> So shepherd service authors still cannot write blocking code but need >> to yield? > > IMO, service should not block service, if they have something important > to do before running a program, why not fork first and do it in the > child-process (maybe wrap the actual startup program in Guile script)? service should not block service ^^^ daemon -- Retrieve my PGP public key: gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F Zihao signature.asc Description: PGP signature
Re: The Shepherd on Fibers
> So shepherd service authors still cannot write blocking code but need > to yield? IMO, service should not block service, if they have something important to do before running a program, why not fork first and do it in the child-process (maybe wrap the actual startup program in Guile script)? -- Retrieve my PGP public key: gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F Zihao signature.asc Description: PGP signature
Re: The Shepherd on Fibers
Hi Ludo! That you add fibers to Shepherd is great news.
On Wed, Mar 23, 2022 at 11:36:30PM +0100, Ludovic Courtès wrote:
> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks. That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.
To understand what you mean, I needed to read shepherd commit
934204fbd158aa9fbd42914b89aa960f99eef125
+ ;; Run Fibers in such a way that it does not create any POSIX thread,
+ ;; because POSIX threads and 'fork' cannot be used together
So shepherd service authors still cannot write blocking code but need
to yield?
> I’ve done some Guix System testing in VMs and didn’t notice any major
> issues. I’d like to merge that branch in ‘master’ and to eventually
> release it as 0.9.0 (with or without socket activation, we’ll see.)
> Hopefully, we could be running it within a couple of weeks.
I wanted to test too how much time it takes to boot to GDM on my slow
2010 dual-core Macbook, but I can’t quite figure out each shepherd-
service failing to build. How did you test the new shepherd in Guix?
florian@florianmacbook ~/git/guix [env]$ gzip -cd
/var/log/guix/drvs/f3/g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv.gz
Backtrace:
16 (primitive-load "/gnu/store/4qk3x8dl8hxnwry637kq4ahh2z2?")
In ice-9/eval.scm:
619:8 15 (_ #(# #))
159:9 14 (_ #(# #))
In ice-9/boot-9.scm:
3326:17 13 (resolve-interface (shepherd service) #:select _ #:hide ?)
In ice-9/threads.scm:
390:8 12 (_ _)
In ice-9/boot-9.scm:
3252:13 11 (_)
In ice-9/threads.scm:
390:8 10 (_ _)
In ice-9/boot-9.scm:
3536:20 9 (_)
2835:4 8 (save-module-excursion #)
3556:26 7 (_)
In unknown file:
6 (primitive-load-path "shepherd/service" #)
In shepherd/service.scm:
26:0 5 (_)
In ice-9/boot-9.scm:
3409:4 4 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
2594:24 3 (call-with-deferred-observers #)
3422:24 2 (_)
222:17 1 (map1 (((fibers)) ((fibers scheduler) #:select (#)) # ?))
3329:6 0 (resolve-interface (fibers) #:select _ #:hide _ #:prefix ?)
ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (fibers)
Anyway, testing my boot time is not so important. I’d probably have
to change something in gnu/services/shepherd.scm. Or I should rebuild
shepherd in a less crude way.
Feel free to ignore me, though for completeness, here’s what I did:
* Pull shepherd wip-fibers commit
abbdd1e8b0f8bd4b3952efc5614f560aab9a79bf.
* I changed its Makefile.am and added -O1 to '$(GUILD) compile'
because Guix normally does likewise in an origin snippet.
* Then `guix shell -D shepherd autoconf automake guile-fibers help2man
texinfo -- sh -c 'autoreconf -vif && ./configure
--prefix=/home/florian/git/shepherd/out && make && make install'`.
* Then to Guix I added to gnu/packages/admin.scm:
(define-public new-shepherd
(package
(name "shepherd")
(version "0.9.0")
(source (local-file "/home/florian/git/shepherd/out" #:recursive? #t))
(build-system copy-build-system)
(propagated-inputs (list guile-fibers))
(arguments
`(#:install-plan '(("./" ""
…)) ;; and synopsis, description, license, homepage
and at the top #:use-module (gnu packages guile-xyz) to add
guile-fibers to new-shepherd’s propagated inputs. Also #:use-module
(guix build-system copy).
* Lastly I grafted
(define-public shepherd
(package
(name "shepherd")
(replacement new-shepherd)
…
* Compiled Guix and reconfigured. Building
/gnu/store/f3g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv
failed.
The same error happens when using new-shepherd as shepherd and
rebuilding the GNOME world.
Regards,
Florian
