Re: [PATCH 00/48] Extend bag-build to gexps.

[Development of GNU Guix and the GNU System distribution.]

Here's a more complete proof of concept:

The attached file guix.scm, when run a checkout of emacs branch emacs-29
with guix build -f guix.scm will :
- compile everything as if run locally but with the patches provided by
guix sources.
- a local edit and the rerun of guix build -f guix.scm will only compile
  new / changed code. 

Now I can try developping emacs without worring about huge compilation
times and without the developping issues that would happen if I ran the
build-system by hand without the patches. 

We could even imagine adding an option to guix shell -D that could drop
such a file for any package, since most of the code is reproducible
(although applying a derivation this way is still quite hacky).



guix.scm
Description: Binary data

Cheers!

Nicolas


On 2024-01-08 08:51, Nicolas Graves wrote:

> Rationale:
> Almost all build-systems are defined with gexpressions in functions
> that return derivations. Derivations are not easily extensible while
> gexps are. An example usage is given below.
>
> This is a pretty big rewrite that should recompile almost all packages,
> but a lot of grafting happens such as I could rebuild my system quickly.
>
> I was trying to get the build-phases of an existing package to apply to
> a local repository, because guix as a development tool for heavy packages
> (emacs, ungoogled-chromium) is tedious, and there are precious info in
> build-phases that can be applied in a local repository. I'm not aware of
> prior work on this particular issue.
>
> These patches allow to do extensions such as:
>
> (build-system
>   (name 'local-gnu)
>   (description "GNU Build System applied in the current directory")
>   (lower
>(lambda* args
>  (let ((old-bag (apply
>  (build-system-lower
>   (package-build-system emacs-pgtk))
>  args)))
>(bag
>  (inherit old-bag)
>  (build
>   (lambda* build-args
> (mlet %store-monad
> ((builder (apply (bag-build old-bag) build-args)))
>   (return (with-imported-modules '((guix build utils))
> #~(begin
> (use-modules (guix build utils))
> (with-directory-excursion #$(getcwd)
>   #$builder
>
> Of course this type of build-system isn't directly applicable because of
> the chroot of the builder, but this other trick makes it happen :
>
>   ;; We can't use package->derivation directly because we want the user rather
>   ;; than the daemon to build the derivation.
>   (with-store store
> (run-with-store store
>   (mlet* %store-monad ((bag -> (package->bag pkg))
>(drv(bag->derivation bag pkg)))
> ;; ensure inputs are in the store.
> (built-derivations (derivation-inputs drv))
> (with-environment-excursion
>  (apply invoke (derivation-builder (pk 'd drv))
> (derivation-builder-arguments drv))
>
> This isn't polished yet, but could serve as an handy way to develop
> heavy packages locally while taking advantage of the code that's
> already in guix build phases.
>
>
> Nicolas Graves (48):
>   guix: packages: Extend bag-build to support gexp.
>   build-system: gnu: Improve gnu-cross-build style.
>   build-system: gnu: Redefine gnu-build and gnu-cross-build.
>   build-system: agda: Redefine agda-build.
>   build-system: android-ndk: Redefine gnu-build.
>   build-system: ant: Redefine ant-build.
>   build-system: asdf: Redefine asdf-build.
>   build-system: cargo: Redefine cargo-build and cargo-cross-build.
>   build-system: chicken: Redefine chicken-build.
>   build-system: clojure: Redefine clojure-build.
>   build-system: cmake: Redefine cmake-build and cmake-cross-build.
>   build-system: composer: Redefine composer-build.
>   build-system: copy: Redefine copy-build.
>   build-system: dub: Redefine dub-build.
>   build-system: dune: Redefine dune-build.
>   build-system: elm: Redefine elm-build.
>   build-system: emacs: Redefine emacs-build.
>   build-system: font: Redefine font-build.
>   build-system: glib-or-gtk: Improve glib-or-gtk-cross-build style.
>   build-system: glib-or-gtk: Redefine glib-or-gtk-build functions.
>   build-system: go: Redefine go-build and go-cross-build.
>   build-system: guile: Redefine guile-build and guile-cross-build.
>   build-system: haskell: Redefine haskell-build.
>   build-system: julia: Redefine julia-build.
>   build-system: linux-module: Redefine linux-module-build functions.
>   build-system: maven: Redefine maven-build.
>   build-system: meson: Redefine meson-build and meson-cross-build.
>   build-system: minify: Redefine minify-build.
>   build-system: mix: Redefine mix-build.
>   build-system: node: Redefine

Re: How/where/when to ask for a patch review?

[Tomas Volf]

On 2024-01-10 00:33:11 +0100, Ludovic Courtès wrote:
> Hi again Tomas,
> 
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > What I tried so far:
> >
> > 1) Asking on the IRC (2x).  Once I even got a review, but from a 
> > non-committer.
> > 2) Sending email directly to the main author of the edited source file.  No
> >response.
> >
> > Technically I even mentioned it to the main author directly in person, but 
> > it
> > was at a conference, so I can hardly blame them for it just slipping by. :)
> 
> Uh, I think I’m the guilty party, my sincere apologies!

Do not worry about it!  I will not pretend it is not a bit frustrating to have
patches just sit there, but I do realize how busy your are and much you do
already.  I do appreciate all the work you do on Guix and around it.

It is unavoidable that some things just slip through.

> 
> I have to admit I have a hard time keeping up.  I review tons of
> patches, but I also like to do stuff by myself, which means I tend to
> select “easy” patches.
> 
> As a project, the ratio of available reviewer time to incoming patches
> remains chronically small.  I feel like there are more people reviewing
> and pushing patches these days, but there’s also more patches coming
> in—a good problem to have, but certainly also a source of frustration.

  I do not propose any magical solution, because I do not have one. :) Let's
hope it keeps getting better.

> 
> The good news is that I’ve now reviewed
>  and it looks almost ready to me.  So
> perhaps you’ll have to ping me or some other committer one last time,
> but I encourage you to not give up, not so close to the finish line.

Thank you very much for the review, I will send the next revision sometime this
week.

Have a nice day,
Tomas Volf

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

PS:

On 2024-01-10 17:38:17 +0100, Tomas Volf wrote:
> scheme@(guile-user)> ((@@ (shepherd service) pseudo-process?) 688)

The pid is different than above, because this was after a reboot.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: #:modules and #:imported-modules, and more

[Tomas Volf]

On 2024-01-10 11:23:22 +0100, Ludovic Courtès wrote:
> Hi!
> 
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > However even in that light, I still consider the 3 and 4 to be worth
> > considering.  Aaand even without 3, at least doing 4 in some shape or form 
> > would
> > be in my opinion useful.  Current situation is bit... copy&pasty.
> 
> In ‘core-updates’, commit 9e4ce281dbd92e3c52b831824ebb1f77023c960c by
> Maxim should address these concerns (IIUC) by distinguishing and
> renaming the variables for the default set of imported and in-scope
> modules for ‘gnu-build-system’.
> 
> Is that one of the things you had in mind?

Ah, yes, thank you for pointing that commit out, pretty much that, just for all
the build systems (it seems only gnu-build-system was modified on core-updates).

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

On 2024-01-10 00:34:48 +0100, Ludovic Courtès wrote:
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > On 2024-01-07 15:08:59 +0100, Ludovic Courtès wrote:
> 
> [...]
> 
> >>   ** Do not accidentally wait for Linux kernel thread completion
> >>  ()
> >>
> >>   In cases a PID file contained a bogus PID or one that’s only valid in a
> >>   separate PID namespace, shepherd could end up waiting for the 
> >> termination of
> >>   what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
> >>   situation is now recognized and avoided.
> >
> > This is great, I will not have to remember to run `modprobe -r mt7921e' 
> > before
> > each shutdown anymore.  I hope.  Looking forward to getting it in the Guix 
> > :)
> 
> D’oh, why did you have to do that?

Otherwise the shepherd would be stuck on shutdown waiting for process named

[mt76-tx phy0]

to terminate with messages along the lines of:

shepherd[1]: waiting for process termination (processes left: (1 678))

It is a kernel thread as far as I can tell (based on
https://stackoverflow.com/a/12231039):

$ cd /proc/678
$ cat cmdline
$ readlink exe; echo $?
1

Removing the module mt7921e stops the thread, so shepherd does not wait for it.

> How did Shepherd end up with “wrong” PID?

That I do not know.  It is visible in `ps' output, so I assume shepherd picked
it up on its own somehow.

> 
> I hope this release fixes it!

As far as I can tell, the 0.10.3 was already added into guix:

$ ps 1 | cat
  PID TTY  STAT   TIME COMMAND
1 ?Sl 0:01 
/gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile 
--no-auto-compile 
/gnu/store/06mz0yjkghi7r6d7lmhvv7gryipljhdd-shepherd-0.10.3/bin/shepherd 
--config /gnu/store/klkqq2y65k141rlipq4ls0w2rlhds12h-shepherd.conf

So I have to say it sadly did not resolve this issue.  I am unsure why though.
I am not familiar with Shepherd's code base, but quick look at the git log
suggested that procedure (@@ (shepherd service) pseudo-process?) is the relevant
one.  When I try it from a REPL, it returns #t.

$ guix shell guile shepherd guile-fibers -- guile
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> ,use (shepherd service)
scheme@(guile-user)> ((@@ (shepherd service) pseudo-process?) 688)
$1 = #t

So it *should* work?  However the issue is caused by non-free WiFi driver on a
corrupted kernel, so I am not sure if it is even problem that needs to be
solved...  I would (obviously) like to see it resolved, but I probably cannot
even bug report it, since it requires non-free hardware and software to
reproduce.

Tomas

PS: It is interesting that `guix shell guile shepherd' is not enough, the
guile-fibers have to be explicitly specified as well.  Is that expected?

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: How/where/when to ask for a patch review?

[Development of GNU Guix and the GNU System distribution.]

Hi Ludo'

On Wed, Jan 10 2024, Ludovic Courtès wrote:

> I have to admit I have a hard time keeping up.

You work too hard!  Jethro the Midianite high priest [1] liked to focus
on the "grand design":

https://www.youtube.com/watch?v=oG0a9WFkgzU

More practically, Jethro said:

(13) It came about on the next day that Moses sat down to judge the
people, and the people stood before Moses from the morning until the
evening.  When Moses' father in law saw what he was doing to the people,
he said, "What is this thing that you are doing to the people? Why do
you sit by yourself, while all the people stand before you from morning
till evening?"  Moses said to his father in law, "For the people come to
me to seek God.  If any of them has a case, he comes to me, and I judge
between a man and his neighbor, and I make known the statutes of God and
His teachings."  Moses' father in law said to him, "The thing you are
doing is not good.  You will surely wear yourself out both you and these
people who are with you for the matter is too heavy for you; you cannot
do it alone.  Now listen to me. I will advise you, and may the Lord be
with you. [You] represent the people before God, and you shall bring the
matters to God.  And you shall admonish them concerning the statutes and
the teachings, and you shall make known to them the way they shall go
and the deed[s] they shall do.  But you shall choose out of the entire
nation men of substance, God fearers, men of truth, who hate monetary
gain, and you shall appoint over them [Israel] leaders over thousands,
leaders over hundreds, leaders over fifties, and leaders over tens.  And
they shall judge the people at all times, and it shall be that any major
matter they shall bring to you, and they themselves shall judge every
minor matter, thereby making it easier for you, and they shall bear [the
burden] with you.  If you do this thing, and the Lord commands you, you
will be able to survive, and also, all this people will come upon their
place in peace."  Moses obeyed his father in law, and he did all that he
said.  Moses chose men of substance out of all Israel and appointed them
as heads of the people, leaders of thousands, leaders of hundreds,
leaders of fifties, and leaders of tens.  And they would judge the
people at all times; the difficult case they would bring to Moses, but
any minor case they themselves would judge. [1]

Kind regards
Felix

[1] https://www.chabad.org/library/bible_cdo/aid/9879/jewish/Chapter-18.htm#v13

Re: Performance of computing cross derivations

[Christopher Baines]

Efraim Flashner  writes:

> [[PGP Signed Part:Signature made by expired key 41AAE7DCCA3D8351 Efraim 
> Flashner ]]
> On Fri, Jan 05, 2024 at 04:41:14PM +, Christopher Baines wrote:
>> 
>> Ludovic Courtès  writes:
>> 
>> > Hi,
>> >
>> > Christopher Baines  skribis:
>> >
>> >> When asked by the data service, it seems to take Guix around 3 minutes
>> >> to compute cross derivations for all packages (to a single
>> >> target). Here's a simple script that replicates this:
>> 
>> ...
>> 
>> > One idiom that defeats caching is:
>> >
>> >   (define (make-me-a-package x y z)
>> > (package
>> >   …))
>> >
>> > Such a procedure returns a fresh package every time it’s called,
>> > preventing caching from happening (because cache entries are compared
>> > with ‘eq?’).  That typically leads to lower hit rates.
>> >
>> > Anyway, lots of words to say that I don’t see anything immediately
>> > obvious with cross-compilation, yet I wouldn’t be surprised if some of
>> > these cache-defeating idioms were used because we’ve payed less
>> > attention to this.
>> 
>> I've got a feeling that performance has got worse since I looked at this
>> originally, I've finally got around to having a further look.
>> 
>> I spent some time looking at various metrics, but it was most useful to
>> just write the cache keys of various types to files and have a read.
>> 
>> The cross-base module was causing many issues, as all but one of the
>> procedures there produced new package records each time. There is also
>> make-rust-sysroot which showed up.
>> 
>> I've sent some patches as #68266 to add memoization to avoid this, and
>> that seems to speed things up.
>> 
>> Looking at other things in the cache, I think there are some issues with
>> file-append and local-file. The use of file-append in svn-fetch and
>> local-file in the lower procedure in the python build system both bloat
>> the cache for example, although I'm less sure about how to address these
>> cases.
>> 
>> One thing I am sure about though, is that these problems will come
>> back. Maybe we could add some reporting in to Guix to look through the
>> cache at the keys, lower them all and check for equivalence. That way it
>> should be possible to automate saying that having [1] in the cache
>> several thousand times is unhelpful. The data service could then run
>> this reporting and store it.
>> 
>> 1: #> gnu/packages/version-control.scm:2267 7f294d908840> "/bin/svn">
>
> I grabbed the patch for make-rust-sysroot to try it out:
> Native builds:
> time GUIX_PROFILING="object-cache" ./pre-inst-env guix build --no-grafts 
> $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head 
> -n 100) -d

...

> That's a massive drop in the size of the cache and a big decrease in the
> amount of time it took to calculate those 100 items.

I think you're right, while I send some other changes in #68266, I think
it's this change around make-rust-sysroot that has pretty much all the
effects on performance.

I think the tens of thousands of duplicated packages from cross-base
that I was looking at are almost entirely coming from
make-rust-sysroot. As Ludo mentions in [1], maybe this has something to
do with use of cross- procedures in native-inputs, although I'm not sure
that moving those calls out of native-inputs is a correct thing to do.

I don't know what the correct approach here is, but I think something
needs doing here to address the performance regression.

1: https://lists.gnu.org/archive/html/guix-patches/2024-01/msg00733.html


signature.asc
Description: PGP signature

Re: Proposition to streamline our NAR collection to just zstd-compressed ones

[Ludovic Courtès]

Hello,

Maxim Cournoyer  skribis:

> It's been on my head for quite a bit of time (about 2 years, according
> to [0]), to streamline our offering of cached nars.  Letting go of gzip
> 2 years ago, along a more aggressive garbage collection policy allowed
> us to reduce our storage needs by at least 6.5 TiB.  I'm proposing to do
> the same with our lzip compressed nars, to let go of an additional 3.9
> TiB:

Those space savings would be welcome.

> The above suggests that zstd compressed nars are about 5% larger than
> the lzip ones, which is not big enough to justify carrying both, in my
> opinion.  In exchange for a little bit more bandwidth, users would have
> the nars decompressed much faster with less CPU overhead locally.

The difference is slightly higher, with lzip being 8% smaller, for a big
package like ungoogled-chromium or icecat:

--8<---cut here---start->8---
$ wget -qO- 
https://ci.guix.gnu.org/7n95j1zlnwzc44azjs7nj8givnzdfs87.narinfo|grep -B1 
^FileSize
Compression: lzip
FileSize: 85783483
--
Compression: zstd
FileSize: 92796393
$ wget -qO- 
https://ci.guix.gnu.org/prpjnnnhay0alanmkgjh66vfwjlb98kq.narinfo|grep -B1 
^FileSize
Compression: lzip
FileSize: 295991
--
Compression: zstd
FileSize: 323456
--8<---cut here---end--->8---

But yeah, even though adaptive compression selection on the client is a
minor improvement, whether it warrants the extra space is debatable.

> What do you think?  Should we go ahead and effect the following simple
> change for the Berlin build farm?
>
> modified   hydra/modules/sysadmin/services.scm
> @@ -683,7 +683,7 @@ to a selected directory.")
> ;; 
> 
> ;; for the compression ratio/decompression speed
> ;; tradeoffs.
> -   (compression '(("lzip" 9) ("zstd" 19)))
> +   (compression '(("zstd" 19)))

No objection from me, but…

… an important consideration: zstd support was added in 1.3.0, released
in May 2021.

>From experience we know that users on foreign distros rarely, if ever,
upgrade the daemon (on top of that, upgrading the daemon is non-trivial
to someone who initially installed the Debian package, from what I’ve
seen, because one needs to fiddle with the .service file to adjust file
names and the likes), and we can be sure that many are still running an
old daemon.  We spent a lot of time on user support after gzip
substitutes had been removed (‘guix substitute’ would just crash) and we
must avoid that.

(guix store) emits a warning when connecting to an “old” daemon, but
only for daemons older than 2018.  We could emit a warning based on
whether or not “builtin:git-download” is available, but maybe that’s too
early?

In addition to the warning, we should communicate in advance and make
sure our instructions on how to upgrade the daemon are accurate and
clear.

Thoughts?

Ludo’.

Re: #:modules and #:imported-modules, and more

[Ludovic Courtès]

Hi!

Tomas Volf <~@wolfsden.cz> skribis:

> However even in that light, I still consider the 3 and 4 to be worth
> considering.  Aaand even without 3, at least doing 4 in some shape or form 
> would
> be in my opinion useful.  Current situation is bit... copy&pasty.

In ‘core-updates’, commit 9e4ce281dbd92e3c52b831824ebb1f77023c960c by
Maxim should address these concerns (IIUC) by distinguishing and
renaming the variables for the default set of imported and in-scope
modules for ‘gnu-build-system’.

Is that one of the things you had in mind?

Thanks,
Ludo’.

Re: Welcome Oleg (sharlatan) as a new committer

[Efraim Flashner]

On Tue, Jan 09, 2024 at 11:19:51PM -0500, Maxim Cournoyer wrote:
> Hello Guix!
> 
> I'm happy to start the 2024 year with a new committer onboard: Sharlatan
> (Oleg).
> 
> Sharlatan is maintaining a growing collection of astronomy packages in
> Guix, among others.
> 
> Let's Wish them a warm welcome!

Welcome!

-- 
Efraim Flashner  רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: Guix wiki

[Attila Lendvai]

> 1. People find the [data] service provides value (can someone restate what 
> that
> value is exactly? Is it needed e.g. to power


if you allow hijacking the above into the wiki discussion:

this is a good example where a wiki page (central, easily editable, capturing 
the current state) would tremendously help this discussion. who, where, why, 
what, etc...

such a wiki doesn't need to be completely open for self-registration, which is 
the source of most issues. kinda like the commit bit, but with more relaxed 
requirements. maybe invite-only, or a somewhat hidden static secret could be 
used for gatekeeping the registration.

it's an illusion that everything is captured by the mailing list archive when 
finding stuff is inefficient in a discussion log. also, the wiki displays the 
current state, not the entire bumpy road getting there.

i know about https://libreplanet.org/wiki/Group:Guix but the search engines 
don't really. and i don't know how others feel about it, but i subconsciously 
don't take it seriously because the search box is not focused on guix, and in 
this form it feels like just an aftertought, not a guix wiki.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If you are ever tempted to look for outside approval, realize that you have 
compromised your integrity. If you need a witness, be your own.”
— Epictetus (c. 55–135 AD)