Re: Reproducibility of "core" packages in GNU Guix

[Vagrant Cascadian]

On 2022-05-02, Vagrant Cascadian wrote:
> $ guix challenge --diff=none $(cat guix-base-set)
>
> /gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5 contents 
> differ:

Proving more difficult than I'd hoped for, smallish diffs in the .ko
files and in the bzImage and System.map, but nothing obvious leaping out
at me. The corresponding files are reproducible in Debian bookworm...

Working on this lead me to notice a bug in diffoscope at least:

  https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/305


> /gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3 contents differ:

Patch:

  https://issues.guix.gnu.org/55758

There was already a patch in debian to set the date using an environment
variable. Might be worth working up a patch to support SOURCE_DATE_EPOCH
and push it upstream... or nudging upstream to drop the timestamp
entirely. :)


> /gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23 contents differ:

Patch:

  https://issues.guix.gnu.org/55757

Disabling parallel building in guix fixes it for me consistently,
although Debian's "isl" package is reproducible but... builds with
parallelism. (well, not reproducible on i386, but who's really
counting?)

What about other distros? Do you do anything to make "isl" reproducible?


> /gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0 contents 
> differ:
> /gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8 contents differ:

Both of these were not *just* due to parallelism as I'd
hoped... inscrutible guile...


So, 2 out of the 5 remaining packages have plausible fixes (out of 47
total)... not too bad!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: proposal: guix-ment...@gnu.org list/alias

[Gábor Boskovits]

zimoun  ezt írta (időpont: 2022. jún. 2., Cs
0:13):

> Hi,
>
> On Wed, 01 Jun 2022 at 22:17, Ricardo Wurmus  wrote:
>
> > What do you think about this?
>
> This is a great idea!
>

I agree. Having this would be nice.

>
>
> > 3. update the Contributing section in the manual (and the website) to
> > suggest Cc-ing guix-ment...@gnu.org for a first contribution.
>
> Maybe instead of manually CC-ing guix-men...@gnu.org, we could have a
> tiny script on the top of git-send-email which would add X-Debbugs-CC.
>
>
> Cheers,
> simon
>
>
>

Re: proposal: guix-ment...@gnu.org list/alias

[zimoun]

Hi,

On Wed, 01 Jun 2022 at 22:17, Ricardo Wurmus  wrote:

> What do you think about this?

This is a great idea!


> 3. update the Contributing section in the manual (and the website) to
> suggest Cc-ing guix-ment...@gnu.org for a first contribution.

Maybe instead of manually CC-ing guix-men...@gnu.org, we could have a
tiny script on the top of git-send-email which would add X-Debbugs-CC.


Cheers,
simon

Re: A corner case of broken reproducibility

[Maxime Devos]

raingloom schreef op wo 01-06-2022 om 22:41 [+0200]:
> Could we instead check for existing homes and set uids in
> /etc/passwd based on that instead? That's practically O(1), but is a
> bit more involved.

For this to work, the home directory may not be changed.  As Ludo wrote
(albeit about user names, not home directories):

> How do you know that user “maxime” created today is “the same” as
> that “maxime” deleted a while back?  You can’t.

Maybe it's an option, but there are other complications as well that
need to be resolved somehow.


signature.asc
Description: This is a digitally signed message part

Re: A corner case of broken reproducibility

[raingloom]

On Wed, 01 Jun 2022 22:09:11 +0200
Maxime Devos  wrote:

> Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> > There’s a talk by Lennart Poettering where he explains that,
> > contrary to what one might think, “chown -R $HOME” turns out to be
> > fast enough that systemd-homed can do that unconditionally (off the
> > of my head).  
> 
> Interesting.
> Taking "find $HOME > /dev/null" as an approximation of how long "chown
> -R $HOME" would take:
> 
> $ time find . > /dev/null 
> 
> real  0m7,066s
> user  0m0,427s
> sys   0m1,341s
> 
> Assuming that ‘unconditionally = only chown -R $HOME if the uid of
> $HOME isn't what was expected’ (otherwise, +7sec for every boot),
> that's not too bad.
> 
> That's on a SSD and not a hard disk though, I'd image it would be
> worse on a hard disk.
> 
> Depending on the size of $HOME, it could be a lot longer though:
> 
> $ time find /gnu/store > /dev/null 
> 
> real  1m9,729s
> user  0m4,135s
> sys   0m13,840s
> 
> Might still be acceptable as long as uids aren't switched too often
> ...
> 
> Can we, as-is in Guix, assume we can modify /home/foo though?  E.g.
> what if it's put on a separate storage medium not attached at boot.
> 
> Greetings,
> Maxime.

Could we instead check for existing homes and set uids in
/etc/passwd based on that instead? That's practically O(1), but is a
bit more involved.

Re: Merging the purge-python2-packages branch

[Maxime Devos]

zimoun schreef op wo 01-06-2022 om 21:51 [+0200]:
> Any user of Guix, scientist or not, can be surprised that their
> perfectly working packages are suddenly removed without a period of
> grace.  Yes, these packages could have been removed before today since
> they are EOL since 2 years.  It does not change my concern.
> 
> What is the emergency in the maintenance burden that we cannot publicly
> announce the purge and wait a grace* period?  For the broken packages, I
> understand.  I am sorry but I am still missing for the perfectly working
> packages.
> [...]
> *grace period: it could have been short as couple of weeks.

The grace period has been started back in 2008 and ended in 2015.
I don't know the original location where this has been publicly
announced, but there is [0]:

> (from [0])
> Sunsetting Python 2
> [...]
> We did not want to hurt the people using Python 2. So, in 2008, we
> announced that we would sunset Python 2 in 2015,

This is a 5 year grace period, which is already a lot of weeks.
Futhermore, this has even been extended for five additional years:

> (from [0])
> and asked people to upgrade before then. Some did, but many did not.
> So, in 2014, we extended that sunset till 2020

So we're at a ten-year grace period.  And some distros have informally
extended the grace period a bit longer to some limited degree.

That said, I suppose it would have been better to also repeat the
message about the grace period on the blog, channel news and info-guix
(and maybe on IRC too, why not) as you seem to suggest, for people that
follow distro news but not upstream news, given the large impact (many
python2 packages)?

[0]: https://www.python.org/doc/sunset-python-2/

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

proposal: guix-ment...@gnu.org list/alias

[Ricardo Wurmus]

Hi,

to help new contributors get their first contribution into Guix I think
it would be good to do this:

1. create a new private mailing list or mailing alias guix-ment...@gnu.org

2. ask for experienced Guix contributors who are committed to helping
new contributors to subscribe to the list

3. update the Contributing section in the manual (and the website) to
suggest Cc-ing guix-ment...@gnu.org for a first contribution.

4. have subscribers to guix-ment...@gnu.org “claim” a contribution
within 48 hours, to avoid dilution of responsibilit.  Every contribution
must be acknowledged.

Mentors would then not merely review the submission but shepherd it,
making necessary modifications and push the adjusted patches as soon as
possible.  The goal is not to do the usual review but to create
quick successes, a great first impression.

I recently contributed my first R package to CRAN, and I received an
automatic response that a human reviewer would get back to me within x
days.  This was a great experience, because I didn’t feel like I dropped
off a package into the void.  Expectations were managed and a process
outlined that set the course for the whole interaction.

Our track record with regard to review is not great, and that harm first
contributors more than anyone else.  Having dedicated mentors who agree
to responding and claiming a contribution within 48 hours addresses this
problem.

I cannot commit to reviewing enough patches that come in on
guix-patches, but I sure can promise to join the guix-mentors list and
take on and shepherd first contributions.

What do you think about this?

-- 
Ricardo

Re: A corner case of broken reproducibility

[Maxime Devos]

Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> There’s a talk by Lennart Poettering where he explains that, contrary to
> what one might think, “chown -R $HOME” turns out to be fast enough that
> systemd-homed can do that unconditionally (off the of my head).

Interesting.
Taking "find $HOME > /dev/null" as an approximation of how long "chown
-R $HOME" would take:

$ time find . > /dev/null 

real0m7,066s
user0m0,427s
sys 0m1,341s

Assuming that ‘unconditionally = only chown -R $HOME if the uid of
$HOME isn't what was expected’ (otherwise, +7sec for every boot),
that's not too bad.

That's on a SSD and not a hard disk though, I'd image it would be worse
on a hard disk.

Depending on the size of $HOME, it could be a lot longer though:

$ time find /gnu/store > /dev/null 

real1m9,729s
user0m4,135s
sys 0m13,840s

Might still be acceptable as long as uids aren't switched too often ...

Can we, as-is in Guix, assume we can modify /home/foo though?  E.g.
what if it's put on a separate storage medium not attached at boot.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: A corner case of broken reproducibility

[Maxime Devos]

Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> Things that seem missing here to me:
> 
>    * a mechanism for remembering that an uid is still in use even
> though
>  the user has been removed (previously mentioned solutions: keep
> the
>  uid in /etc/passwd even though it is ‘removed’, or keep a
> separate
>  /etc/passwd-graveyard or such, etc.).  For system accounts and
> user
>  accounts.  Won't help in this particular case but would make
> more
>  general adding/removing user accounts less fragile (avoid 
>  accidental reuse).

How do you know that user “maxime” created today is “the same” as that
“maxime” deleted a while back?  You can’t.

The point above was about remembering that uids (arbitrary unique
numbers) are still in use, and only tangentially about user names (the
/etc/passwd bit).

We can have a mechanism for registering that an uid that hasn't an
associated name anymore but might still be in use somewhere
(by a process or a file) and hence shouldn't be reused automatically,
without having to touch the concept of user names at all.

We don't have to know that they are the same or different (though we
could implement some kind of heuristic that old "foo" is the same as
new "foo" if that's desired), we only need some kind of mechanism to
stop automatically breaking things.

> (gnu build accounts) is stateful in that it makes sure UIDs aren’t
> reused.  (This is roughly the same algorithm as used by Shadow.)

It doesn't?  AFAICT it only takes /etc/passwd and /etc/groups in
account and there was some bug report reusing uids in system accounts
after removing a service (something about tor and gdm?), adding another
service and re-adding the original service or something like that.

> >    * a mechanism for telling Guix ‘I'm renaming the user account,
> > not
> >  creating and removing a new one, so keep the uid’
> 
> Every system generation stands alone though; it’s functional,
> stateless, and all that.  What does “rename” mean in this context?

A declaration in the configuration that in the past the user "bar" has
been named "foo", so if during reconfiguration Guix doesn't find "bar"
in /etc/passwd but it does find "foo", it shouldn't assign a new uid
and remove "foo" or such but rather change "foo:...:THIS-UID:.." entry
to "bar:...:THAT-UID":

(user-account
  (name "foo")
  (old-names '("bar" "baz"))
  (old-home-directories '("/home/bar" "/home/baz")) ; rename the old
/home/bar to /home/foo
  [...])

It's a bit state-y though in the sense that it refers to the previous
system configurations (and it wouldn't interact well with rollbacks
because the old configurations wouldn't have "foo" in old-names), so
some other solutions may be needed!

E.g., here's an alternative solution:

  * Require each user to be given unique identifier (UUID?) that cannot
be changed (unlike uid or username or home directory).
Record the UUID in /etc/passwd or such somehow.

Example:

+ Configuration 1

(user-account
  (name "foo")
  (uid 1234)
  (persistent-identifier "THE-UUID")
  (home-directory "/home/foo"))

At first boot, this creates a foo:...:1234:...:THE-UUID entry in
/etc/passwd as no user with THE-UUID exists yet.  Likewise, it also
creates /home/foo

+ Configuration 2:

(user-account
  (name "bar")
  (user 4321)
  (persistent-identifier "THE-UUID")
  (home-directory "/home/bar"))

At reconfiguration, Guix notices that THE-UUID already exists, so
it renames the old home directory /home/foo (listed in
/etc/passwd) to /home/bar, chowns /home/foo to 4321, and changes
the passwd entry to use 4321 as uid and "bar" as username

+ Rolling-back to configuration 1: likewise.

A complication however: home directories on external media
that might not even be attached during reconfiguration, or
external media that are read-only.  And not being able to rename the
THE-UUID is a limitation.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Merging the purge-python2-packages branch

[zimoun]

Hi,

On Wed, 01 Jun 2022 at 18:21, Ludovic Courtès  wrote:

> The question boils down to: how can we maintain a general-purpose
> package collection?

I agree and I never said that we have to maintain packages EOL since 2
years.

As I pointed, many packages of these set are not broken… yet.

Any user of Guix, scientist or not, can be surprised that their
perfectly working packages are suddenly removed without a period of
grace.  Yes, these packages could have been removed before today since
they are EOL since 2 years.  It does not change my concern.

What is the emergency in the maintenance burden that we cannot publicly
announce the purge and wait a grace* period?  For the broken packages, I
understand.  I am sorry but I am still missing for the perfectly working
packages.

Again, I agree with the purge, I disagree with the process.

*grace period: it could have been short as couple of weeks.


> It’s great that you’re voicing the concerns of the scientific community.
> At the end of the day though, someone has to maintain all this code.
> We’re removing packages from Guix proper, letting interested users
> either pin their software environment or maintain those packages in a
> channel of their own.  In the latter case, the maintenance burden is
> transferred.

My concern is mainly about the process, not the purge.  As you can read
in the Git log, I have removed many broken Python 2 packages.  And as
you can also read in the mailing list archive, I have been concerned by
this topic and I tried to propose (many times) a plan for a smooth
transition because package removals.

Maybe I am wrong but I see a difference in a transition plan between
collectively maintain all this code for 2 years and remove many working
packages without a public announce.

I agree with the purge and it is nice that it happens.  But I am
surprised by the abrupt process.  A grace period could have smoothed the
transition for the few interested users, if any. :-)

Maybe my ideal world is wrong, but to me, the collective process would
have somehow been on Guix side: patches, branch and CI, announce on
guix-devel, announce on info-guix and publish a blog post (because the
script is unique, awesome and really worth), then done.  In my ideal
world, we were at the announce on guix-devel step.  Hence my surprise.


> The transition can be difficult; surely, some user out there will
> discover all of a sudden that their favorite package disappeared.  As
> engineers who support scientific users, you and I (and others) can help
> smooth that by pasting package definitions that we know are still used
> into Guix-Past or some other channels.

It is a bit more than pasting; whatever. :-)

It is really interesting: so much care about “guix environment” to avoid
any breakage of any workflow vs a massive purge without even an announce
on guix-devel: be aware, many Python 2 will be dropped on .
Anyway.

As I said, I am surprised.  But the world is not exploding. :-) Come on,
it is Python 2 EOL since 2 years!

Even, I am very grateful that this boring janitor task of purging is
almost done.  Many thanks to Maxim for the hard and not fun work!

The script and various tools around are really great materials exposing
how Guix is powerful.  Awesome and thanks!


> In the end, it can have a good side effect: getting scientists aware of,
> and ideally involved in, the maintenance of their own infrastructure.
> Maybe you have an argument to recruit an new engineer on your team?  :-)

Too much optimism? :-)

To be honest, I get two kind of feedback:

 1. from scientists end-user, a) they do not have the packages they need
 when these packages are easily available elsewhere, b) many tiny
 annoyances which do not make daily usage smooth compared to others;

 2. from “sysadmin”, Guix is not enough stable and not ready for production.

Both are not technical but are most about perception. I will not drift
off topic. ;-)

About “my team”, do you mean recruit myself?  Even, I am probably the
only potential recruit in my complete Institute. ;-)


Bah yeah optimism!  I am surprised and surprised are often fun! :-)


Cheers,
simon

‘staging’ to be merged soon: help needed!

[Ludovic Courtès]

Hi!

Christopher Baines  skribis:

> There is starting to be some information in the QA data service instance:
>
>   
> https://data.qa.guix.gnu.org/compare-by-datetime/package-derivations?base_branch=master_datetime=_branch=staging_datetime==x86_64-linux=none_change=broken_name=_results=_results=on
>
> That page should show things that build on master, but fail to build on
> staging.

Neat.

Let’s look at these failures and address them with an eye on merging the
branch within a week.  How does that sound?

> There are some false positives though, like python-protobuf that was
> recently fixed on master and should be working on staging once master is
> merged in.
>
> There's probably some false negatives as well, since the build
> information is coming from bordeaux.guix.gnu.org, and it's not yet
> attempted to build everything for staging yet.

For the record, the weather looks nice for x86_64-linux:

--8<---cut here---start->8---
$ git log |head -6
commit 64c043e63a4be97f59fd1906c47973a74eedda67
Merge: b1f763de54 75af73e1b7
Author: Efraim Flashner 
Date:   Wed Jun 1 12:31:09 2022 +0300

Merge remote-tracking branch 'origin/master' into staging
$ ./pre-inst-env  guix weather -c10 --substitute-urls=https://ci.guix.gnu.org
computing 20,543 package derivations for x86_64-linux...
looking for 22,106 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  92.1% substitutes available (20,357 out of 22,106)
  at least 104,809.2 MiB of nars (compressed)
  154,511.9 MiB on disk (uncompressed)
  0.009 seconds per request (69.8 seconds in total)
  106.1 requests per second

  16.4% (286 out of 1,749) of the missing items are queued
  at least 1,000 queued builds
  aarch64-linux: 725 (72.5%)
  x86_64-linux: 272 (27.2%)
  powerpc64le-linux: 2 (.2%)
  i686-linux: 1 (.1%)
  build rate: 175.69 builds per hour
  i686-linux: 177.08 builds per hour
  x86_64-linux: 3.87 builds per hour
1454 packages are missing from 'https://ci.guix.gnu.org' for 'x86_64-linux', 
among which:
  6436  rust@1.39.0 
/gnu/store/4rlxnj7p7dmb8bzvsh9784nj0pvsv4v8-rust-1.39.0-cargo 
/gnu/store/1gmbsq336mav5ygfsrcf75svvj0sdm9i-rust-1.39.0 
  4083  texlive-latex-listings@59745
/gnu/store/4796jq8racfm3wdpxyhqwzjzz675r8c3-texlive-latex-listings-59745 
  3783  libva-without-mesa@2.13.0   
/gnu/store/qr66najl2lqpdgbpq7dmf8gg40h2cvd1-libva-without-mesa-2.13.0 
  3217  python-pretend@1.0.9
/gnu/store/yfphba6zvcdk9cr08283qy4sifgx1z8z-python-pretend-1.0.9 
  3174  python-setuptools-rust@1.1.2
/gnu/store/lj4kr4d917nn7nn2s4y73dj59fqc75kv-python-setuptools-rust-1.1.2 
  2560  ld-wrapper@0
/gnu/store/wwqhibfn789v8488xfrqxfzhj2a28qxl-ld-wrapper-0 
  2098  texlive-latex-pgf@59745 
/gnu/store/cc44x8fcp1zi8w05cjckdh3z8q5w49r2-texlive-latex-pgf-59745 
  2032  texlive-updmap.cfg@59745
/gnu/store/cmz8cya03f3f5q1q3vk9i2vyj247hhbj-texlive-updmap.cfg-59745 
  1409  docbook2x@0.8.8 
/gnu/store/nxic9hilvl4rjnrmkx20d7yxkn261n40-docbook2x-0.8.8 
  1193  texlive-updmap.cfg@59745 
/gnu/store/wl6dbk462hdjd5vdmdp57lzhas2chrmz-texlive-updmap.cfg-59745
   773  go-std@1.17.9   
/gnu/store/9ygfirvg18iih36sgg4j5q88nrpfp94l-go-std-1.17.9 
   679  hlint@3.2.7 /gnu/store/1mb1zgxa2sli1zxrblj03jmgg5c2bv0d-hlint-3.2.7 
/gnu/store/x2xbwgivgj2s2j16hynmgmicxbkfpl9j-hlint-3.2.7-static 
   676  go-std@1.17.9   
/gnu/store/9ygfirvg18iih36sgg4j5q88nrpfp94l-go-std-1.17.9 
   675  go-std@1.17.9   
/gnu/store/9ygfirvg18iih36sgg4j5q88nrpfp94l-go-std-1.17.9 
   628  classpath@0.99  
/gnu/store/dln1zc2xm62bxjinbsrdpqjlfkz7q79q-classpath-0.99 
   624  ecj-javac-wrapper@3.2.2 
/gnu/store/8810739m279mbg3cgn82mr9r85h4sy5h-ecj-javac-wrapper-3.2.2 
   400  java-org-ow2-parent-pom@1.3 
/gnu/store/5crmq1sw07gh5m9rs26q0zzchbaf0nym-java-org-ow2-parent-pom-1.3 
   398  java-hamcrest-parent-pom@1.3
/gnu/store/gyjslkf5n7rfd1wndqr81phq3g14pfj0-java-hamcrest-parent-pom-1.3 
   260  rust@1.39.0 
/gnu/store/4rlxnj7p7dmb8bzvsh9784nj0pvsv4v8-rust-1.39.0-cargo 
/gnu/store/1gmbsq336mav5ygfsrcf75svvj0sdm9i-rust-1.39.0 
   246  libva-without-mesa@2.13.0   
/gnu/store/qr66najl2lqpdgbpq7dmf8gg40h2cvd1-libva-without-mesa-2.13.0 
   240  ld-wrapper@0
/gnu/store/wwqhibfn789v8488xfrqxfzhj2a28qxl-ld-wrapper-0 
   240  python-pretend@1.0.9
/gnu/store/yfphba6zvcdk9cr08283qy4sifgx1z8z-python-pretend-1.0.9 
   240  python-setuptools-rust@1.1.2
/gnu/store/lj4kr4d917nn7nn2s4y73dj59fqc75kv-python-setuptools-rust-1.1.2 
   232  texlive-latex-listings@59745
/gnu/store/4796jq8racfm3wdpxyhqwzjzz675r8c3-texlive-latex-listings-59745 
   223  docbook2x@0.8.8 
/gnu/store/nxic9hilvl4rjnrmkx20d7yxkn261n40-docbook2x-0.8.8 
   206  dune@3.1.1  /gnu/store/dz5c54kavgwq5qn37l0a0fymf1bv4b71-dune-3.1.1 
   183  kdbusaddons@5.70.0  
/gnu/store/r47l5cr1bxrb53mwm0gkbpfq6dxgx1a2-kdbusaddons-5.70.0 
   175  java-slf4j-parent@1.7.25
/gnu/store/vw6310g510ghbaicjzj813jnadkvaa81-java-slf4j-parent-1.7.25 
   

Re: A corner case of broken reproducibility

[Ludovic Courtès]

Maxime Devos  skribis:

> I don't think the problem is that the uid of /home/... was wrong,
> rather I think the problem is that Guix has forgotten the uid and hence
> invents a new one to put in /etc/passwd instead of keeping the old one.
>
> A pitfall (noticed in the context of system accounts): the user could
> have created files outside /home (e.g. in /tmp).  IIUC, this would also
> require a reboot to keep name<->uid consistent after "guix system
> reconfigure".
>
> A 'chown -R' doesn't seem great to me from a security perspective
> (seems very easy to get something wrong, and the TOCTTOU-free chownat
> hasn't been merged yet in Guile), a performance perspective (what if
> you have a huge $HOME).  Also extra io -> slower boot + disk wear.
> It also destroys some information, it's possible to intentionally have
> files owned by other users inside $HOME.

There’s a talk by Lennart Poettering where he explains that, contrary to
what one might think, “chown -R $HOME” turns out to be fast enough that
systemd-homed can do that unconditionally (off the of my head).

> Things that seem missing here to me:
>
>   * a mechanism for remembering that an uid is still in use even though
> the user has been removed (previously mentioned solutions: keep the
> uid in /etc/passwd even though it is ‘removed’, or keep a separate
> /etc/passwd-graveyard or such, etc.).  For system accounts and user
> accounts.  Won't help in this particular case but would make more
> general adding/removing user accounts less fragile (avoid 
> accidental reuse).

How do you know that user “maxime” created today is “the same” as that
“maxime” deleted a while back?  You can’t.

(gnu build accounts) is stateful in that it makes sure UIDs aren’t
reused.  (This is roughly the same algorithm as used by Shadow.)

>   * a mechanism for telling Guix ‘I'm renaming the user account, not
> creating and removing a new one, so keep the uid’

Every system generation stands alone though; it’s functional, stateless,
and all that.  What does “rename” mean in this context?

>   * some heuristics for detecting mistakes (e.g.: if Guix thinks it
> should create a directory /home/foo for uid 1234, but it notices
> there is already a directory /home/bar with that uid 1234, then
> that's super suspicious.  Likewise, if Guix thinks the home
> /home/foo should be owned by uid 1234, but it notices it's already
> owned by 1235!=1234, that's also suspicious).

Yeah.

>   * some mechanism for resolving mistakes 

Sure.

Ludo’.

Re: Finding a “good” OpenPGP key server

[Ludovic Courtès]

Vagrant Cascadian  skribis:

> We've already got the keyring branch in guix.git, maybe adding an
> upstream-keys branch wouldn't be madness? Or a separate git
> repository. And then you could get it archived at software heritage or
> archive.org or whatever trivially.

Yes, that sounds reasonable to me.

Ludo’.

Re: Finding a “good” OpenPGP key server

[Ludovic Courtès]

Maxime Devos  skribis:

> Ludovic Courtès schreef op ma 30-05-2022 om 17:34 [+0200]:

[...]

>> We could also have our own key server.  Just like ‘guix lint -c
>> archival’ triggers SWH archival, we could have a tool that triggers
>> key download on the server so that crypto material never vanishes.
>
> Is archival important here though?  If the crypto material vanishes,
> presumably that means the corresponding author stopped updating the
> source code, so it won't be useful anymore (except for after-the-fact
> verification?).

If you want to be able to authenticate software, even after the fact,
then key material needs to be available (that’s why the commit
authentication framework lets you store keys in the repo).

> What benefit would a Guix key server bring us?

It would allow us to archive signing keys of all the software packages
ever added to Guix.

I can picture a new ambitious project that we could call:
OpenPGP Key Heritage.

> I guess my suggestion is to skip any intermediate infrastructure and
> let the Guix repo itself be the key ‘server’ (when using local-file
> (*)) or download directly from the site where the key is located.
>
> (*) if space is concern, there are some GPG options that can be used
> for stripping out the photo ids and the various signatures by other
> people and keep only the bits actually required by Guix.

Let’s assume 10K packages are signed, and that the signing key changes
once per year.  After 5 years, we’d have accumulated 50K OpenPGP
certificates in the repo.  Even if they are stripped (no user ID, no
photo, etc.), that’s still non-negligible.

So yes, I’d rather have it out-of-band.  :-)

Ludo’.

Re: Cuirass and SQL

[Ludovic Courtès]

Hi,

Arun Isaac  skribis:

>> Years before, Hydra (https://nixos.org/hydra) also dropped its SQLite
>> backend in favor of PostgreSQL only.
>>
>> Like you, not being a database person, I liked that SQLite was easy to
>> deploy and had a clear model: it just touches this one file and that’s
>> it.
>
> Exactly! :-)
>
> If we use guile-dbi, it should be possible to support both sqlite and
> postgresql. Popular projects like Nextcloud do allow the user to choose
> their preferred database system. But, then again, Cuirass is built for
> very large scale. So, perhaps it is best to not try and also cover the
> small scale end.

That too was my understanding some years ago.  :-)

But then I discovered that in practice, you optimize for a specific
DBMS.  The way you’d optimize a Postgres or an SQLite database for your
application can be quite different.  An abstraction wouldn’t let you do
that.

> Scale may not be an issue at least for the CI. Unlike Guix which needs
> to rebuild the entire world of software, most other software are only
> going to have a handful of jobs---easily less than 5 or less than 10 in
> the worst case. So, even if we trigger all these jobs on every commit,
> the total number of runs will easily be manageable with sqlite.

Yes, could be!

Thanks,
Ludo’.

Re: Merging the purge-python2-packages branch

[Ludovic Courtès]

Hi,

zimoun  skribis:

> On Tue, 31 May 2022 at 15:07, Maxim Cournoyer  
> wrote:
>
>>> Well, as a hobbyist, I am fine with such purge.  As a scientific
>>> practitioner using Guix at work, it is more annoying…
>>
>> Agreed.  My understanding is that scientists making use of Guix already
>> use a variety of Guix channels, so I'd assume the now missing bits can
>> be fitted in Guix-Past or a suitable place without causing too much of a
>> change to their workflow.
>
> This assumption about scientists is not rooted, IMHO.  What I can say is
> that, in my lab, some people are still using python2- variants as ’bamm’
> for example.  They are not packager and they have other fishes to fry;
> they use Guix to have the things done, they are not hobbyists who like
> tweaking their computational environment. :-)

I’m sure you’re right.

The question boils down to: how can we maintain a general-purpose
package collection?

It’s great that you’re voicing the concerns of the scientific community.
At the end of the day though, someone has to maintain all this code.
We’re removing packages from Guix proper, letting interested users
either pin their software environment or maintain those packages in a
channel of their own.  In the latter case, the maintenance burden is
transferred.

The transition can be difficult; surely, some user out there will
discover all of a sudden that their favorite package disappeared.  As
engineers who support scientific users, you and I (and others) can help
smooth that by pasting package definitions that we know are still used
into Guix-Past or some other channels.

In the end, it can have a good side effect: getting scientists aware of,
and ideally involved in, the maintenance of their own infrastructure.
Maybe you have an argument to recruit an new engineer on your team?  :-)

Ludo’.

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

Arun Isaac schreef op wo 01-06-2022 om 13:04 [+0530]:
> Hi Guix,
> 
> The switch-symlinks function from (guix utils) is often required in
> activation-service G-expressions. But, only (guix build utils) and not
> (guix utils) is available to such G-expressions. See, for example, the
> pcscd-activation G-expression in (gnu services security-token) where
> switch-symlinks gets
> redefined. 
> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/security-token.scm#n70
> 
> Could we move switch-symlinks to (guix build utils)? Would such a change
> require a core-updates cycle?

To avoid a world-rebuild, you could for now make a module (guix build
symlinks) or such?  An alternative is (gnu build activation), but then
some (guix ...) modules would depend on (gnu ...).

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

The Little Guixer

[Yasuaki Kudo]

Ahem, so the title says it all?  In the style of The Little Schemer.  I wonder 
who/what/when/how it will be written - why is obvious!  -Yasu

Release v1.4?

[zimoun]

Hi,

It is time for a release!  We were almost there on January… time
flies. ;-)

Many things are pending and I feel we need a small impulsion for a
general motivation. :-)

Well, it seems conditioned by the status of the build farms.  Is all
fine in this area?


Schedule a release is the occasion to tackle some not-so-fun tasks as
merging the ’staging’ branch [1], many upgrades as Haskell [2] or Julia,
etc.

Even, it could also be the occasion to complete the purge of Python 2. :-)

Vagrant proposed [3] to fix low-hanging fruit issues in synopsis and
description.  It is an occasion for a meetup [4].

For instance, a release date on July could be neat.  WDYT?


Please comment bug#53214 [5] or answer to this message for listing the
blocking points; as discussed in this thread [6].


1: 
2: 
3: 
4: 
5: 
6: 


Cheers,
simon

Re: Supporting upstream supported Python versions

[Ricardo Wurmus]

Hi,

> From: jgart 
> To: Guix Devel 
> Subject: Supporting upstream supported Python versions
> Message-ID: <20220517003848.GB18763@gac>
[…]
> What is the opinion on supporting current upstream supported versions
> of python?
>
> The master branch only has 3.9 but I'd like to have substitutes available
> for 3.7, 3.8 and 3.10:

The Guix Past channel now also has python-3.8.

I recently needed to be able to build a profile with Python packages
that were built with Python 3.8, but without replacing each and every
instance of Python (because I don’t want to build librsvg, gtk+, etc).

Here’s the manifest I came up with:

--8<---cut here---start->8---
;;; Manifest to build the latest Python packages with Python 3.8.
(use-modules (guix packages)   ; for "package", "package-arguments"...
 (guix build-system python)
 (guix utils)
 (ice-9 match))

;; The list of Python packages (or rather specifications) that we want
;; to build with an older Python.
(define packages
  (list "python-numpy"
"python-pytorch"
"python-matplotlib"
"python-scipy"
"python-scikit-learn"
"python-seaborn"))


(define old-python
  (specification->package "python@3.8"))

(define old-python-wrapper
  ;; We use wrap-python3 to create a "python" executable.  Python
  ;; itself only comes with "python3".  Python 3.8 is available in the
  ;; guix-past channel.
  ((@@ (gnu packages python) wrap-python3) old-python))

(define (python-package? package)
  (or (eq? python-build-system
   (package-build-system package))
  ;; Special cases: packages that produce Python modules but don't
  ;; use the Python build system.
  (member (package-name package)
  (list "pybind11"

;; I'd love to just use modify-inputs here, but this python-wrapper vs
;; python complication forces me to do this manually.
(define (replace-python-in-inputs inputs python-wrapper python)
  "Some packages have an explicit Python in the inputs, or use a
separate output of the Python package.  We use PYTHON-WRAPPER wherever
the \"python\" executable may be needed and the plain PYTHON package
where we need selected outputs."
  (map (match-lambda
 (("python" package out) `("python" ,python ,out))
 (("python" package) `("python" ,python-wrapper))
 (anything anything))
   inputs))

(define (package-with-different-python pkg python-wrapper python)
  "Return a new package based on PKG that uses PYTHON during the build."
  (package/inherit pkg
(arguments
 (if (eq? (package-build-system pkg)
  python-build-system)
 (ensure-keyword-arguments
  (package-arguments pkg)
  `(#:python ,python-wrapper
#:tests? #false))   ;running tests is slow, so why bother?
 (package-arguments pkg)))
(native-inputs
 (replace-python-in-inputs (package-native-inputs pkg)
   python-wrapper python))
(inputs
 (replace-python-in-inputs (package-inputs pkg)
   python-wrapper python))
(propagated-inputs
 (replace-python-in-inputs (package-propagated-inputs pkg)
   python-wrapper python

;; This is a recursive package transformer.  When given a package
;; "pkg" it checks if it is a Python package by looking at its build
;; system; if that is the case, it will return a package variant that
;; is built with the old Python.  It does this recursively, so all
;; dependencies are also modified.
(define use-old-python
  (package-mapping
   (lambda (pkg)
 (if (python-package? pkg)
 (let ((modified (package-with-different-python pkg
old-python-wrapper
old-python)))
   (match (package-name pkg)
 ;; This package also needs a newer version of setuptools;
 ;; Python 3.8 comes with an older version of setuptools.
 ((or "python-importlib-metadata"
  "python-ipython")
  (package/inherit modified
(native-inputs
 (modify-inputs (package-native-inputs modified)
   (prepend (specification->package "python-setuptools"))
 ;; This package expects typing.py to export
 ;; _SpecialGenericAlias, but this version of Python does
 ;; not define it.
 ("python-typing-inspect"
  (package/inherit modified
(arguments
 (substitute-keyword-arguments (package-arguments modified)
   ((#:phases phases '%standard-phases)
`(modify-phases ,phases
   (add-after 'unpack 'do-not-import-SpecialGenericAlias
 (lambda _
   (substitute* "typing_inspect.py"
 

Move switch-symlinks to (guix build utils)

[Arun Isaac]

Hi Guix,

The switch-symlinks function from (guix utils) is often required in
activation-service G-expressions. But, only (guix build utils) and not
(guix utils) is available to such G-expressions. See, for example, the
pcscd-activation G-expression in (gnu services security-token) where
switch-symlinks gets
redefined. 
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/security-token.scm#n70

Could we move switch-symlinks to (guix build utils)? Would such a change
require a core-updates cycle?

Thanks!
Arun