Re: emacs-guix

[Simon Tournier]

Hi,

On lun., 30 sept. 2024 at 10:10, Christopher Howard 
 wrote:

> I'm not really interesting much in being a guix developer right now,
> but I've gained enough proficiency in Emacs and elisp to at least
> troubleshoot/investigate. The bug I am focusing currently is 73462. 

As far as I remember, the package Emacs-Guix needs some love.  The
maintenance had been moved around 2020, see:

[sr #110376] Creating an Emacs-Guix Git repository for Guix
Ludovic Courtès 
Mon, 16 Nov 2020 04:14:29 -0500
id:20201116-101427.sv15145.79...@savannah.nongnu.org
https://lists.gnu.org/archive/html/guix-devel/2020-11
https://yhetil.org/guix/20201116-101427.sv15145.79...@savannah.nongnu.org

And since then, the package has entered in low maintenance mode.  From
my understanding, the package Emacs-Guix is looking for
co-maintainers. :-)

Cheers,
simon

Re: Rebuilding a package after removing a build step

[Simon Tournier]

Hi Konrad,

On mer., 18 sept. 2024 at 10:33, Konrad Hinsen  
wrote:

> Unfortunately, when there are several packages with identical name and
> version number in two channels, Guix silently chooses one of them.

Choose when doing what? :-)

When running “guix shell” or “guix package”, it should warn.  It
doesn’t?  Ah?  I thought it does…  Hum?

When packaging, you choose: you use one module or the other, or both.
Guile tells you if there is a symbol clash, so then you resolve it by
using some #:prefix or #:hide at #:use-module.  Else you choose one
symbol or the other inside the package definition.

Cheers,
simon

Re: Are Guix-generated Docker images reproducible?

[Simon Tournier]

Hi Konrad,

On lun., 16 sept. 2024 at 13:27, Konrad Hinsen  
wrote:

> Suppose you do
>
>   guix time-machine --channels=channels.scm -- \
>   pack --format=docker --manifest=manifest.scm
>
> You keep a copy of channels.scm and manifest.scm, and run the same
> command a few months (and "guix pull"s) later, can you expect to get the
> exact same Docker image file, bit for bit? If not, why not?

That’s the idea but as noticed in the thread, there is still some
roadblocks to have a bullet-proof machinery.

FWIW, we can go a bit further and ask: if the binary Docker image had
been produced by Guix, and that’s all we have, are we still able to know
exactly how it had been produced?  And thus rebuild it bit-to-bit?

One step in this direction is explained in this post:

  https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/

And the other steps are the ones noticed. ;-)

Cheers,
simon

toward a plan? (was Re: Reducing "You found a bug" reports)

[Simon Tournier]

Hi,

As Ian pointed out earlier, here some “guix pull” bugs:

55066
58309  closed
61520  closed
62023  closed
62830

And most of them are transient or hard to reproduce.  More had been
listed in [1], it reads:

63451
63830
64489
64659  v1.4.0
64753
64963

And it’s often the same: transient or hard to reproduce.  Here a larger
collection continuing…

64753
65495
65549
65560  v1.4.0
66600  v1.4.0
67035
67179  v1.2.0
67482
67806  v1.4.0
67956  v1.2.0
67965  v1.4.0
68397  v1.4.0
69127  v1.4.0
69334
69726  v1.3.0
70075  v1.3.0
70192
70200  v1.4.0
70201  v1.4.0
70297  v1.3.0
70646  v1.4.0
70649
70650  v1.4.0
70651
70658  v1.4.0
70667
70681  v1.3.0
70940
71426
71437  v1.4.0
71691  v1.4.0
71908
71945
72028  v1.4.0
72100
72135  upgrade from v1.2.0
72332  v1.4.0
72353  v1.4.0
72563  v1.4.0
72639  v1.4.0

Please note v1.4.0 means the host revision was v1.4.0.


After looking at some of these, we have 3 classes of bugs:

 1. transient
 2. pulling from too old host revision
 3. mix of both


IMHO, the next actions are:

 a) Replace this message:

--8<---cut here---start->8---
  (message (format #f "You found a bug: the program '~a'
failed to compute the derivation for Guix (version: ~s; system: ~s;
host version: ~s; pull-version: ~s).
Please report the COMPLETE output above by email to <~a>.~%"
--8<---cut here---end--->8---

by something like: “sorry, could you try again guix pull --commit=~s
and report if it fails again.”


 b) Put here and there some logging [2] information.  Patch#68946 [2]
provides logging facilities but is missing concrete user.

It could be worth to have it.  So then, once “guix pull” fails, we
could ask to re-run “guix pull --commit= --log-level=debug”.

This would help in having some information at failure time instead
of asking them after.

Moreover, it would provide information in order to diagnose all
these transient errors and see if they could be catched up instead
of erroring.

WDYT?

In all cases, feel free to pick one or more bugs from the list above and
investigate.  Many do not have any answer – which is not welcoming and
friendly.

Cheers,
simon



1: collection of “guix pull“ bug reports
Simon Tournier 
Wed, 23 Aug 2023 18:17:20 +0200
id:86jztl20of@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2023-08
https://yhetil.org/guix/86jztl20of@gmail.com

2: [bug#68946] [RFC PATCH 0/1] Add logging capability to Guix
Maxim Cournoyer 
Mon, 05 Feb 2024 23:12:00 -0500
id:cover.1707192720.git.maxim.courno...@gmail.com
https://issues.guix.gnu.org/68946
https://issues.guix.gnu.org/msgid/cover.1707192720.git.maxim.courno...@gmail.com
https://yhetil.org/guix/cover.1707192720.git.maxim.courno...@gmail.com

Re: ‘core-updates’ is gone; long live ‘core-packages-team’!

[Simon Tournier]

Hi Ludo,

On Fri, 06 Sep 2024 at 11:01, Ludovic Courtès  wrote:

> Once it’s “known good”, I see two possibilities:

[...]

>   • Attach to a “merge train”.  For instance, assume there’s a branch
> changing ‘gsl’: this is totally unrelated to ‘ffmpeg’ but it also
> triggers a lot of rebuilds.  We could tack that second branch on top
> of the known-good ‘ffmpeg’ branch, and, once it’s all good, merge
> that “train” into ‘master’.

As Andreas pointed out:

Once the first branch is good, why not simply merge it to master and 
then
rebase the second branch on master and test it, instead of postponing 
the
merge? After all, building is costly, not merging.

Somehow, I have the same question: if “gsl” branch is “known good”, why
not directly merge it to master.  As the other possibility suggests…

>   • Merge into ‘master’, especially if it turns out that binaries are
> already available on the build farms.

…However, in this case, if the branch changing ’ffmpeg’ is “known good”
because it had been built, then the 521 rebuilds are wasted because the
branch “gsl” is cooking and also triggers these same 521 rebuilds.

Therefore, it would be wiser to merge the ’ffmpeg’ branch into the ’gsl’
branch and rebuild only once.  (I am not pushing the button “please save
the planet” but I am thinking about it very strongly. ;-))

Somehow, a tool is missing, IMHO.

How to know which branch needs to be rebased onto which other one?  How
to know which rebuilds from one specific branch are not independent to
some other branch?

Maybe it would help as a first step to have the intersection list of
“guix refresh” applied to two sets of packages.

Assuming two 2 branches are not continuously built but only when ready
to merge, I still have the same question [1]:

Bah the question how to merge different branches containing world
rebuilds without the big catch-all old core-updates branch is not
addressed under the constraint of reducing as much as possible the
number of world rebuilds, for what my opinion is worth.


Cheers,
simon


--8<---cut here---start->8---
$ guix refresh -l gsl
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 | sort | uniq > gsl.deps
$ for i in $(seq 2 7); do guix refresh -l ffmpeg@$i \
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 ;done | sort | uniq > ffmpeg.deps

$ wc -l gsl.deps ffmpeg.deps 
 1473 gsl.deps
  521 ffmpeg.deps
 1994 total

$ for line in $(cat ffmpeg.deps); do grep -n ^$line gsl.deps ;done | wc -l
521
--8<---cut here---end--->8---

1: Re: ‘core-updates’ is gone; long live ‘core-packages-team’!
Simon Tournier 
Wed, 04 Sep 2024 14:58:37 +0200
id:87v7zby3r6@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2024-09
https://yhetil.org/guix/87v7zby3r6@gmail.com

Naming “build train” instead of “merge train”?

[Simon Tournier]

Hi Ludo, all

On Fri, 06 Sep 2024 at 11:11, Ludovic Courtès  wrote:

> In the end, perhaps we’ll have to negotiate on a case-by-case basis.
> The important thing to me is: independent testing as much as possible,
> and well-defined responsibilities and scope for the people/teams
> engaging in such changes.

I agree.

My main concern is about the potential (excessive) unnecessary wasted
rebuilds.

We need a policy that clarifies how to start large rebuilds, which
implies cross the final line with a merge.


Let take the example from this another message [1] as one case.


On Fri, 06 Sep 2024 at 11:01, Ludovic Courtès  wrote:

>   • Attach to a “merge train”.  For instance, assume there’s a branch
> changing ‘gsl’: this is totally unrelated to ‘ffmpeg’ but it also
> triggers a lot of rebuilds.  We could tack that second branch on top
> of the known-good ‘ffmpeg’ branch, and, once it’s all good, merge
> that “train” into ‘master’.
>
> (To be clear, the term “merge train” originates from GitLab-CI and
> similar CI tool, which use it as a merge scheduling strategy:
> .  GitLab-CI
> can create merge trains automatically I believe, but in our case we’d do
> that manually, at least for now.)

If we consider this specific case, the “merge train” workflow means:

  a) the branch changing “gsl” is built, so 1473 rebuilds.

  b) the branch changing “gsl” and “ffmpeg” is also built in parallel, so
 521 rebuilds.

The “merge train” workflow also reads:

  i) the branch changing “ffmpeg” is built, so 521 rebuilds.

 ii) the branch changing “gsl” and “ffmpeg” is also built in parallel, so
 1473 rebuilds including all the 521 rebuilds of i).

Therefore, for each scenario, 521 builds are “wasted“, compared to the
optimal 1473 in this case.

I do not think it’s what you have in mind when speaking about “merge
train”.  Is it?

Maybe it points that “merge train” as described by Gitlab is not what we
need.  The “merge train” workflow runs *in parallel* the builds of
several branches and I am not convinced this is wanted for heavy
branches as it might happens.

Therefore in order to avoid some confusion, maybe we could avoid the
name “merge train” and use another name for the strategy we are
discussing and we would like to implement.  For instance “rebase train”
or “build train”.  Or yet another name. :-)

Let try to not waste resource.  I think we could have a policy when a
branch contains “heavy” rebuilds.  Because we cannot continuously
rebuild the world. ;-)

Somehow, the team of this “heavy” rebuild branch says: “hey build train
of branch X starts soon”, meaning the team is ready and the state of the
branch X looks good so it require to set up the CI, i.e., the team is
going to trigger more than 1000 rebuilds.  The message “build train of
branch X starts soon” is sent to guix-devel and one week is let to the
other teams to rebase their own branch onto branch X, if they have
something to build, are ready, etc.

The team who asked the “build train” is responsible for crossing the
final line (the merge to master) and also responsible to synchronize
with the wagons, e.g., by sending some updates about how is going the
world rebuild, when the final merge is planned, etc.

If no one takes this train, no big deal because a next train is probably
going to start soon. ;-)

WDYT?

Cheers,
simon

--8<---cut here---start->8---
$ guix refresh -l gsl
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 | sort | uniq > gsl.deps
$ for i in $(seq 2 7); do guix refresh -l ffmpeg@$i \
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 ;done | sort | uniq > ffmpeg.deps

$ wc -l gsl.deps ffmpeg.deps 
 1473 gsl.deps
  521 ffmpeg.deps
 1994 total

$ for line in $(cat ffmpeg.deps); do grep -n ^$line gsl.deps ;done | wc -l
521
--8<---cut here---end--->8---

PS: Please note the number here are not true the number of rebuilds but
the number of packages that triggers all the necessary rebuilds.

1: Re: ‘core-updates’ is gone; long live ‘core-packages-team’!
Ludovic Courtès 
Fri, 06 Sep 2024 11:01:46 +0200
id:87h6at2m11@gnu.org
https://lists.gnu.org/archive/html/guix-devel/2024-09
https://yhetil.org/guix/87h6at2m11@gnu.org

Re: bug#72891: [PATCH RFC] doc: Define the purpose, membership, and creation of teams.

[Simon Tournier]

Hi Ludo,

As you mentioned elsewhere, this patch deserves the attention of
guix-devel, hence CC: guix-devel. :-)

[bug#72891] [PATCH RFC] doc: Define the purpose, membership, and creation 
of teams.
Ludovic Courtès 
Fri, 30 Aug 2024 12:12:04 +0200
id:fb9721adfc71b841dcb6a2575539449ce756e401.1725012038.git.l...@gnu.org
https://issues.guix.gnu.org/72891

https://issues.guix.gnu.org/msgid/fb9721adfc71b841dcb6a2575539449ce756e401.1725012038.git.l...@gnu.org

https://yhetil.org/guix/fb9721adfc71b841dcb6a2575539449ce756e401.1725012038.git.l...@gnu.org


Cheers,
simon

Re: ‘core-updates’ is gone; long live ‘core-packages-team’!

[Simon Tournier]

Hi Ludo, all,

On Sat, 31 Aug 2024 at 15:03, Ludovic Courtès  wrote:

> To reduce world rebuilds, perhaps we’ll sometimes create “merge trains”,
> whereby we’ll merge, say, the branch upgrading CMake and that ungrafting
> ibus on top of ‘core-packages-team’, and then merge this combination in
> ‘master’.

I do not see how the world rebuild would be reduced.

Correct me if my understanding is misleading, the branch ’core-updates’
had been extended to more than core packages because the project had
(has?) not enough CPU power for continuously building any change with a
massive impact.

Therefore, this ’core-updates’ branch was built only every X months in
order to reduce the workload. As you perfectly know, all these grouped
changes is a mess and it requires a lot of effort to stabilize.  Hence,
a very boring task… i.e., an arbitrary X. ;-)

For sure, that’s a good idea to have a core-packages-team that focuses
only on core packages – the initial aim. :-)

However, it does not address the issue with changes that have a massive
impact.  This new core-packages-team just becomes one of other branches
that will also contain packages triggering world rebuilds.

And the question is how do we manage that?


>The key being: these branches will have been developed and
> tested independently of one another by dedicated teams, and the merge
> train will be a mere formality.

In this picture of “merge train”, the CI workload and world rebuilds
will increase, no?

Consider the two teams: science and this new core-packages.  Then
science takes care of openblas (6000+ dependent packages) and
core-packages of grep (6000+ dependent packages).

When science team wants to merge to master, they ask for a rebuild.

When core-packages team wants to merge to master, they ask for a
rebuild.

Therefore, we have two world rebuilds.  Or the both teams need to
synchronize and thus the branches are indeed developed independently but
not tested independently.  Are they?

Well, my understanding of “merge train” is an automatic synchronization
workflow: science team does not ask for a rebuild but ask for a merge
and core-packages team idem – so they cannot evaluate their impact and
what needs to be fixed by their changes – then both merges are the two
wagons of the train.  Well, it’s not clear for me what CI builds: first
one wagon then the two wagons?  Or first the two wagons and then what?

Aside, the core-packages or science merges might have an impact to
packages tracked by other teams.  Other said, fixes unrelated to their
scope.  Somehow, it requires that other teams also follow – it means a
kind of synchronization.

That’s why core-updates became the big catch-all and complex mess.
That’s just a poor man way to synchronize: first it minimizes the number
of world rebuilds because they are more or less manually triggered, and
second it eases to have fixes unrelated to core changes.

For sure, I agree that having more branches will help to reduce the
complexity of merging core-updates-like changes.  However, it’s not
clear how it will work in practise.  Especially, the number of world
rebuilds will increase, IMHO.

Well, all in all, I understand the “merge train” story with “regular”
branches.  But I do not see how it would work for different branches
that trigger world rebuilds.

Could you detail a bit your perspective with “merge train”?  Or the
workflow for merging branches that contains world rebuild changes.

I know about [1] but I am lacking imagination for applying it to our
situation.  Because, if we have 3 merge requests (A, B and C), the
“merge train” workflow implies the pipeline is run 3 times:

 with A on top of master,
 with A+B (B on the top of A on the top of master),
 with A+B+C (C on the top of B on the top of A on the top of master).

If A+B fails then B is removed from the train and the pipeline runs A+C.

And you can also consider if A fails then B and B+C is built.

The “merge train” workflow makes sens when the number of changes in A, B
and C is high and thus it helps to keep the merges under control.
However, one implicit assumption is a low-cost pipeline, IMHO.

Bah the question how to merge different branches containing world
rebuilds without the big catch-all old core-updates branch is not
addressed under the constraint of reducing as much as possible the
number of world rebuilds, for what my opinion is worth.

Cheers,
simon

1: https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html

PS: My understanding of Gitlab Merge Train selling point is that several
pipelines are run in parallel.  How to waste power for increasing the
throughput?

Re: Reducing "You found a bug" reports

[Simon Tournier]

Hi,

On Sun, 21 Jul 2024 at 15:16, Ludovic Courtès  wrote:

>>> I’m fine removing the “report a bug” message, maybe replacing it with
>>> some clearer diagnostic and suggestion?  WDYT?
>>
>> Could we detect if it comes from networking?  Somehow catch the error
>> and run some code that checks stuff and so report more “accurate”
>> messages?
>
> Someone™ would need to analyze some of the reports we have (some of them
> have already been commented on) to determine what happened and whether
> that is something we could diagnose differently.

Over the years, I have spent some time in answering to some of these bug
reports; see [1] for one instance of an attempt to be this Someone™ ;-).

More than often, the issue is transient, hard if not impossible to
reproduce and thus hard to analyze.  Even, when it happens for me,
sometimes I say ok let try to collect some information for helping in
debugging that but then just running again “guix pull” makes it pass.

Excluding the bug between the keyboard and the chair, most of the time
it seems coming from either user’s network, or either substitute
servers.  Most of the time the bang is transient.

Bah I do not know exactly what or how – otherwise I would have proposed
a patch or something ;-) – but I think we need to “instrument“ the
command “guix pull” in order to collect a trace for then diagnosing.

Somehow, maybe it could be a good application of Maxim’s proposal for
logging [2].

Well, instead of an “useless” backtrace, it would be more helpful to
have a kind of log or coredump; maybe something under /var/log/guix/.

I think the next step is not « Someone™ would need to analyze some of
the reports » because the reports often provide only the same “useless“
backtrace, so instead the next step seems: « Someone™ would need to
implement a better way for generating good reports ». ;-)

Cheers,
simon

1: collection of “guix pull“ bug reports
Simon Tournier 
Wed, 23 Aug 2023 18:17:20 +0200
id:86jztl20of@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2023-08
https://yhetil.org/guix/86jztl20of@gmail.com

2: [bug#68946] [RFC PATCH 0/1] Add logging capability to Guix
Maxim Cournoyer 
Mon, 05 Feb 2024 23:12:00 -0500
id:cover.1707192720.git.maxim.courno...@gmail.com
https://issues.guix.gnu.org/68946
https://issues.guix.gnu.org/msgid/cover.1707192720.git.maxim.courno...@gmail.com
https://yhetil.org/guix/cover.1707192720.git.maxim.courno...@gmail.com

Re: Question about changing versioning for TeX Live packages

[Simon Tournier]

Hi,

On Fri, 19 Jul 2024 at 18:38, Nicolas Goaziou via "Development of GNU Guix and 
the GNU System distribution."  wrote:

> I'm only changing the `version' field. For the record, "core-updates"
> currently contains all TeX Live packages with their version switched to
> "2024.2".

Cool!  Nice it’s updated. :-)


>   In the worst case, maybe a notice in the guix news will be
> sufficient.

Hum, I am not convinced “guix news” would be enough.  Maybe we could
provide something that display a warning when upgrading a profile.

Cheers,
simon

Re: Should we document how to detect if build machines are reachable before trying to offload?

[Simon Tournier]

Hi Ludo,

On Sun, 21 Jul 2024 at 14:56, Ludovic Courtès  wrote:

> I guess this is probably what we should permit: building locally when we
> cannot offload.
>
> Does that make sense?

Yes! This feature seems wanted. ;-)

Aside it would satisfy requests [1,2] randomly picked up :-) it would
help in closing this old report, I guess.

bug#24496: offloading should fall back to local build after n tries
ng0 
Wed, 21 Sep 2016 09:39:48 +
id:8760ppr3q3@we.make.ritual.n0.is
https://issues.guix.gnu.org/24496
https://issues.guix.gnu.org/msgid/8760ppr3q3@we.make.ritual.n0.is
https://yhetil.org/guix/8760ppr3q3@we.make.ritual.n0.is

Cheers,
simon

1: How to offload builds only when some of the offload build servers are 
available
Kyle Andrews 
Sun, 01 May 2022 16:01:21 +
id:875ympmdqw@posteo.net
https://lists.gnu.org/archive/html/help-guix/2022-05
https://yhetil.org/guix/875ympmdqw@posteo.net

2: guix offload
Aleksandr Vityazev 
Wed, 20 Dec 2023 14:02:27 +0300
id:87o7el9lx8@gmail.com
https://lists.gnu.org/archive/html/help-guix/2023-12
https://yhetil.org/guix/87o7el9lx8@gmail.com

Re: Question about changing versioning for TeX Live packages

[Simon Tournier]

Hi Nicolas,

Sorry if this had been answered elsewhere, I have missed it.

On Sat, 15 Jun 2024 at 19:07, Nicolas Goaziou via "Development of GNU Guix and 
the GNU System distribution."  wrote:

> I'd like to change versioning for TeX Live packages. Currently, it
> refers to a revision number in the upstream SVN repository, such as
> 70951.
>
> I'd rather use tags from TeX Live releases, e.g., "2023.0", for at least
> two reasons:
>
> 1. We don't need a granularity to the revision level for TeX Live
>packages;
> 2. Revision numbers are not very user-friendly, while tags are.
>
> In a nutshell, my plan is to change version for all packages from
> (revision) 66594 to the equivalent tag "2023.0". Unfortunately, as
> mathematics go, 2023 is lesser than 66594. As a consequence, this may
> introduce trouble when upgrading those packages. Therefore, my question
> is the following: how could I proceed to make this version change as
> smooth as possible for end-users?

Well, if I read correctly, there is:

  guix/build-system/texlive.scm:

(define %texlive-tag "texlive-2023.0")
(define %texlive-revision 66594)

  gnu/packages/texlive.scm:

(define %texlive-date "20230313")
(define %texlive-year (string-take %texlive-date 4))


And the issue seems:

(define-public texlive
   (package
(name "texlive")
(version %texlive-date)

and

(define-public texlive-scripts
  (package
(name "texlive-scripts")
(version (number->string %texlive-revision))


Therefore, indeed it will be complicated to replace the ’version’ of
’texlive-scripts’ by something as ’2023’.

But why not a ’version’ as something as ’texlive’?  Or just 2023XY?
Where XY is something to determine as the month or something else.

Are we speaking a change only for the package field ’version’?  Or is
the discussion also about replacing the way to fetch from upstream?

Cheers,
simon

Re: Reducing "You found a bug" reports

[Simon Tournier]

Hi,

On Mon, 17 Jun 2024 at 14:59, Ludovic Courtès  wrote:

>> It doesn’t feel great to tell users to report a bug for things that
>> aren’t bugs.  They’re either closed, or never followed up on; it’s a
>> poor experience on both ends.
>
> I agree, it’s pretty bad.
>
> I’m fine removing the “report a bug” message, maybe replacing it with
> some clearer diagnostic and suggestion?  WDYT?

Could we detect if it comes from networking?  Somehow catch the error
and run some code that checks stuff and so report more “accurate”
messages?

Well, easier to say… :-)

Cheers,
simon

Re: [PATCH Cuirass 1/4] specification: Ensure name is a symbol.

[Simon Tournier]

Hi,

On Mon, 17 Jun 2024 at 15:16, Ludovic Courtès  wrote:

> statically-typed language programmers would be
> right to laugh at us here, I admit

Nah because the new feature is really nice! :-) Thanks.  But it
remembers me one of our first in-person discussion back then on December
2018… My 6-years patience is paying off. ;-)

Cheers,
simon

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Thu, 11 Jul 2024 at 11:38, Ludovic Courtès  wrote:

> Good point.  In practice we’re already doing a combination of the above
> and I agree that there are probably advantages to keep it that way.

Well, from my perspective, the question is architecture per
architecture.  I mean, I think it’s doable to find sponsor (companies or
academic institutions) for x86_64 because it’s commonly used.  And
that’s not the case for less used others.

Other said, I think it would be a bad investment to buy x86_64 machines
when they could be donated and/or hosted.  However, it would be harder
for aarch64 or power9 or else, IMHO.

Cheers,
simon

Re: ci.guix.gnu.org is getting back to life

[Simon Tournier]

Hi,

On Thu, 11 Jul 2024 at 12:26, Ludovic Courtès  wrote:

> These things require constant attention.

Thank you for such attention!  Thanks Chris for playing the role of
“database administrator”. :-)

Well, it reminds me that our discussion [1] “Sustainable funding and
maintenance for our infrastructure” has been focused on “funding” and
“hardware” more than “maintenance”.

Cheers,
simon



1: Sustainable funding and maintenance for our infrastructure
Ludovic Courtès 
Tue, 02 Jul 2024 16:24:06 +0200
id:87sewr98jd@gnu.org
https://lists.gnu.org/archive/html/guix-devel/2024-07
https://yhetil.org/guix/87sewr98jd@gnu.org

Re: How to test fixed output derivations in the bordeaux build farm?

[Simon Tournier]

Hi,

On Sat, 01 Jun 2024 at 15:26, Ludovic Courtès  wrote:

> Yes.  You can’t “change the behavior of derivations” at build time,
> except by passing GUIX_DOWNLOAD_METHODS to guix-daemon itself, provided
> GUIX_DOWNLOAD_METHODS is also in ‘impureEnvVars’ of said derivation, as
> you wrote.

It’s inflexible but the behaviour « can’t “change the behavior of
derivations” at build time » appears to me reasonable, no?  Well, I am
missing some context.

That’s said, since we are speaking about fixed-output derivations, one
way would to compute another derivation with the expected behaviour.
Somehow, the context is different [1] but rewriting the derivation
allows to bypass such inflexibility.

Bah, I do not know… I am missing context about the issue. :-)

1: https://simon.tournier.info/posts/2024-04-11-rewrite-drv.html

Cheers,
simon

Re: anonymized nginx logs (was Re: Guix user statistics and upstream/downstream dependencies)

[Simon Tournier]

Hi Ricardo,

On Wed, 10 Jul 2024 at 22:15, Ricardo Wurmus  wrote:

> Do you want the plain log files or would the processed HTML report
> (generated with goaccess) be sufficient?

The HTML report will be sufficient as a start. :-)

>   The latest report is on
> ci.guix.gnu.org:/var/log/report.html.

It’s not publicly served, right?

Cheers,
simon

Re: GNU Mes 0.27 released

[Simon Tournier]

Hi Andrius,

On Tue, 09 Jul 2024 at 23:15, Andrius Štikonas  wrote:

>> Do you mean remove guile-bootstrap from the picture?  The root of this
>> graph:
>
> I think what janneke means (correct me if I'm wrong) is that before now gash 
> and gash utils could only run on guile but not on mes. So that arrow that was 
> pointing from gash to guile-bootstrap will point to mes.

It would mean to have mes-bootstrap (instead of guile-bootstrap) that
would be indirectly used by mes-boot.

Right?


>> https://www.gnu.org/software/mes/manual/images/gcc-mesboot-graph.png
>> 
>> ?  What’s the plan?  Replace the requirements of stage0-posix provided
>> by Gash and Gash-Utils by increasing bootstrap-seeds?
>
> bootstrap seeds bubble in that graph refers to [1], so it wouldn't be 
> changing 
> as we don't add anything else there.
>
> stage0-posix requirements are related to package manager and/or limitations 
> of 
> source distribution method. If stage0-posix exists in unpacked form, it can 
> run the whole bootstrap chain itself using either kaem-optional-seed [2] in 
> userspace/POSIX bootstrap (e.g. just kernel and initramfs with 
> bootstrap-seeds 
> and source) or builder-hex0 [3] (plus sources) in MBR/BIOS bootstrap (where 
> we 
> bootstrap linux kernel too). And then no guile-bootstrap is needed at all. 
> Both of these problems are already solved on x86 (though now new x86 machines 
> generally come without BIOS and bootstrapping on UEFI is only partially done).

Thanks for explaining.

[...]

> What would be interesting from bootstrapping perspective is to make those 
> additional seeds (guile-bootstrap in particular) fully bootstrappable/
> reproducible not just from inside Guix but also without Guix, so that we 
> don't 
> have chicken and egg problem of how to build guile-bootstrap binary when we
> don't yet have Guix.

Hum, I am missing something, I guess.  Well, from my poor understanding,
one way is what you explained just above or something as “Extreme“
Bootstrapping as discussed in [4], and the other way is to root the
graph in some binary driver (as guile-bootstrap).

For sure, the binary guile-bootstrap could be replaced by another
lighter, smaller and surely more reproducible other binary as
mes-bootstrap but that would not tackle the chicken-or-the-egg problem,
IMHO.  However, it could be nice to reduce again the binary footprint.

Well, from my weak understanding, maybe one direction (but huge work)
would to implement all the features required by the whole graph and
provided by gash-boot and gash-utils-boot directly in auditable binary.
Yeah, it’s a piece of work and I do not volunteer. ;-)

Cheers,
simon

4: https://guix.gnu.org/en/blog/2019/reproducible-builds-summit-5th-edition/

anonymized nginx logs (was Re: Guix user statistics and upstream/downstream dependencies)

[Simon Tournier]

Hi Maxim,

On Tue, 14 May 2024 at 07:57, Maxim Cournoyer  wrote:

> We have detailed anonymized nginx logs of all downloaded substitutes
> from Berlin, at least.  If you are interested to use this data as a
> source we could probably make it available to you since it's anonymized
> (real IP addresses have been obfuscated).

I just had a chat this afternoon with a colleague about massaging such
data.  Could I have access to them?

Cheers,
simon

Re: GNU Mes 0.27 released

[Simon Tournier]

Hi Janneke,

On Sat, 06 Jul 2024 at 11:16, Janneke Nieuwenhuizen  wrote:

> We are happy to announce the release of GNU Mes 0.27.

Cool!  Really nice.

>   Remove indirect Guile
> dependencies (via Gash and Gash-Utils) from the Mes bootstrap in Guix.

Do you mean remove guile-bootstrap from the picture?  The root of this graph:

https://www.gnu.org/software/mes/manual/images/gcc-mesboot-graph.png

?  What’s the plan?  Replace the requirements of stage0-posix provided
by Gash and Gash-Utils by increasing bootstrap-seeds?

Well, gcc-core-mesboot0 which is higher in the graph also depends on
guile-bootstrap, right?

Cheers,
simon

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi Ricardo, all,

I agree with many words in the thread.

Reading all the messages in the thread, the solution seems to try a mix
of the three options.  We could imagine “buy and host” more exotic
hardware but not a complete data center neither, “rent” for specific
needs and ask for more “sponsor”.

However, the implicit question is a chicken-or-the-egg problem: in order
to know what we would like to invest in, we need to know how much we
would be able to invest; and to know how much, we need to determine for
what.


On Mon, 08 Jul 2024 at 14:02, Ricardo Wurmus  wrote:

>>> The various options and back-of-the-envelope estimates we came up with
>>> are as follows:
>>>
>>>   1. Buying and hosting hardware:
>>>   250k€ for hardware
>>>   3k€/month (36k€/year)
>>>
>>>   2. Renting machines (e.g., on Hetzner):
>>>   6k€/month (72k€/year)
>>>
>>>   3. Sponsored:
>>>   get hardware and/or hosting sponsored (by academic institutions or
>>>   companies).

[...]

> Option #2 is rather quick to set up and quick to abandon should we run
> out of money.  It does, however, depend on continuous donations, which
> we are currently unable and possibly even unwilling to solicit.

IMHO, the discussion of any potential option should be coupled with a
“investment plan”.  One without the other is a dead end, for what my
opinion is worth.

Well, because I am in the business of academics, I am able to imagine
some path forward for option #3 (sponsor).  For instance, [1] appears to
me a very good news.  Then, I am a bit clueless* about financing option
#1 or #2 – whatever the amount.  I mean, the first question is how to
secure several k€ or k$ – how much would we be able to secure?

Let assume Guix Foundation would be able to manage the money, the
discussion becomes: considering the current incomings, how do we scale?

Is the project ready to regularly look for incomings and run after them?
>From my point of view, it’s easier to get hardware and/or hosting
sponsored by academic institutions or companies than to run by our own
for incomings in order to buy hardware and/or pay hosting.  But as said,
I am biased here. :-)

Then if we have a plan for buying hardware – depending on the amount –
it means we also need a plan to deal with such asset; on the financial
side and on the ecological side.

All in all, the first question in order to prepare the more or less next
years of our infrastructure seems about what appears financially
feasible.  Drawing what is financially doable right now, what would be
doable with work or what is just impossible in the near future, drawing
all this picture appears to me the next move in the discussion.

Other said, with my Guix Foundation hat, my proposal could be to set
this as an objective: polish Guix Foundation for preparing to scale up.


*clueless: Not fully. :-)  For sure, we could opt for crowd-founding or
 some grant here or there.  But still… how much?

Cheers,
simon

1: New North American based Guix Substitute Server, cuirass.genenetwork.org Now 
Available
"Collin J. Doering" 
Sat, 06 Jul 2024 19:35:44 -0400
id:871q46ytyn@rekahsoft.ca
https://lists.gnu.org/archive/html/guix-devel/2024-07
https://yhetil.org/guix/871q46ytyn@rekahsoft.ca

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès  wrote:

>   The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.

Could you explain the rationale?  I understand and fully agree that
sustainable funding and maintenance for infrastructure are key topics
for the project.  Do we need to move ci.guix soon?  Related to Ricardo
announcement [1]?

Well, I am missing some context or steps.  Currently, the project is
mainly in Option #3 (sponsored).  The main sponsor is MDC located in
Berlin.  The second sponsor is personal funds coupled to hardware bought
by us or donated to us – I have in mind the build farm behind the name
Bordeaux; thanks Chris! And the third sponsor – at some extent – is
Inria located in Bordeaux.

We had discussions about reinforcing the second sponsor by replacing
personal funds by project-wide funds, say Guix Foundation, community,
etc.

Is this description correct?


> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>   250k€ for hardware
>   3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>   6k€/month (72k€/year)
>
>   3. Sponsored:
>   get hardware and/or hosting sponsored (by academic institutions or
>   companies).

Well, on the paper, option #1 appears to me appealing but how do we get
this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
for over almost 7 years, right?

Except if we have a large donation that I am not aware, I do not see how
it would be possible to sign in being sure to secure 3k€/month for over
almost 7 years; considering the project has 12 years.

Other said, option #1 does not appear to me an option.

Option #2 could be a temporary option for a short time.  But again,
that’s something.


> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).

It remains option #3. :-)

For me, that’s the only viable option at scale.  The main costs should
be covered by sponsor as academical ones.  From my point of view, the
only sustainable option is to group people behind GuixHPC (I recall the
domain name hpc.guix.info is paid by Guix Foundation ;-)) and ask: a) if
their institutions are ready to donate and/or a) if we could run for
some grants altogether.

Somehow, Guix starts to be run in various scientific data centers and we
could take advantage of this opportunity.

Indeed, it locks in some relation with the hosting organizations and/or
the person in touch with them.  That’s said, Ricardo showed it works
well – or at least it can. :-) The key appears to me to not put all the
eggs in the same basket.

That’s my half-baked current opinion.  WDYT?

Cheers,
simon


1: I'm retiring (for a while); help needed
Ricardo Wurmus 
Fri, 31 May 2024 08:08:15 +0200
id:87o78mldio@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87o78mldio@elephly.net

Breath, let take a short break :-)

[Simon Tournier]

Hi MSavoritias,

This message is not to cut any discussion but maybe it could be helpful
or a bit saner if you refrain to rehash again and again the same to all
messages, replying the same (or almost) to each person expressing
different opinions.

No blame, and I also include myself: being very enthusiastic to defend
ideas and values.  However, a storm of replies is maybe not the best
mean to achieve such defense. :-)

I think people got your points and your opinion, quickly summarized as:

 1. SWH broke “implicit social rules”,
 2. Because of that, Guix must make a clear public “pressure” against SWH.


Let look how the thread looks like:

https://yhetil.org/guix/87a5jfjoey@gmail.com/T/#rc72a0743026006ee9d4758cfa794df42a9964a55
(or this other one: https://yhetil.org/guix/87il1mupco.fsf@meson/#r)

Then, for what my humble point of view is worth here, I think that your
opinion is maybe not the consensus.  Obviously, the discussion is still
open and your opinion is welcome – yeah obviously welcome! – but maybe
not by replying to all, each time.

You are advocating for a safe place, right?  From my eyes, when I see
the structure of the thread, it does not generate a safe place where
collaboration is encouraged.

My feeling, when I do a step back and look to the structure of the
thread, is that some opinions are silent because it’s hard to have the
space to express them.

Sometimes, a breath is helpful.  Somehow, FWIW, I suggest you to let the
discussion aside, then some days later read again some messages, try to
differently understand what other peers are trying to express, and
comment to few on a fresh mindset.

All opinions are very welcome.  We are all here because we value Free
Software, community, people, etc. and not necessary in that order.  And
that’s very important to be able to express all the diversity.

Again, this message is not a mean to cut any discussion.  Instead, this
message is a call to slow down. :-)

WDYT?

Cheers,
simon

Draft: dry-run + Exclude checker with package properties

[Simon Tournier]

Hi,

Patch #71697 [1] introduces dry-run for the checkers and a way to
exclude some checkers directly in the package definition.  In addition
to exclude checkers from the command-line.

FWIW, I think it covers:

>   but it is too easy to forget and once the cat is out
> of the bag privacy is lost

Well, the way to display can be improved, IMHO.

1: https://issues.guix.gnu.org/71697#4

Cheers,
simon

Re: Exclude checker with package properties [draft PATCH]

[Simon Tournier]

Hi Felix,

On Fri, 21 Jun 2024 at 20:37, Felix Lechner  wrote:

> > Is debbugs.gnu.org having issues?
>
> Yes, the community0p server crashed this morning.  Luckily, Debbugs
> appears to be back online and added messages I sent during the outage.
> Maybe yours will get there, too.

Thanks.  Yeah the message reached issues.guix.gnu.org so I guess all
is fine. :-)


> > See attached the patch implementing that.
>
> Thank you!  Do you see a chance we can amend the patch so I can block
> such package definitions from being used by 'guix deploy', 'guix system
> reconfigure' and 'guix home reconfigure'?

My input of this will wait after my holidays. ;-)

Cheers,
simon

Re: Exclude checker with package properties [draft PATCH]

[Simon Tournier]

On Fri, 21 Jun 2024 at 19:51, Simon Tournier  wrote:

> Well, thinking about indeed it could helpful in some context to specify
> the checkers to exclude at the package definition level.  Other said,
> this patch could be generalized.  Work in progress… :-)

Done here: https://issues.guix.gnu.org/71697#1

Cheers,
simon

Exclude checker with package properties [draft PATCH]

[Simon Tournier]

On Fri, 21 Jun 2024 at 09:41, Dale Mellor  wrote:

>`-x archival` does it, but it is too easy to forget 
[...]
> at least there should be a flag in the package definition. 

See attached the patch implementing that.

>From 8cb162bcde91d3b39453de576caadb9a6f8f8733 Mon Sep 17 00:00:00 2001
Message-ID: <8cb162bcde91d3b39453de576caadb9a6f8f8733.1718990517.git.zimon.touto...@gmail.com>
From: Simon Tournier 
Date: Fri, 21 Jun 2024 19:17:57 +0200
Subject: [PATCH] guix: lint: Honor 'no-archival?' package property.

* guix/lint.scm (check-archival): Skip the checker if the package is marked.
* doc/guix.texi: Document it.

Change-Id: I2e21b60ee4f02255f298740a2e9ebb1717e490ff
---
 doc/guix.texi |  15 -
 guix/lint.scm | 154 ++
 2 files changed, 93 insertions(+), 76 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 769ca1399f..5c1cb89686 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -71,7 +71,7 @@
 Copyright @copyright{} 2019 Alex Griffin@*
 Copyright @copyright{} 2019, 2020, 2021, 2022 Guillaume Le Vaillant@*
 Copyright @copyright{} 2020 Liliana Marie Prikler@*
-Copyright @copyright{} 2019, 2020, 2021, 2022, 2023 Simon Tournier@*
+Copyright @copyright{} 2019, 2020, 2021, 2022, 2023, 2024 Simon Tournier@*
 Copyright @copyright{} 2020 Wiktor Żelazny@*
 Copyright @copyright{} 2020 Damien Cassou@*
 Copyright @copyright{} 2020 Jakub Kądziołka@*
@@ -15380,6 +15380,19 @@ Invoking guix lint
 prints a message and the @code{archival} checker stops doing anything until
 that limit has been reset.
 
+Sometimes it is not desired to send a request for archiving each time
+@command{guix lint} is run.  The package might be marked to skip the
+@code{archival} checker by honoring the @code{no-archival?} property in
+package definition:
+
+@lisp
+(define-public python-scikit-learn
+  (package
+(name "python-scikit-learn")
+;; @dots{}
+(properties '((no-archival? . #t)
+@end lisp
+
 @item cve
 @cindex security vulnerabilities
 @cindex CVE, Common Vulnerabilities and Exposures
diff --git a/guix/lint.scm b/guix/lint.scm
index 68d532968d..4c33ec6598 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -1717,84 +1717,88 @@ (define (check-archival package)
 (lookup-directory-by-nar-hash (content-hash-value hash)
   (content-hash-algorithm hash)))
 
-  (parameterize ((%allow-request? skip-when-limit-reached))
-(catch #t
-  (lambda ()
-(match (package-source package)
-  (#f ;no source
-   '())
-  ((and (? origin? origin)
-(= origin-uri (? git-reference? reference)))
-   (define url
- (git-reference-url reference))
-   (define commit
- (git-reference-commit reference))
-   (define hash
- (origin-hash origin))
-
-   (match (or (lookup-by-nar-hash hash)
-  (if (commit-id? commit)
-  (or (lookup-revision commit)
-  (lookup-origin-revision url commit))
-  (lookup-origin-revision url commit)))
- ((or (? string?) (? revision?))
-  '())
- (#f
-  ;; Revision is missing from the archive, attempt to save it.
-  (save-package-source package
-  ((? origin? origin)
-   (if (and=> (origin-hash origin)  ;XXX: for ungoogled-chromium
-  content-hash-value)   ;& icecat
-   (let ((hash (origin-hash origin)))
- (match (or (lookup-by-nar-hash hash)
-(lookup-content (content-hash-value hash)
-(symbol->string
- (content-hash-algorithm hash
-   (#f
-;; If ORIGIN is a version-control checkout, save it now.
-;; If not, check whether HASH is in the Disarchive
-;; database ("Save Code Now" does not accept tarballs).
-(if (vcs-origin origin)
-(save-package-source package)
-(match (lookup-disarchive-spec hash)
-  (#f
-   (list (make-warning package
-   (G_ "source not archived on Software \
+  (if (not (assq 'no-archival? (package-properties package)))
+(parameterize ((%allow-request? skip-when-limit-reached))
+  (catch #t
+(lambda ()
+  (match (package-source package)
+(#f ;no source
+ '())
+((and (? origin? origin)
+  (= origin-uri (? git-reference? reference)))
+ (define url
+   (git-referen

About SWH, let avoid the wrong discussion

[Simon Tournier]

Hi all,

For the record, the Software Heritage initiative is supportive of the
Guix project since years.

It means that members of Guix community have or had interactions with
Software Heritage (SWH) teams since years.  For example, the blog post
“Connecting reproducible deployment to a long-term source code archive”
[1] published in 2019.  And more recently, the scientific communication
“Source Code Archiving to the Rescue of Reproducible Deployment” [2].

Almost 6 years of friendly interactions and shared values.

Could we avoid to express definitive opinions based on partial
considerations about multi-dimensional topics?

Since years, several members of Guix community are helped in one way or
the other by SWH team members in improving free software ecosystem.

Well, I speak for myself: I have been invited to several events
organized by SWH and it’s up to you to trust me when I say: SWH team
works very hard to embrace all the diversity of FOSS communities.  For
example, I recently attended to a talk organized by SWH about Commons;
that talk had been a very good food for thought and maybe it could feed
our current discussion about governance/sociocracy via comments here or
there I could commit, I do not know, maybe.

Well, I am very grateful for the opportunity to interact with SWH teams.

For the record, SWH provided various supports for the organization of 10
Years of Guix, back in 2022.  Please remember that SWH team members were
there and some stayed all the three days; probably because we are a nice
community?  All the video stream and good videos of the 10 Years of Guix
event you probably watched or maybe watch again is because the tireless
work of multi-hats person (Debian Developer, Debian Video Team, … and
working at SWH) helped by Guix community members.

Please check the Copyright header for the subcommand “guix locate”.
Yes, it had been partly written by one SWH team member because, yes they
run Guix.  Yes, their day-job is at SWH and they are also part of our
Guix community by contributing to Guix source code.

Now, you take it as it is: I am sad by what people are concluding!

Yes I understand why people are angry.  Yes discussions must happen.

However, I was expecting more benefit of the doubt considering history
and track record.  Hum, even, maybe, I am asking myself if Guix
community is indeed nice or if this time the community is just harsh and
unfair.

Do we forget the track record and the common history?

Then, for what my opinion is worth, fighting against SWH while thinking
it’s fighting against LLM/AI is the wrong fight.  Because 1. we are all
in the team.  And 2. because SWH could be a facilitator for helping in
some regulations, maybe, I do not know.  Somehow, I agree with Ekaitz.

You take it as it is: I was expecting more humility by Guix community
members.  Do you really think that a collective of people involved in
various FOSS communities with different roles, dedicating their free
time to free software or open source movements, do you think they are
the bad actors here?

My humility tells me, as I expressed several times, nothing is ignored.

Yes I also got the point about the lack of transparency.  As I said
above, FWIW, I am in touch with SWH team.  Well, I do not have special
information from SWH and I trust them to have listened or are still
listening various communities.  So my understanding is: work is in
progress…  Somehow, wait and see.

Yes I know we cannot wait forever.  Again, do we forget the track record
and the common history?  Do we consider that a multi-layers topic
involving legal or ethics questions is straightforward to articulate?

My humility tells me to wait to have clear and better understanding
about SWH motivations, their rationale, the measures and
counter-measures they maybe have in mind.  Be patient and tolerant as I
am with my friends.

Long enough email and thread.  That’s all from me! :-)

My last message.  Not because I am bored but because one week of
holidays is starting now for me. ;-)

1: 
https://guix.gnu.org/en/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/
2: https://hal.science/hal-04586520v1

Cheers,
simon

Re: Next Steps For the Software Heritage Problem

[Simon Tournier]

Hi,

On Thu, 20 Jun 2024 at 19:42, Dale Mellor  wrote:

> I'm sure guix lint tried to push my code out to them the last time I
> tried.

Yes, it’s the checker ’archival’.

Therefore, running “guix lint -x archival” does not send any request to
SWH.

Cheers,
simon

Re: Next Steps For the Software Heritage Problem

[Simon Tournier]

Hi MSavoritias, all,

On Thu, 20 Jun 2024 at 09:51, MSavoritias  wrote:

>> Not to avoid the question but from a pragmatic point of view, one
>> might ask if the source code you write and do not want to be included
>> in the training dataset, if this source code is concretely part of
>> that training dataset.

[...]

> Thats all fair and valid. Sadly tho SWH:
> - 
> there is provenance. (unless i start searching
>   HuggingFace.

Being concrete and explicit, could you please share:

 1. Which part of your code is included in the pretraining dataset?

It’s easy, you can copy/paste a snippet and it returns the location
from where it comes from.

https://huggingface.co/spaces/bigcode/search-v2a


 2. What is your code that is included in SWH archive?

Again, it’s easy: checkout some commit of your repository, then
inside this repository, you can run:

echo "https://archive.softwareheritage.org/swh:1:dir:$(guix hash -S git -f 
hex -H sha1 .)"

Do not miss the ’.’ (dot) once entering the repository.  This
command returns SWHID.  Other said, using this identifier, you might
know if the repository is stored by SWH.  (Be careful with temporary
artifacts as .go files or else.)

Or you can also check for one specific content:

  $ echo "https://archive.softwareheritage.org/swh:1:cnt:$(guix hash -S git -f 
hex -H sha1 COPYING)"
  
https://archive.softwareheritage.org/swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2

And the URL display the content of the file COPYING.  Here GPL 3
license for instance.


 3. Where such source code from #2 and #3 is packaged by Guix?
 

That said, if the source is hosted on GitHub or GitLab.com or SourceHut
or CodeBerg or some other popular forges or even mirrored without your
consent on one of these, please consider that your code had been
ingested by ChatGPT without any mean to verify.  Obviously, that’s not
an argument to accept the situation with HuggingFace and I understand
that you do not want that your publicly release copyleft source code
could be reused by any LLM.

However, as said several times, rooting this willing of non-inclusion is
larger than your own willing once you publicly released such source code
under some copyleft license.  I hope we agree on that.

Again, I am not trying to avoid something.  And again, we all have heard
your points.  Nothing is ignored.  To my knowledge, the path forward is
not yet well-defined.

Since we are discussing at length with various different inputs, it
means that a common understanding and/or opinion does not seem obvious.


>> Well, I do not know if the outcome will be aligned with your current
>> opinion, but be sure that your concerns as the others raised by Guix
>> community members are taking into account.
>
> Thank you for giving me an honest and detailed answer.

I feel you are pushy on the topic and for what my opinion is worth, it
is not helpful to raise again and again that you want a way to opt-out.
Yeah, people got it. :-) And you are probably not alone, I guess.

It would help if you could provide a source code that your wrote and
answer the three criteria above: included in pretraining dataset,
included in SWH, packaged by Guix.

I do not have special information from SWH but I am sure SWH people are
working on the topic.  And again, maybe the outcome will not be aligned
with your opinion.  Another story.

Now, the other question you ask to Guix: do we continue to help SWH in
harvesting?  You propose to stop, IIUC.  Ok, we got it, too. :-) From my
point of view, the path forward is not to speak on the abstract but to
root on concrete numbers; it would help in bounding what we are speaking
about.

Concretely, if you would like to be able to opt-out, could you point:

1. the piece from the Guix source code you are the author?

2. source code you are the author that is packaged by Guix?

Again, I am not trying to avoid the discussion.  Instead, I would prefer
to root the discussion on concrete examples.  Then it would appear to me
easier to make progress.

As Greg or Ekaitz also wrote: opting out has implications on the meaning
of freedom behind “free software“.

IMHO, that’s not because we would like to opt-out that we could, would
be able to or allowed to.  Therefore, instead of holding opinions on the
abstract, let try to make progress and start on the concrete: which
piece of source code are we speaking about?

Cheers,
simon

   

Re: Next Steps For the Software Heritage Problem

[Simon Tournier]

Hi MSavoritias, all,

Let me provide more context.

The concern started couple of months ago, to my knowledge.  And
discussion is still on going.  So I think that’s incorrect to say “any
result for over 6 months”.

Moreover, I feel you have a misunderstanding about HuggingFace and SWH
partnership.  From the reading of public information, HuggingFace and
BigCode trains on a subset of SWH source code archive.  I mean, it is a
snapshot and to my knowledge, they provided the list of source code that
had been used for training.

Not to avoid the question but from a pragmatic point of view, one might
ask if the source code you write and do not want to be included in the
training dataset, if this source code is concretely part of that
training dataset.

HuggingFace is not training continuously with source code from SWH.

And technically, SWH is an archive i.e., the code is not stored hot.  I
do not know and I have not read all details by HuggingFace of their
method; i.e., which kind of data they process – independent unique
files, complete repository, etc.  What I know is that the piece when
fetching from SWH is named SWH Vault; it requires to “cook” and prepare
all the files that take times, from minutes to days.


All that to say two key points:

1. People behind SWH are well-aware about various sides of the concerns.
As said, they are long-time free software supporters.  Be sure they have
eared community concerns.  Some discussions are still pending because as
explained, all sides of ethical questions needs to be cautious.

Please do not think it is ignored.


2. FWIW, I am in touch with SWH people – among other members from Guix
community.  For instance, in order to feed the discussion, Roberto from
SWH pointed to me this blog point by Bruce Perens:

https://perens.com/2019/10/12/invasion-of-the-ethical-licenses/

Well, I do not know if the outcome will be aligned with your current
opinion, but be sure that your concerns as the others raised by Guix
community members are taking into account.


Cheers,
simon

Re: Next Steps For the Software Heritage Problem

[Simon Tournier]

Hi Ian, all,

On Tue, 18 Jun 2024 at 10:57, Ian Eure  wrote:

> Guix is continuing to partner with SWH in spite of their continued 
> support of these violations.

Quickly because I am in the middle of a busy day. :-)

I think that LLM asks ethical and legal question that even FSF or EFF or
SFC does not provide clear answers.  (And that probably the level where
the discussion should happen.)  That’s not a light topic and we should
not rush in one definitive conclusion.

Thank you for the rise of the concern some weeks ago.  It appears to me
good that people had expressed their concerns.  And still does.
Although I am reading there or overthere an aggressive tone; useless.

Again, people behind SWH are long-term free software activists and be
sure that they do not take this concern lightly.  FYI, people of SWH are
in touch with some people from Guix to speak about all that.


1. Legal.

These license violations are your interpretation of the law and to my
knowledge nothing have been in Court, yet.

Today, it does not really matter if we (or I) share this opinion.
Because for now, it’s just an opinion.

However, no one is a lawyer here and drawing a clear line is not simple.

Thus, FWIW, I would not jump in hard conclusions based on my own opinion
because today I am not confidant enough to emit a definitive legal
position.


2. Ethical.

If we speak about ethical concerns, we need to be very cautious.  We all
share the same core of values about free software.  Then we all do not
bound these values to the same point.  Some of us extend them to some
topics, other restrict a bit.

Here the issue is that other values than the ones about free software
are dragged in the picture to emit a position.  That’s where we need to
be cautious because we need to embrace the diversity and do not morally
judge what is outside our free software project.

About SWH, FWIW, here is my moral reasoning; as you see, it is far to be
definitive.

I think that LLM/IA is morally bad in climate change context; a moral
value outside free software, BTW.  By extension, HuggingFace appears to
me morally bad.

Then, is SWH morally bad because they did a partnership with
HuggingFace?  Is it morally bad to help SWH in harvesting source code?
Well, the answers do not jump to my eyes.

An analogy could be: Am I morally bad when I use my Github account to
report bugs of free software there?  Or when I contribute to free
software hosted on Github?  Let do not drift; I am just trying to expose
that moral questions are often more complex that yes or no.

All is not 0 and 1.  There is tradeoff and balance.

Back to SWH.  I consider that free software source code is part of human
culture and it must be preserved. Preserving source code is morally
good.

Thus, I think the mission of SWH is morally good.  Because their
partnership with UNESCO in order to collect and preserve this human
culture is morally good.  Then, helping in that mission appear to me
morally good.

Moreover, being able to rescue is also morally good.  For example, in
scientific context where the trust in scientific knowledge depends on
software that vanish.  This trust appears to me vitally important.

Therefore, it appears to me very harsh to jump in definitive moral
conclusion about the SWH initiative.


All that said, back to my busy day. :-)

Cheers,
simon

Re: "guix pack -f docker" does too much work

[Simon Tournier]

Hi,

On Sat, 01 Jun 2024 at 15:58, Ludovic Courtès  wrote:

>> I think it would be great if "guix pack -f docker" could avoid building
>> all these identical layers again and again.  Perhaps it would be
>> possible to have a single derivation for each layer?  This way we
>> wouldn't have to recreate the same layer archives every time.
>
> That sounds nice in terms of saving CPU time.  It’s less nice in terms
> of disk usage: a single ‘guix pack -f docker’ run would populate the
> store with roughly twice the size of the closure.
>
> I think each solution (single derivation vs. one derivation per layer)
> makes a different tradeoff.  I don’t have a strong feeling about which
> one is better.

I share Ricardo wish.  From my perspective, I do not care much about
polluting my local Guix store when building Docker images.  Because all
that will be removed at the next GC – once all the work is loaded
elsewhere.

However, it appears frustrating to build again and again complete large
images when the difference is sometimes just a couple of packages.

I would be in favor to share more derivations between images. :-)

Cheers,
simon

Re: Postmortem of service downtime

[Simon Tournier]

Hi Ludo,

On Thu, 23 May 2024 at 19:31, Ludovic Courtès  wrote:

> From Sunday May 19th to Tuesday may 21st, for about 36h,
> bayfront.guix.gnu.org, the machine behind many services went down:
>
>   https://lists.gnu.org/archive/html/info-guix/2024-05/msg0.html
>
> Affected web sites and services included:
>
>   guix.gnu.org
>   bordeaux.guix.gnu.org
>   logs.guix.gnu.org
>   hpc.guix.info
>   foundation.guix.info
>   packages.guix.gnu.org
>   qa.guix.gnu.org

Oh, I am going outside of my “cellar“… I have not noticed. :-)

> Here’s the series of events that led to this:

Thanks for all the work and the detailed report!
And thanks Andreas for the rescue team.

Cheers,
simon

Re: Meetup in Nice on 22/06/2024

[Simon Tournier]

Hi,

On Mon, 27 May 2024 at 21:25, nat-418  wrote:

> I am organising a meetup for folks from the Guix, Nix, Aux, Lix, etc. 
> communities at Foam in Nice on Saturday, June 22nd. The address is 1 
> Place du Pin, and I will be there from lunch until evening. The venue is 
> a family-friendly bar/restaurant. If anyone wants to propose a talk or 
> discussion topic, feel free to reach out to me or reply here.

Cool!  Let us know how it is going.

Cheers,
simon

Re: Are 'guix gc' stats exaggerated?

[Simon Tournier]

Hi,

On Sun, 26 May 2024 at 13:13, Felix Lechner via "Development of GNU Guix and 
the GNU System distribution."  wrote:

> By my math, about 65.8 GiB were recovered.
>
> When 'guix gc' was done, it announced:
>
> [184389 MiB] deleting '/gnu/store/...'
> deleting `/gnu/store/trash'
> deleting unused links...
> note: currently hard linking saves 59224.03 MiB
> guix gc: freed 110,649.49 MiBs

Well, 180 GiB does not count deduplication, I guess.  And as Efraim
said, the ratio on average is 3:1 so 65 GiB vs 180 GiB seems consistent,
right?

However, the question is then: what are these 110 GiB?

Cheers,
simon

Extension for improving Guix search?

[Simon Tournier]

og-internship-wrap-up/

8: Re: Mechanism for helping in multi-channels configuration (and Xapian index)
Simon Tournier 
Mon, 06 May 2024 14:05:50 +0200
id:87pltzp2ld@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87pltzp2ld@gmail.com

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi Richard,

On lun., 13 mai 2024 at 20:52, Richard Sent  wrote:

> You're correct. This solution wouldn't be sufficient to avoid "Computing
> Guix Derivation" for every possible A or B. To my understanding it could
> reduce the frequency this occurs.

[...]

> Assuming D changes significantly less frequently than A, B, C..., I
> would think this should be something we could feasibly substitute (at
> least for recent D and Z).

Well, the package ’guix’ has changed 14 times over the past year.
Therefore, this D cannot be this package ’guix’, IMHO.

--8<---cut here---start->8---
$ git log --format="%cd %s" --since="1 year ago" | grep 'gnu: guix: Update' 
Mon May 13 18:22:53 2024 +0200 gnu: guix: Update to 7ca9809.
Tue Mar 12 14:27:01 2024 +0100 gnu: guix: Update to 4c94b9e.
Mon Mar 11 23:14:37 2024 +0100 gnu: guix: Update to 8f4ffb3.
Sat Dec 2 15:37:44 2023 +0100 gnu: guix: Update to 1.4.0-16.aeb494322c.
Thu Nov 30 07:15:36 2023 +0100 gnu: guix: Update to 1.4.0-15.e0885fcfbb.
Thu Nov 9 10:42:55 2023 +0200 gnu: guix: Update to a60ff46.
Fri Oct 6 12:26:44 2023 +0200 gnu: guix: Update to e863274.
Thu Sep 28 11:44:08 2023 +0200 gnu: guix: Update to d0438fc.
Mon Sep 18 12:31:52 2023 +0200 Revert "gnu: guix: Update to 
1.4.0-11.658de25e99."
Mon Sep 18 06:49:46 2023 +0200 gnu: guix: Update to 1.4.0-11.658de25e99.
Tue Aug 22 21:30:49 2023 +0200 gnu: guix: Update to 1.4.0-10.4dfdd82210.
Tue Aug 22 11:17:52 2023 +0200 gnu: guix: Update to 30355c1.
Mon Oct 2 09:28:02 2023 +0200 gnu: guix: Update to 1.4.0-12.b9fae146d6.
Mon Aug 21 18:44:49 2023 +0200 gnu: guix: Update to 0e6215a.
Fri Jun 9 22:11:14 2023 +0200 gnu: guix: Update to 44bbfc2.
--8<---cut here---end--->8---

Maybe I have a bad practise but here my “guix pull” history:

--8<---cut here---start->8---
$ guix pull -l | grep Generation
Generation 1nov. 17 2023 13:18:58
Generation 2déc. 11 2023 10:55:51
Generation 3févr. 02 2024 01:56:52
Generation 4mars 25 2024 18:22:25
Generation 5mai 13 2024 19:28:31(current)
--8<---cut here---end--->8---

Therefore, I am not convinced that replacing "Computing Guix derivation"
(build-aux/build-self.scm) by the package ’guix’ would be robust enough.

(Assuming another package ’guix’, lighter e.g., without requiring the
test suite, etc.)

All that said, any experiment – even if it appears at first clunky – is
very welcome!  This part will be improved only if there is a collective
effort / discussion / try, IMHO, i.e., by challenging the status quo. :-)

Cheers,
simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi,

On lun., 13 mai 2024 at 17:11, Richard Sent  wrote:

> Instead of A and B building C directly, A and B download the
> substitutable Guix package D, then use D to build C. Because D is a
> reproducible package, it should be substitutable from both A and B.
> Then, because D->C is the same for everyone, we could substitute the
> Guix derivation computation for C.

Maybe I am missing some details.  From my understanding, your D is the
result of the “Computing derivation” dance.  And it is a minimalist
build because it must work for both cases: with or without substitutes.

Somehow, my understanding of the current process is:

  A -> D  -> C
  B -> D* -> C

And, D or D* can be the same script.  Here, the property is a kind of
idempotency.  Hence, C is substitutable.

IIUC, your proposal is to fix D (the intermediary step).  You propose
the package named ’guix’ that changes barely, but it could be something
more minimalist.  The requirements is: susbtitutable.  The problem is
transferred to the first arrow, no?

How can we be sure that A and B points to the same D?

Other said, A lives in the past.  Later, we changed D becoming D*
because some bugs.  The other revision B lives after this change.
Therefore, we have the same picture:

  A -> D
  B -> D*

But then how do we go to C since D and D* does not have a kind of
idempotent property.

>From my understanding, the current situation/process is very similar
with the one for compiling compiler: you have your new Source of
compiler and an OldCompiler binary, so you run:

  OldCompiler(Source) -> Compiler
  and again Compiler(Source) -> NewCompiler

Somehow, whatever the choice of some OldCompiler binary, you get the
same NewCompiler binary (aside trusting trust attack and friends,
obviously ;-))

The story of “guix pull” is not so different; from my understanding.

Again, maybe I am missing something.


Cheers,
simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi,

On lun., 13 mai 2024 at 17:04, Edouard Klein  wrote:

> - Why is this step not substitutable ? The inputs are known, a hash can
> be derived, a substitute server could be queried for an output of that
> hash ? What am I missing ? Does the guix derivation not end up in the
> store ? What makes it so special that it can't be served by a substitute
> server ?

Assume we are running two different Guix revisions, say A and B.  And at
the end of our respective “guix pull”, we expect to have the same
revision, say C.  We expect to then run the same Guix.

Other said, how can we “compile” the code of C using one machinery from
A and another potentially different from B and expect to have the same
result?

Somehow, we need an intermediary step: something minimal that is
independent of A and B but produces the same C.  And it’s the aim of
“Computing derivation” with the script build-aux/build-self.scm.

The inputs are known, indeed.  However, the ones from A and from B are
not necessary the sames.  For instance, Guile of A might be different of
Guile of B.  Somehow, that “Computing derivation” is what allows to time
travel.

Well, that’s my understanding and I could have missing something.


For instance, I am running:

$ guix describe
Generation 4mars 25 2024 18:22:25   (current)
  guix 929ddec
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 929ddec8f4a181be653152c7436581c2adc54eee

Assume it is revision A.  From there, let run ’guix pull’.

--8<---cut here---start->8---
$ time guix pull
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 6ba29e0 (51 new commits)...
Building from this channel:
  guix  https://git.savannah.gnu.org/git/guix.git   6ba29e0
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 module-import  2KiB   305KiB/s 00:00 ▕██▏ 100.0%
 module-import-compiled  1.2MiB306KiB/s 00:04 ▕██▏ 100.0%
 compute-guix-derivation  1015B1.8MiB/s 00:00 ▕██▏ 100.0%
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/115prkj2rp58gl2lr0qy6higpliw8xpy-profile.drv
  /gnu/store/56dsm4f5snfmnfgkki4x41fj0xcl394m-guix-daemon.drv
  /gnu/store/77r2c33r1cq03qrrrfsf3g6mvmjp0w92-guix-command.drv
  /gnu/store/0y0fq1lmz3dwqi0fpbq4g6swkc9yljmp-guix-cli-modules.drv
  /gnu/store/bbczy39rid3q6j191mj397ak589s1101-guix-system-modules.drv
  /gnu/store/gx3ngbxmb6ayzjpxv9afsigvajc5d7h4-guix-extra-modules.drv
  /gnu/store/i0b6inna8a8bxg75k3dxr4xl6jrn53cm-guix-cli-core-modules.drv
  /gnu/store/iww6bi0ciaqkjjbqp26iclq2ijd1bgck-guix-system-tests-modules.drv
  /gnu/store/x0ai22yrq99kp3il0x3nf1d3yj5fydqy-guix-packages-base-modules.drv

70,7 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

 guix-package-cache  872KiB303KiB/s 00:03 ▕██▏ 100.0%
 guix-cli  315KiB  241KiB/s 00:01 ▕██▏ 100.0%
 guix-cli-core  922KiB 194KiB/s 00:05 ▕██▏ 100.0%
 guix-cli-modules  260B253KiB/s 00:00 ▕██▏ 100.0%
 guix-cli-core-modules  261B   450KiB/s 00:00 ▕██▏ 100.0%
 guix-extra  2.8MiB363KiB/s 00:08 ▕██▏ 100.0%
 guix-manual  5.2MiB   273KiB/s 00:19 ▕██▏ 100.0%
 guix-extra-modules  261B  187KiB/s 00:00 ▕██▏ 100.0%
 guix-packages-base  17.4MiB   271KiB/s 01:06 ▕██▏ 100.0%
 guix-system  7.9MiB   264KiB/s 00:31 ▕██▏ 100.0%
 guix-packages-base-modules  265B  279KiB/s 00:00 ▕██▏ 100.0%
 guix-system-modules  262B 282KiB/s 00:00 ▕██▏ 100.0%
 guix-system-tests  921KiB 332KiB/s 00:03 ▕██▏ 100.0%
 guix-system-tests-modules  267B   279KiB/s 00:00 ▕██▏ 100.0%
 guix-6ba29e021-modules  31.2MiB   299KiB/s 01:47 ▕██▏ 100.0%
 guix-module-union  2KiB   385KiB/s 00:00 ▕██▏ 100.0%
 guix-command  655B1.0MiB/s 00:00 ▕██▏ 100.0%
 guix-daemon  402B 711KiB/s 00:00 ▕██▏ 100.0%
 guix-6ba29e021  802B  499KiB/s 00:00 ▕██▏ 100.0%

[...]

real5m8,633s
user1m53,567s
sys 0m1,530s
--8<---cut here---end--->8---

Ok, all is substitutable (wow! 70MiB that’s not nothing… another story)
except the part “Computing derivation”.

Now, from another revisio

Re: Changing the defaults for --localstatedir and --sysconfdir?

[Simon Tournier]

Hi,

On jeu., 02 mai 2024 at 11:00, Ludovic Courtès  wrote:
> Richard Sent  skribis:
>
>> As everyone who's built Guix from source knows, when running ./configure
>> on a system with an existing Guix installation you must remember to
>> specify --localstatedir=/var and --sysconfdir=/etc. I think we should
>> consider whether those variables should default to those values.
>
> I think it would make sense.

I have always been for changing the defaults but GNU standards…

> Before I advocated that we should preferably stick to the GNU standards
> and have users make informed choices (that’s how we ended up with the
> ‘GUIX_CHECK_LOCALSTATEDIR’ macro that warns users but lets them
> explicitly pass ‘--localstatedir’.)

…and I have never understood this argument. :-)


Well, these GNU Standards read:

  ‘sysconfdir’

 [...]

  This directory should normally be /usr/local/etc, but write it as
  $(prefix)/etc. (If you are using Autoconf, write it as
  ‘@sysconfdir@’.)

  ‘localstatedir’

  [...]

  $(localstatedir) should normally be /usr/local/var, but write it
  as $(prefix)/var. (If you are using Autoconf, write it as
  ‘@localstatedir@’.)

https://www.gnu.org/prep/standards/standards.html#Directory-Variables


As Ricardo pointed, it could be confusing to have some defaults as GNU
Standards recommend and only 2 different.  Especially, $(prefix) might
appear elsewhere as recommendation: ’exec_prefix’ so then ’bindir’,
’sbindir’ etc., ‘datarootdir’ so ’datadir’, ‘sharedstatedir’,
‘includedir’, etc.

Therefore, setting also by default ’prefix’ as /. would mean ’bindir’
equals to /bin.  That’s fine… we need to adjust accordingly
guix-install.sh, to my knowledge.

Well, this ’prefix’ default matters only in one case: other distro
packaging Guix, right?

And I guess these distro do not run “make install” with the default GNU
Standards ’prefix’ recommendation (/usr/local/bin/) and modifies it to
/usr/, I guess.

On foreign distro:

$ ls -a /usr/local/bin/
.  ..  guix

This ’guix’ comes from guix-install.sh.

All in all, the surprise when defaulting ’prefix’, ’localstatedir’ and
’sysconfdir’ should be very limited, no?

Cheers,
simon

3 kinds of bootstrap (was Re: backdoor injection via release tarballs combined with binary artifacts)

[Simon Tournier]

Hi,

I am late to the party…


On mer., 10 avril 2024 at 15:57, Ludovic Courtès  wrote:

>> That has happened to me too.
>> Why not use Git directly always?
>
> Because it create{s,d} a bootstrapping issue.  The
> “builtin:git-download” method was added only recently to guix-daemon and
> cannot be assumed to be available yet:
>
>   https://issues.guix.gnu.org/65866

[...]

> I think we should gradually move to building everything from
> source—i.e., fetching code from VCS and adding Autoconf & co. as inputs.
>
> This has been suggested several times before.  The difficulty, as you
> point out, will lie in addressing bootstrapping issues with core
> packages: glibc, GCC, Binutils, Coreutils, etc.  I’m not sure how to do
> that but…

[...]

> … live-bootstrap can probably be a good source of inspiration to find a
> way to build those core packages (or some of them) straight from a VCS
> checkout.

IMHO, we need to distinguish because there is different types of issues
and thus different potential workarounds. :-)

  1. Bootstrap how to download source code.
  2. Bootstrap how to build core packages.
  3. Bootstrap the driver (say guix-daemon and helpers).

Well, having solutions for #1 and #3 would naturally provide a solution
for #2.  Although the devil is about details. ;-)


About #1


You cannot use the binary ’git’ in order to download the source code of
Git to build the binary ’git’.  Yeah, circular dependency. :-)
Therefore, Git source code is pulled using another method, say from
tarball, such method which also needs to be built from source, so it
also needs yet another method.  The usual chicken-or-the-egg problem.

The current workaround is to “hide” the problem and introduce a
“builtin:download” method: it’s an “opaque” binary that is hard to
inspect.  Roughly, the workaround had been introduced by [1] on
Oct. 2016.  Almost 8 years ago, so it works! :-)

The argument for accepting this “opaque” method is because it is a
fixed-output derivation.  Other said, we know beforehand the SHA256
checksum.  Thus the claim is: being “opaque” does not matter because the
SH256 checksum can be computed independently and all the source code can
be audited.

For cutting another cycle, another “opaque” had be introduced:
“builtin:git-download”.  All applies similarly.

Do not take me wrong with “opaque”.  I mean that the method depends on
the couple user-revision and daemon-revision.  Other said, it is not
straightforward to know when Alice and Bob are using the exact same
method for downloading source code.  Since it is not fully transparent,
it is “opaque”. :-)

Somehow we are applying to all what we need for cutting a specific
circular dependency.  We have some packages named ’foo-bootstrap’ that
are aimed to solve some dependency problem about packages, then we do
not use them for all; we just use them for cutting a circular
dependency.  I think a similar strategy should be applied for the fetch
methods.

We could have “git-fetch” relying on the initial Git method, i.e., a
transparent derivation where it’s straightforward to audit all: the
dependencies and the builder.

And for some specific cases, we could have “git-fetch/bootstrap” relying
on “builtin:git-download”.  It eases to know which packages are very
important to care.

I think that “builtin:download” and “builtin:git-download” applied to
all “url-fetch” and “git-fetch” both downgrade the complete transparency
level for solving very specific bootstrapping problem.

Last about #1, please note that the transparency does not come for free
and has drawbacks: when running say “guix time-machine -C past.scm --
build -S”, all the dependencies for downloading would be the ones of
past.scm.  Other said, for downloading today the source code of a 5
years old package, say using ’hg-fetch’, we need Python and Mercurial as
they were 5 years ago – when we do not expect any difference on the
content with the Python and Mercurial of today.


About #3


That’s the very hard topic!  The bootstrapping story is not fully done
yet.

Assuming trust for #1, the bootstrap of Guix starts with
’bootstrap-seeds’, roughly 232KiB.  Take a moment, that’s impressive, :-)
right?

Obviously, I let aside Haskell, Ocaml@5 etc.

Well, diving further.  These 232K alone are not enough.  It also
requires helpers: tar (1.3MiB), bash (1.3MiB), mkdir (0.7MiB) and xz
(0.844MiB).

More, it requires two drivers: static Guile binary (14MiB) and
guix-daemon.

You get it: How to trust these helpers?  Two approaches: (a) implement
something directly in hex/assembler and/or (b) exploit the Guile binary
(à la Scheme on bare metal).

About guix-daemon, one solution is a daemon directly in Guile, and
compatible with the very Guile binary.  Or at least, a minimalist daemon
with just enough features for building up to guix-daemon.

Or another option is the “Extreme bootstrapping” [3] – my understanding
of live-bootstrap.  Somehow, remove guix-daemon from the picture and
convert the derivat

Re: Mechanism for helping in multi-channels configuration (and Xapian index)

[Simon Tournier]

Hi,

Sorry for the long delay.

On lun., 18 mars 2024 at 16:05, Christina O'Donnell  wrote:

>> 2: https://issues.guix.gnu.org/issue/39258

> As I said above, [2] is a fairly long thread, but I think I get the 
> general idea. It seems that Xapian was implemented but didn't have the 
> desired speedup. Am I getting the right impression there?

Not really.  From my memories, the first blocker for implementing search
with Xapian was adding Xapian as dependency of Guix.  This addition
would be a bad idea, IMHO.  However…

At the time of discussing Xapian-based “guix search”,
GUIX_EXTENSIONS_PATH was at its infancy.  Therefore, it was not really
on the table.

… Xapian-based “package search” appears to me an option if it is turned
into a Guix extension.  This way, adding Xapian as dependency is not for
all but only for those who want more features. :-)

I have on my TODO list to resume the work:

 1. Benchmark Xapian-based search
 2. Benchmark Xapian index building

Then depending on that, it draws the directions.


Cheers,
simon

Re: Scheduling a new release?

[Simon Tournier]

Re,

On lun., 06 mai 2024 at 13:12, Simon Tournier  wrote:

> Although these days I do not have much free time, let make a new release
> as soon as possible.  WDYT?
>
> Who’s in?

Well, the patch review sessions could be helpful.  Maybe we could run
some online hackathons.  IMHO, having a schedule together would help –
at least me ;-) – to keep the flow until crossing the finish line.

WDYT?

Cheers,
simon

Scheduling a new release?

[Simon Tournier]

Hi all,

Here or there, we have bugs as:

https://issues.guix.gnu.org/70659
https://issues.guix.gnu.org/70726

And our answer looks like:

> Additionally, I strongly advise upgrading guix-daemon, as noted in the
> bug report above.

Well, the bugs appear because the user is upgrading guix-daemon. ;-)

In both cases (#70659 and #70726), it comes from a fresh install (latest
release v1.4.0) and then the first ’guix pull’ aiming to upgrade all
leads to that reported error.

Therefore, I strongly advise upgrading latest Guix release. ;-)


Although these days I do not have much free time, let make a new release
as soon as possible.  WDYT?

Who’s in?

Cheers,
simon

time-bomb and CI? (was bug#69800: kcalendarcore time-bomb)

[Simon Tournier]

Hi,

Reading this message [1]:

 Start of forwarded message 
Subject: bug#69800: kcalendarcore is a time bomb
To: 69...@debbugs.gnu.org
Date: Thu, 14 Mar 2024 20:20:43 +0100
From:  Vivien Kraus via Bug reports for GNU Guix 

Dear Guix,

Kcalendarcore does not build anymore. According to CI, it stopped
working this march, on gnome-team and rust-branch (I can’t build it
either on master).

http://ci.guix.gnu.org/search?query=kcalendarcore

Best regards,

Vivien
 End of forwarded message 

and this fix:

* gnu/packages/kde-frameworks.scm (kcalendarcore) [#:phases]: Add
'disable-failing-test.

My question is: Could we automatically detect by CI such time-related
test?


Other said, would it be possible to have one (or more) node that build
with a date in the future?  Say 3 years.  In Bordeaux?  Or in Berlin?
Using the “Dolorean VM” [2], it could be nice to setup one node.

Some packages would be build by this node.  Somehow, it means some
random-selected packages would be built in the future.  All in all and
considering the rate of building, it would help us to detect problematic
packages.  And fix them now avoid to break “guix time-machine” in the
future.

Well, I am trying to have a more systematic approach here at work – bah
not sure it will see the light –, so still awaiting, having one node in
the future would help, IMHO.

WDYT?

Cheers,
simon

PS: Do we speak about bug of year 2038? ;-)

1: bug#69800: kcalendarcore is a time bomb
Vivien Kraus via Bug reports for GNU Guix 
Thu, 14 Mar 2024 20:20:43 +0100
id:994ee26ab0444037bcc93ee91af18e5a0b389c33.ca...@planete-kraus.eu
https://issues.guix.gnu.org/69800
https://issues.guix.gnu.org/msgid/994ee26ab0444037bcc93ee91af18e5a0b389c33.ca...@planete-kraus.eu
https://yhetil.org/guix/994ee26ab0444037bcc93ee91af18e5a0b389c33.ca...@planete-kraus.eu

2:
https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment

Re: Guile CSE elimination of record accessor?

[Simon Tournier]

Hi Andy,

Thanks for the explanations.

On mar., 30 avril 2024 at 16:43, Andy Wingo  wrote:

>> The first question is: is it still correct?  Because this module had
>> been implemented before many Guile compiler improvements.
>
> No, the comment is incorrect.  The type check on whatever accessor is
> called first (unspecified in scheme; probably we should just bite the
> bullet and do predictable left-to-right semantics, as racket does) will
> dominate the rest and eliminate those checks.  The assert-type is
> unnecessary.

Good to know.

> To see this, do ,optimize-cps at the repl, and count the number of
> e.g. struct? checks with and without the assert-vlist.  There is only
> one, either way.

Hum, I am not sure to understand how to use ,optimize-cps at the repl.
Naively, I get:

--8<---cut here---start->8---
scheme@(guile-user)> ,use(ice-9 vlist)
scheme@(guile-user)> ,optimize-cps (vlist-cons 'foo vlist-null)
L0:   ;  at :102:14
  v0 := self
  L1(...)
L1:
  receive()
  v1 := current-module()  ; mod While executing 
meta-command:
In procedure +: Wrong type argument in position 1: #f
--8<---cut here---end--->8---

Since ’,help compile’ reads,

,optimize-cps EXP [,optx] - Run the CPS optimizer on a piece of 
code and print the result.

I assume that I do not feed with the correct expression EXP.  What would
be the invocation?


>   (A type check is a heap-object? check, then struct?,
> then get the vtable, then check against the global variable .
> All of these duplicates get eliminated.)

Ah yeah, it makes sense. :-)

Cheers,
simon

Guile CSE elimination of record accessor?

[Simon Tournier]

Hi,

In Guile module (ice-9 vlist), one reads:

--8<---cut here---start->8---
;; Asserting that something is a vlist is actually a win if your next
;; step is to call record accessors, because that causes CSE to
;; eliminate the type checks in those accessors.
;;
(define-inlinable (assert-vlist val)
  (unless (vlist? val)
(throw 'wrong-type-arg
   #f
   "Not a vlist: ~S"
   (list val)
   (list val

[...]

(define (vlist-head vlist)
  "Return the head of VLIST."
  (assert-vlist vlist)
  (let ((base   (vlist-base vlist))
(offset (vlist-offset vlist)))
(block-ref (block-content base) offset)))
--8<---cut here---end--->8---

Other said, the argument ’vlist’ is “type-checked” with ’assert-vlist’
and thus that is exploited by Guile compiler, if I understand correctly
the comment.

The first question is: is it still correct?  Because this module had
been implemented before many Guile compiler improvements.


The second question, if the comment above is still valid, is: could we
also “win” for some record inside Guix source code?

Concretely, one example about the record , there is some
procedures such that:

--8<---cut here---start->8---
(define* (package->manifest-entry package #:optional (output "out")
  #:key (parent (delay #f))
  (properties (default-properties package)))
  "Return a manifest entry for the OUTPUT of package PACKAGE."
  ;; For each dependency, keep a promise pointing to its "parent" entry.
  (letrec* ((deps  (map (match-lambda
  ((label package)
   (package->manifest-entry package
#:parent (delay entry)))
  ((label package output)
   (package->manifest-entry package output
#:parent (delay entry
(package-propagated-inputs package)))
(entry (manifest-entry
 (name (package-name package))
 (version (package-version package))
 (output output)
 (item package)
 (dependencies (delete-duplicates deps))
 (search-paths
  (package-transitive-native-search-paths package))
 (parent parent)
 (properties properties
entry))
--8<---cut here---end--->8---

which fits the comment above: a record as argument and record accessor
call.

And that could also be applied to other records, I guess.


Any answers, explanations or references are very welcome. :-)

Cheers,
simon

PS: Raining day and weird pastime… diving into Guile source code. ;-)

Re: Adding plumbing subcommand 'derivation'?

[Simon Tournier]

Hi,

On ven., 19 avril 2024 at 16:02, Ludovic Courtès  wrote:

> We should see how that fits into the set of tools we already have, in
> particular the (guix derivations) interface and the REPL meta-commands.
>
> My gut feeling, with a Schemer bias, is that we’d rather enrich the
> Scheme API and/or REPL than add more commands (this is not Nix :-)).
>
> But I don’t know, maybe we can have both?

Yes, I think that’s orthogonal and a good idea.

Maybe a new plumbing generic subcommand, as “guix inspect” or “guix
store”, where showing the fields of a derivation would be a
sub-subcommand.  For instance, diffing two derivations from the
command-line seems helpful when  debugging – I often do that with Emacs
exploiting buffer facilities. :-)

In addition, it could be helpful to improve the readability for the
pretty-printer.  Other said, somehow redesign this:

--8<---cut here---start->8---
(set-record-type-printer! 
  (lambda (drv port)
(format port "# ~a ~a>"
(derivation-file-name drv)
(string-join
 (map (match-lambda
   ((_ . output)
(derivation-output-path output)))
  (derivation-outputs drv)))
(number->string (object-address drv) 16
--8<---cut here---end--->8---

It looks like a plan. ;-)

Cheers,
simon

Re: bug#63267: gcc-toolchain is missing libstdc++.so

[Simon Tournier]

Hi,

On mer., 17 avril 2024 at 05:21, John Kehayias via Bug reports for GNU Guix 
 wrote:

> I've just pushed, as b47ae1ecc43baaf726701ab2d2f810ecfaa75428,

Cool!  Thank you for crossing the finish line.

Cheers,
simon

Adding plumbing subcommand 'derivation'?

[Simon Tournier]

Hi Leo,

On ven., 12 avril 2024 at 20:04, Leo Famulari  wrote:

>> Do you think it would be useful to package it?  Or maybe to include it
>> as another subcommand (or part of some subcommand)?
>
> I'd love for this to be built in to Guix. I'm often struggling to read
> derivations while debugging or analysing something.

So I propose to add the plumbing command ’derivation’.  Any objection?

Cheers,
simon

Guix extension to display derivation (and rewrite fixed-output)

[Simon Tournier]

Hi,

As an Emacs user, exploring Derivation (the .drv files) is easy since
there is an Emacs mode.  However, I have been annoyed with some pipe
through ’sed’ and friends.

Therefore, I wrote a very simple Guix extension [1] that outputs
derivation content using recutils format.

--8<---cut here---start->8---
$ guix drv-show $(guix build hello -d)
name: /gnu/store/qr00sgbh3vwwqswmgjjymg6wkys9r4i2-hello-2.12.1.drv
outputs:
+ /gnu/store/6fbh8phmp3izay6c0dpggpxhcjn4xlm5-hello-2.12.1   [out]
inputs:
+ /gnu/store/3ds56xg6njpw6hnp2w4xpx4psw5mka5q-glibc-2.35.drv
[out]
+ /gnu/store/3zh2qpi897s2x229s93iakji86b08a20-hello-2.12.1.drv  
[out]
+ /gnu/store/5bqhdbbl71r9r936w6w8zzqlk41md3wx-glibc-2.35.drv
[out]
+ /gnu/store/67nh3fzviy3q4s8ar8cg0dzhyzgwrwdd-module-import-compiled.drv
[out]
+ /gnu/store/7fsz44vifdc0ws0amnpwnmig3ra6hb53-gcc-11.3.0.drv
[lib]
+ /gnu/store/fchdaawcrxb35llbl7fj7lcsq5asmk4b-guile-2.0.14.drv  
[out]
+ /gnu/store/ky030dkfkfr3l8xgdbv45j6bs87988lx-gcc-11.3.0.drv
[lib]
+ /gnu/store/n9kblf5cx4lphrydjr90sp3zfvcdr1pb-glibc-utf8-locales-2.35.drv   
[out]
system: x86_64-linux
builder:
+ /gnu/store/4p1l5bdxxbyyqc3wh0d07jv9rp1pdcy7-guile-2.0.14/bin/guile
+ --no-auto-compile
+ -L /gnu/store/a6acf6dds8s9fw7dp5div03rwik0x4x2-module-import
+ -C /gnu/store/yk897hj2p5mdx6hw47s90n8x9pn6s36c-module-import-compiled
+ /gnu/store/fiy8arwqm8vwaqs4h8b361kbmjmd1yra-hello-2.12.1-builder
environment:
+ allowSubstitutes: 0
+ guix properties: ((type . graft) (graft (count . 2)))
+ out: /gnu/store/6fbh8phmp3izay6c0dpggpxhcjn4xlm5-hello-2.12.1
+ preferLocalBuild: 1
--8<---cut here---end--->8---

Do you think it would be useful to package it?  Or maybe to include it
as another subcommand (or part of some subcommand)?

Let me know. :-)

My motivation to do it now is this story:

https://simon.tournier.info/posts/2024-04-11-rewrite-drv.html


Cheers,
simon

1: https://gitlab.com/zimoun/guix-drv

Re: Concerns/questions around Software Heritage Archive

[Simon Tournier]

Hi,

On lun., 18 mars 2024 at 12:38, Ian Eure  wrote:

> They appear to be violating free software licenses on large scale. 
> They are in violation of SWH’s own positions.

[...]

> [1]: https://arxiv.org/html/2402.19173v1
> [2]: 
> https://huggingface.co/spaces/HuggingFaceH4/starchat2-playground
> [3]: https://huggingface.co/datasets/bigcode/the-stack-v2
> [4]: https://github.com/bigcode-project/opt-out-v2/issues

Please note that Software Heritage folks are not co-author of all that;
or I misread.  Do not take me wrong, this is not an attempt to escape
but a query for waiting the feedback of SWH.

As Ludo said, SWH folks are, by the way, also long time Free Software
activists.  For the record, the quality of 10 Years of Guix [1] videos
is the result of tireless work (for free!) by a Debian video team member
(also working for SWH) and one of SWH co-founder had been Debian project
leader.  Let the benefit of the doubt while waiting.

1: https://10years.guix.gnu.org

Cheers,
simon

PS: Thanks for the detailed explanations.  I will provide my reading
later, after some concerns will be separated, eventually.

Re: Concerns/questions around Software Heritage Archive

[Simon Tournier]

Hi MSavoritias,

On lun., 18 mars 2024 at 16:00, MSavoritias  wrote:

> I think you have misunderstood that here we are talking about

> I think you have misunderstood that here we are talking about

What if? Maybe it’s you.  Maybe you, “you have misunderstood that here
we are talking about […]”.

For what my opinion is worth here, I would prefer that you do not assume
on what I might have understood.  Similarly, I am not assuming anything
about your understanding of the various topics at hand.

That’s my last message in this thread.

Cheers,
simon

Re: Concerns/questions around Software Heritage Archive

[Simon Tournier]

Hi MSavoritias,

On lun., 18 mars 2024 at 13:47, MSavoritias  wrote:

> 1.
>
> You seem to be misunderstanding the statement here that was said.
>
> What you can do legally and what you can do socially are not always the 
> same thing.

I do not read where I wrote something like that but anyway.

A program is free software if the program's users have the four
essential freedoms: [1]

  0. The freedom to run the program as you wish, for any purpose.
  1. The freedom to study how the program works, and change it so it does
 your computing as you wish. Access to the source code is a precondition
 for this. 
  2. The freedom to redistribute copies so you can help others.
  3. The freedom to distribute copies of your modified versions to
 others. By doing this you can give the whole community a chance to
 benefit from your changes. Access to the source code is a precondition
 for this.

All is about the philosophy of “free software”.

1: https://www.gnu.org/philosophy/free-sw.en.html


> As advice for the future when somebody says a concern or wish they have, 
> your first statement shouldn't be "but its legal" because that 
> completely dismisses any constructive discussion that could be done.

Again, I am not arguing about “legal” something.  Instead, I am pointing
that this wish does not match the principles of “free software”.

If you accept that the software you create is “free software” then you
cannot complain if this “free software” is used in some contexts that
you consider unethical.

That’s the double sword of “free software”.

Do I consider LLMs as something unethical?  I think yes: most AI appears
to me unethical but that’s another story (rooting my arguments in
arguments about energy [2,3,4]).

2: https://social.sciences.re/@zimoun/112082437445032973
3: https://social.sciences.re/@zimoun/112039562095800532
4: https://social.sciences.re/@zimoun/112038609631116527


> What is in question here is whether Software Heritage respects people 
> enough to do the right thing and respect their wishes without getting 
> lawyers/legal involved.

Again, this is an incorrect frame, IMHO.  Software Heritage (SWH) do the
things you granted them to do.  SWH respects the “ethical” definition of
“free software”.

Again, do I think that feeding LLM after publishing a statement for LLM
code is a good move?  I do not know…  Does it break my ethical values?
Maybe…  Can I complain about my contributions to “free software” reused
in a way that I might consider unethical?  No.

5: https://www.softwareheritage.org/2024/02/28/responsible-ai-with-starcoder2/
6: https://www.softwareheritage.org/2023/10/19/swh-statement-on-llm-for-code/


> Besides with the way you are framing Free Software as not respecting any 
> social rules then that makes Free Software not attractive which is the 
> opposite of what we are trying to do here :)

I do not know what are the “social rules” of “free software”.  At best,
I understand the social rules of a community working on free software.

And this community is far to be an homogeneous whole with clear social
rules.  These social rules vary and the only shared denominator is the
“free software” principles defined by four freedoms.

The only question might be: by allowing ingested source code to be used
to train LLM, is Software Heritage aligned with the values that the Guix
community promote?

To be honest, I cannot answer to that question in a hurry.


> 2.
>
>  > Somehow, a Content-Addressed system is designed around immutable 
> > content. And if one know how to implement a Content-Addressed system 
> > relying on mutable content, I would be very interested to know more 
> > about it.
>
> Please refrain from doing such remarks. Nobody here suggested anything 
> that you mention here and you effectively devalue the discussion by 
> arguing like this and frame other people as stupid.

I will not refrain to say: Talk is cheap!

Positions about the situation with “rewrite history” cannot be a
discussion about opinions but it needs to be rooted in how it
technically works and what does it mean Content-addressed system.


> 3.
>
> You may disagree with this sure, but shutting down the discussion 
> because nobody wrote the code for you is very elitist of you.

We are speaking about which discussion because I am lost.  About LLM or
about “rewrite history”?

About LLM, see point #1.

About “rewrite history”, see point #2


> 4.
>
>  > This language is not acceptable on Guix channel of communication.
>
> Calling out transphobia it is very much accepted here actually :)

No it is not.  Because it is a bold conclusion.

I am asking that the Guix project rewrite right now its history:
changing my identity ’zimoun’ to my identity ’Simon Tournier’.  Since
the Guix project will take the time to check, then I will claim: the
Guix project is French-ph

Re: rewriting history; Was: Concerns/questions around Software Heritage Archive

[Simon Tournier]

Hi,

On lun., 18 mars 2024 at 12:10, MSavoritias  wrote:

> The right of a trans person to ask a project to not advertise their 
> deadname was never in question.
>
> Guix is a place that supports trans people and anybody else that wants 
> to change their name.

There is a difference between “advertise” and “part of the history”.

Do not take me wrong.  The right to be forgotten is one topic.  However,
as many people are saying: it is not an easy question.  There is legal
questions, technical questions, social questions, etc.

For what it is worth, Guix is built around the concept of immutability.
This is a core concept and deep in Guix internals.

Therefore, it would be more constructive if you come with a
proof-of-concept allowing “history rewrite” and strong “software
identification” property [1].  Else, the discussion is leading nowhere,
IMHO.

1: https://guix.gnu.org/en/blog/2024/identifying-software/

Cheers,
simon

Please hold your horses

[Simon Tournier]

Hi MSavoritias,

Could you please stop to propagate tangential or opinionated views?
Please hold your horses.

You wrote several times, about Software Heritage:

>  being also transphobic.

[…]

> I would go a step further actually. Software Heritage is effectively 
> breaking CoC of Guix now.

[…]

>  Software Heritage 
> implements a process that respects trans rights Software Heritage should 
> not be welcome in Guix Spaces.

[…]

>  Software 
> Heritage is breaking CoC here.

This language is not acceptable on Guix channel of communication.

It appears to me much better to stay open and let the benefit
of the doubt.  Let avoid bold conclusions and prefer constructive
arguments.

For instance, I refrain to qualify your opinion because it would not be
helpful… So I apply my own advice letting you the benefit of the
doubt.

Cheers,
simon

Re: Concerns/questions around Software Heritage Archive

[Simon Tournier]

Hi,

On sam., 16 mars 2024 at 08:52, Ian Eure  wrote:

> They appear to be using the archive to build LLMs: 
> https://www.softwareheritage.org/2024/02/28/responsible-ai-with-starcoder2/

About LLM, Software Heritage made a clear statement:

https://www.softwareheritage.org/2023/10/19/swh-statement-on-llm-for-code

Quoting:

We feel that the question is no longer whether LLMs for code
should be built. They are already being built, independently of
what we do, and there is no turning back.  The real question is
how they should be built and whom they should benefit.

Principles:

1. Knowledge derived from the Software Heritage archive must be
given back to humanity, rather than monopolized for private
gain. The resulting machine learning models must be made available
under a suitable open license, together with the documentation and
toolings needed to use them.

2. The initial training data extracted from the Software Heritage
archive must be fully and precisely identified by, for example,
publishing the corresponding SWHID identifiers (note that, in the
context of Software Heritage, public availability of the initial
training data is a given: anyone can obtain it from the
archive). This will enable use cases such as: studying biases
(fairness), verifying if a code of interest was present in the
training data (transparency), and providing appropriate attribution
when generated code bears resemblance to training data (credit),
among others.

3. Mechanisms should be established, where possible, for authors to
exclude their archived code from the training inputs before model
training begins.

I hope it clarifies your concerns to some extent.


Moreover, you wrote: « I want absolutely nothing to do with them. »

Maybe there is a misunderstanding on your side about what “free
software” and GPL means because once “free software”, you cannot prevent
people to use “your” free software for any purposes you dislike.

If you want to bound the use cases of the software you create, you need
to explicitly specify that in the license.  And if you do, your software
will not be considered as “free software”.

That’s the double sword of “free software”. :-)

Cheers,
simon

Content-Addressed system and history?

[Simon Tournier]

Hi,

On sam., 16 mars 2024 at 08:52, Ian Eure  wrote:

> I was also distressed to see how poorly they treated a developer 
> who wished to update their name: 
> https://cohost.org/arborelia/post/4968198-the-software-heritag 
> https://cohost.org/arborelia/post/5052044-the-software-heritag

This asks two questions, IMHO.

1. Can the future you decide who were the past you?

2. What is Content-addressed system?


About #1, that’s somehow a philosophical question. :-)

That’s what the question about changing the public identity asks: you
can act on who you are and who you want to be but because the time is
not reversal, sadly, you cannot change who you were.  It is not possible
to collectively rewrite the history.

Allowing such process leads to dangerous consequences, IMHO.  That’s
another story. :-)

Do not take me wrong.  That’s still an open question and the right to be
forgotten is a topic by itself, e.g., legal.  We will not address it in
the Guix project.



About #2, that’s a technical question.

By definition of a Content-Addressed system, the key associated to the
value is computed by a procedure depending only on the content itself.
Therefore, change the content then change the key.

Git [1] is probably the tool that have popularized that.  Consider a
project using Git and you clone it.  Now, you have a complete copy of
many keys associated to many contents, and also many links between the
keys themselves.  For instance, the key of the object ’Git commit’
depends on its content which depends on the key of the object ’Git
tree’.

Now, if you rewrite any content, then it rewrites the key.  As pointed,
this change might propagate.

All the question becomes the authority.  Because I also have another
copy/clone with the initial set of keys and you have now modified ones,
how do we agree what are the right ones?

Well, at the size [2] of linked posts, the Git history rewriting is
affordable.  Now, I am not convinced that the person would try – or even
think of – such if this project would have hundreds of contributors and
thousands of users.  That’s my opinion and I agree it is not an
argument. :-)

At the level of Guix, allowing a mutable history implies a random
availability of binary substitutes.

To be explicit, rewrite the Git history of Guix implies the break of:

 + local Git repositories of Guix developers
 + regular Guix users and the trust mechanism
 
Somehow, a Content-Addressed system is designed around immutable
content.  And if one know how to implement a Content-Addressed system
relying on mutable content, I would be very interested to know more
about it.


Cheers,
simon


1: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
2: https://github.com/rspeer/python-ftfy/graphs/contributors

Re: Helping with abandoned patches

[Simon Tournier]

Hi Greg,

On mer., 17 janv. 2024 at 12:17, Greg Hogan  wrote:
> What is the preferred process for when a patch review is provided
> (often by a committer) but no response is received from the submitter
> (for many weeks or months)?
>
> Is it appropriate to make the recommended changes and submit an updated patch?

Yes because the aim is to improve the code so the review is not lost.

Moreover, in that case of no response after a delay, if the reviewer
does not have commit access, the best (more helpful) seems sending by
the reviewer or anyone else a new version including the recommended
changes.

Cheers,
simon

Re: Check for ANSI compliance

[Simon Tournier]

Hi,

On lun., 29 janv. 2024 at 17:44, Ludovic Courtès  wrote:

> That used to be the case until commit
> 672d3d4a87839b0692c307df0edb66cd16bcbf1a, which enabled colors even when
> ‘INSIDE_EMACS’ is set.
>
> Pierre, do you remember what the rationale was?

I am not Pierre. :-)  The context seems:

Guix search, colors and INSIDE_EMACS
Pierre Neidhardt 
Tue, 04 Feb 2020 16:23:43 +0100
id:87blqeml4w@ambrevar.xyz
https://lists.gnu.org/archive/html/guix-devel/2020-02
https://yhetil.org/guix/87blqeml4w@ambrevar.xyz

Quoting [1]:

--8<---cut here---start->8---
>  new d7545a6  ui: Only display link in capable terminals.
>  new 672d3d4  ui: Don't disable colors when INSIDE_EMACS is set.

Forgive me if I missed the discussion, but I thought we had reached
rough consensus in favor of the status quo.  What happened?
--8<---cut here---end--->8---

Then quoting [2]:

--8<---cut here---start->8---
I’ve reverted it in c2f9ea2b502a617bb69227d5f858eee9d4288a6a, also
because if was causing a test failure.
--8<---cut here---end--->8---

where c2f9ea2b502a617bb69227d5f858eee9d4288a6a reads,

--8<---cut here---start->8---
Revert "ui: Only display link in capable terminals."

This reverts commit d7545a6b538813e88195d084f75a3e87065c999e.

The commit led to a test failure in 'tests/guix-package-net.sh'.  It
also led to disagreements discussed here:

  https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00353.html

Reverting until these are addressed.
--8<---cut here---end--->8---

and thus 672d3d4 had not been reverted.  Let revert it since it does not
make sense without d7545a6, IMHO.

Cheers,
simon

1: Re: branch master updated (0aa0e1f -> 9b7f9e6)
Ludovic Courtès 
Mon, 24 Feb 2020 16:31:06 +0100
id:87blpom285@gnu.org
https://lists.gnu.org/archive/html/guix-devel/2020-02
https://yhetil.org/guix/87blpom285@gnu.org

2: Re: 01/03: ui: Only display link in capable terminals.
Ludovic Courtès 
Fri, 28 Feb 2020 00:16:25 +0100
id:87wo87aaeu@gnu.org
https://lists.gnu.org/archive/html/guix-devel/2020-02
https://yhetil.org/guix/87wo87aaeu@gnu.org

Re: Request to add a go-team branch and set it on CI.

[Simon Tournier]

Hi,

CC: guix-maintainers
CC: guix-sysadmin

On lun., 22 janv. 2024 at 19:58, Sharlatan Hellseher  
wrote:

> May I ask someone with admin rights to the build farm to set up
>  go-team branch, please?

What is the status of this request?  Is it doable?  I do not see the
specification on .  Do I miss something?

Cheers,
simon

Re: Building container images with nix2container

[Simon Tournier]

Hi Antoine,

Reading this blog post:

https://lewo.abesis.fr/posts/nix-build-container-image/

and from my understanding, “guix pack” is currently something similar to
’dockerTools.buildImage’ [1]

On lun., 26 févr. 2024 at 18:33, Simon Tournier  
wrote:

> Well, I have not followed on which strategy Guix relies.  What is the
> one of nix2container?  The one described here:
>
> https://grahamc.com/blog/nix-and-layered-docker-images/

To answer to my question, the way to build the container image is
different, hence it does not make much sense to speak about a
“strategy“. :-)

However, the blog post says:

To address this issue, we could add a nonReproducible option in
the containerTools.buildLayer function. Instead of only storing
the digest, we would also store the tar. Note in practice, an
important part of nixpkgs is bit reproducible and this would
rarely be needed.

And so the question is how do you know beforehand if the flag
’nonReproducible’ must be applied or not?

Indeed, the approach of nix2container could be helpful in addition to
‘guix pack’.  Maybe an extension… :-)

Cheers,
simon

Re: Introduce Cuirass and remove Hydra on https://www.gnu.org/software/devel.html

[Simon Tournier]

Hi,

Well, I do not see if there is a reply.  If no, sorry!  If yes, I am
just adding my own curiosity. :-)

CC: guix-maintainers and guix-sysadmin
CC: Mark Weaver

On lun., 08 janv. 2024 at 13:19, Jing Luo  wrote:

> I am Jing Luo, a new member from the GNU webmaster team. I noticed that 
> gnu.org has a page on "GNU Development Resources" [1], which has a 
> section about "Hydra: Continuous builds and portability testing". As I 
> know, Guix now uses Cuirass for CI. The text [1] has not been updated 
> since 2012. Would anyone on this list be interested in writing something 
> about Cuirass to replace the Hydra section? Or does anyone have other 
> ideas about this page? You can send an email to webmasters@gnu (plural!) 
> or just reply to me.
>
>
> [1] https://www.gnu.org/software/devel.html

AFAIK, Hydra is now down.  IIRC, Mark was in charge.  Mark?

The webpage [1] starts with:

This page describes many of the development resources available
for GNU developers on GNU Project machines.

Jing Luo, could you provide more details on the purpose of this
webpage?  What is the intent of this webpage?

The Guix project relieson two Continuous Integration systems deployed on
two infrastructures, visible at ci.guix.gnu.org and
bordeaux.guix.gnu.org.  One is indeed Cuirass [2].  The other one is
Build Coordinator [3].

AFAIK, GNU Guile is continuously built on ci.guix but there is no other
GNU projects.  And I do not know if the Guix projects has the capacity
or the resource to host more GNU projects on their infrastructure.

Cheers,
simon








2: https://guix.gnu.org/en/cuirass
3: https://git.savannah.gnu.org/cgit/guix/build-coordinator.git

Re: Building container images with nix2container

[Simon Tournier]

Hi lewo,

On lun., 26 févr. 2024 at 11:09, Antoine Eiche  wrote:

> Does your built images contains several layers?

This had recently been introduced.

0cf75c9b2f23869201144917cea7f6ad49683d3d
AuthorDate: Tue Dec 26 03:54:12 2023 +0300
CommitDate: Mon Jan 8 21:04:44 2024 +0300

> nix2container uses an heuristic to group store paths into layers. The
> goal is to share common layers between images and to avoid full image
> rebuild when only a storepath differs.

Well, I have not followed on which strategy Guix relies.  What is the
one of nix2container?  The one described here:

https://grahamc.com/blog/nix-and-layered-docker-images/

> Do you write the image tarball into your store when you build an image?
>
> nix2container is able to build layers on the fly from the Nix store. The
> goal is to reduce IOs and storage. Instead of writing an image tarball
> into the store, it generates a script which stream layers from store
> paths to the destination (a Docker registry, the Docker deamon, Podman
> or a file).

To my knowledge, this is not implemented in Guix.  And indeed, it could
improve the dance.  Currently, it reads:

docker load < $(guix pack -f docker …)


Cheers,
simon

Re: Mechanism for helping in multi-channels configuration

[Simon Tournier]

Hi Attila,

On mar., 06 févr. 2024 at 17:16, Attila Lendvai  wrote:
>> The wishlist is: provide a machine-readable description on guix-science
>> channel side in order to help in finding the good overlap between
>> commits of different channels.
>
> i wrote about a missing abstraction here:
>
> https://lists.gnu.org/archive/html/guix-devel/2023-12/msg00104.html

You wrote in [1]:

it's probably the same thing that causes the discrepancy between
git commits and substitutes: the build servers are not building
every commit of the git repo. they pick an unpredictable (?)
series of commits, skipping some in between.  if i guix pull, or
guix time-machine to the "wrong" commit, then i'll need to build
some stuff locally. sometimes these can be heavy packages.

To my knowledge:

 + ci.guix (Cuirass) fetches every 5 minutes (IIRC) and builds the last
   commit.

 + bordeaux.guix (Build Coordinator) fetches the batch from the mailing
   list guix-commits:
   

About CI, yes it is unpredictable.  About Bordeaux, it is not really. :-)

1: Re: Should commits rather be buildable or small
Attila Lendvai 
Sun, 10 Dec 2023 23:20:25 +
id:SXjFmdTgxwHYE-Z6t7SZOykuXMBiD454EF2uad96jGQemgJ6hXki_f1C7VxVHKHa4b7_j5UwJmffh_FiQqEz_bIYIBn9tpG4s9F7W1eIDAQ=@lendvai.name
https://lists.gnu.org/archive/html/guix-devel/2023-12
https://yhetil.org/guix/SXjFmdTgxwHYE-Z6t7SZOykuXMBiD454EF2uad96jGQemgJ6hXki_f1C7VxVHKHa4b7_j5UwJmffh_FiQqEz_bIYIBn9tpG4s9F7W1eIDAQ=@lendvai.name

> the git commit log is a too fine-grained granularity here. there
> should be something like a 'guix log' above the git log that could be
> used, among other things, to encode inter-channel dependencies.

Considering the current status and how substitutes are GC, the first
step would be the retention of some substitutes.  And thus the
specification for a policy of such retention.  It would allow to build a
database that could be queried by this hypothetical “guix log” – which
should be more something under “guix weather” IMHO.

For the interested readers, thread about retention:

Building and caching old Guix derivations for a faster time machine
Ricardo Wurmus 
Fri, 10 Nov 2023 10:29:28 +0100
id:87o7g29c94@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2023-11
https://yhetil.org/guix/87o7g29c94@elephly.net

Substitute retention
Ludovic Courtès 
Tue, 12 Oct 2021 18:04:25 +0200
id:87y26ytek6.fsf...@inria.fr
https://lists.gnu.org/archive/html/guix-devel/2021-10
https://yhetil.org/guix/87y26ytek6.fsf...@inria.fr

Although I concur with this need, I do not see how it would be help for
detecting compatibility between channels. :-)

Cheers,
simon

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi Clément,

If read correctly, you answered about Gnus (debbugs.el):

>>> --8<---cut here---start->8---
>>> (with-current-buffer gnus-original-article-buffer
>>>   (message-fetch-field "Message-ID"))
>>> --8<---cut here---end--->8---

[...]

>>> May I add too, that you can add "Message-ID" in gnus-visible-headers.

[...]

> You add '%M' in gnus-summary-line-format.

[...]

> Yes it does provide a built-in access, as I showed you.  Just search for
> "Message-id" in `C-h v gnus-summary-line-format`.

And my message was:

  What appears to me “difficult” is that most of the
 tools as Email client are poorly supporting Message-ID.

Somehow, the reader will judge if Message-ID is smoothly supported. :-)

Cheers,
simon

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi Felix,

On jeu., 15 févr. 2024 at 07:32, Felix Lechner via "Development of GNU Guix and 
the GNU System distribution."  wrote:

> To request a feature in Debbugs.el, please file a bug against the
> "debbugs.gnu.org" package on debbugs.gnu.org.

To be clear, my message [1] was not to report a “bug” or request for a
“feature” in debbugs.el.  My message was:

  What appears to me “difficult” is that most of the
tools as Email client are poorly supporting Message-ID.

For instance,

And I took as one example the venerable debbugs.el for making my point:
most of the tools that deal with Emails poorly support one key of Email
heart: Message-ID.

Personally, I rely on the cool piem.el and some custom Emacs lisp
helpers, then for dealing with complex patch or bug thread, I inject and
process them with notmuch.el. :-)

Therefore, open a feature request is low on my list of TODO. ;-)

Cheers,
simon

1: Re: Guix Days: Patch flow discussion
Simon Tournier 
Wed, 14 Feb 2024 16:48:07 +0100
id:87mss3kpxk@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2024-02
https://yhetil.org/guix/87mss3kpxk@gmail.com

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

On jeu., 15 févr. 2024 at 10:10, Sharlatan Hellseher  
wrote:

> I would push go-team branch to check some lower level modifications to
> go-build-system which are queued right now. I need someone with admin access 
> to
> set the branch on CI as well ;-)

Cool!  Thank you.

Cheers,
simon

Re: Mechanism for helping in multi-channels configuration

[Simon Tournier]

Hi Christina,

On sam., 03 févr. 2024 at 15:27, Christina O'Donnell  wrote:

>   1. Have a script that scrapes all the define-public symbols in every 
> file in
>      every package.

I think you mean ’fold-packages’.

>   2. Have a script that determines the symbols needed by each file. (Macros
>      make this more difficult, but.)

Well, this would be difficult, IMHO.  Somehow, it is what the compiler
does. :-)

>   3. Have both scripts have an incremental version that runs on diffs (for
>      performance).
>   4. Run this for every commit on every branch on every channel caching the
>      result.
>   5. Have a CI script keep this updated for new commits.
>   6. Have a server track incompatibilities.

Here, I think the issue is that one server needs to track all the
channels.  And that’s a too strong assumption, IMHO.

I think the design should be something on channel maintainer side.
Somehow, the main Guix channel could be seen as a Git submodule from the
channel side and the issue is that information is not tracked.

There is this ’.guix-channel’ file which allows to describe channel
dependencies.  And the improvements could be to add more there.  The
question is what to add and how to add it.  Keeping in mind the
simplicity and the maintenance burden-free. :-)


> Full disclosure: I've got nothing lined up for the summer yet, so I'm on the
> prowl for GSoC projects :)

Cool!

In that spirit, one tool that is missing is: search packages in all the
history. Somehow the need is described by this message [1]: how to find
which Guix revision provides which version of Foo?

In addition, “guix search” is slow [2].

Well, I have started the embryo of an extension based on Guile-Xapian
for indexing and improving the search.  Really an embryo. :-)

I think this would fit some GSoC. ;-)


Cheers,
simon

1: Re: List available versions of package.
Philippe Veber 
Tue, 11 Jun 2019 09:43:08 +0200
id:CAOOOohSzUezKvm=ro0bxrgh3m0eo2x0cotvd--varxwoqtc...@mail.gmail.com
https://lists.gnu.org/archive/html/help-guix/2019-06
https://yhetil.org/guix/CAOOOohSzUezKvm=ro0bxrgh3m0eo2x0cotvd--varxwoqtc...@mail.gmail.com

2: https://issues.guix.gnu.org/issue/39258

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi Josselin,

On jeu., 15 févr. 2024 at 12:07, Josselin Poiret  wrote:

> I think b4's ML is more active than the GitHub issues, I have already
> sent some bug reports and patches there that were picked up quite fast.

Yeah, Kyle pointed me that out months ago.  Then I have never taken the
time to report there. :-)

Cheers,
simon

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi Clément,

On jeu., 15 févr. 2024 at 12:45, Clément Lassieur  wrote:

>>> 'b4 shazam' is probably the most trouble-free way to apply patches;
>>
>> I agree*!
>
> I don't agree (both Gwene + Gnus or Emacs Debbugs work perfectly too and
> allow to apply a range of n patches at once) but I don't think there is
> a need for competition here, it's good that we have several tools.

Yes for sure it is good to have several tools.  And the ones you like. :-)

No one is advocating to make ’b4 shazam’ THE only one tool.

Instead, I agree with Maxim that exposing Message-ID and relying on ’b4
shazam’ is the most trouble-free and config-less way to apply patches.


>>  What appears to me “difficult” is that most of the
>> tools as Email client are poorly supporting Message-ID.
>>
>> For instance, debbugs.el (Gnus).  To my knowledge, there is not easy way
>> to get the Message-ID when reading an article (bug/patch) from Debbugs.
>> There is other means for applying patches.  But still each time appears
>> to me weird. :-)
>
> It's
>
> --8<---cut here---start->8---
> (with-current-buffer gnus-original-article-buffer
>   (message-fetch-field "Message-ID"))
> --8<---cut here---end--->8---

[...]

> May I add too, that you can add "Message-ID" in gnus-visible-headers.

And what about Summary buffer?

Well, it makes my point, no? :-)

For sure the Message-ID is there and for sure it is possible to extract
it.  However, it appears to me weird that it is not built-in.  I mean
Message-ID is one of the heart of Emails, and Debbugs is just Emails,
but debbugs.el does not provide a built-in access to it.


Cheers,
simon

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi,

On dim., 11 févr. 2024 at 11:38, Maxim Cournoyer  
wrote:

> 'b4 shazam' is probably the most trouble-free way to apply patches;

I agree*!

> it
> even selects the latest revision it finds in the issue thread.  To make
> finding a message-id easier, I've also recently added a 'Copy
> Message-ID' button to the Mumi interface; try it visiting any issue,
> e.g. .  The message-id of any message
> can be easily copied to your clipboard via the new button.

I also agree! :-) What appears to me “difficult” is that most of the
tools as Email client are poorly supporting Message-ID.

For instance, debbugs.el (Gnus).  To my knowledge, there is not easy way
to get the Message-ID when reading an article (bug/patch) from Debbugs.
There is other means for applying patches.  But still each time appears
to me weird. :-)

So thanks for this Mumi addition!

Cheers,
simon

PS: *agree on B4 although there is tricky bugs as reported here:
.

Re: Guix Days: Patch flow discussion

[Simon Tournier]

Hi Steve,

  ( On a side note, the triage of old bugs is a similar problem.  They
are easy to find [2], read, check and send an email to
12...@debbugs.gnu.org does not appear to me an issue with any tool.

For what it is worth and without any willing of being harsh, I am
able to count the people doing this boring task.

What is hard to solve is the incentives for doing the boring, but
necessary, collective tasks.

Bah the usual problem of lengthy discussions with roommates in any
shared apartment: who clean the bathroom? :-) )


On lun., 05 févr. 2024 at 09:39, Steve George  wrote:

> Our goal for the discussion:
>
>   How do we double the number of patches that are *reviewed* and
>   *applied* to Guix in the next six months?

Thanks for these notes and leading the session.  On my side, it was a
fruitful discussion.

Well, let me try to quickly summarize my conclusion of the session:

 1. We have a social/organisational problem.

 2. We have some tooling annoyances.



The easy first: #2 about tools.  The email workflow is often cited as
part of the issue.  That’s a false-problem, IMHO.

Projects that use PR/MR workflow have the same problem.  For instance,
Julia [1] has 896 open PR.  On my browser, it means 36 pages so if I go
to – 25 PRs per page – the still open submitted PRs:

   + the 6th page:  around Sept.2023 and Oct. 2023
   + the 12th page: around Apr. 2023 and Mar. 2023
   + the 18th page: around Jul. 2022 and Mar. 2022
   + the 24th page: around Jun. 2021 and May  2021
   + the 30th page: around Mar. 2020 and Oct. 2019
   + the 36th page: around Mar. 2017 and May. 2014

Obviously, an example is not a proof or an evidence.  It is just a
first clue. :-)

I will not speak about the channel ’nonguix’ but it gives another
clue.

That said, for sure, the tools need more love.  Thanks all the people
for all hard work over the years in this area – no name, you know, I
fear to forget someone. ;-)

So, yeah we need to smooth the technical burden for reviewing in order
to focus on the review itself.

To be clear, the email workflow might add burden on submitter side but I
am doubtful it is really part of the bottleneck for reviewing and
pushing submissions.


Although the tools might add some unnecessary friction, the net of the
issue is IMHO #1: reviewing is just boring and time-consuming.

Who feel accountable?  And for what?  That’s the question, IMHO.

If the number of submission is doubled, how do we increase the number of
people that feel enough accountable for doing the boring work?

  ( Maybe accountable is not the correct word.  Obligation neither.
Well the kind of feeling that is okish if you skip the task but you
know it will be better if you do it. )


Well, the difficult part is not pressing some buttons for merging and
pushing – whatever the tools or workflow.  The difficult part is to
scrutinize the submission.

I think the bottleneck is not the number of people able to push.
Instead, I think the bottleneck is the number of people confident with
the change for then pushing it.

The question is thus: how to build this confidence?


Look, when a committer has some free-time, most of the time, what is the
process: take first the “easy“ submissions for committing them – from
trivial updates to simple updates.  If free-time remains, then engage
with more “complex” submissions… ah no more free-time. :-)

Why starting by the “easy” submission?  Because it is less boring and
time-consuming; somehow it is easier to feel confident with that sort of
change for pushing it.

As a rule of thumb, about the time it takes – on average –, the order of
magnitude for reviewing is similar as the one for submitting.  Well,
from my experience and although I never did stats. :-)


All in all, I see two paths to move forward:

i) Non-committers can help.  On two fronts:

   + Answer to submitter with the changes for being compliant with Guix
 standards.
   
   + Follow-up on patches already commented but without an updated
 revision: upgrade the re-roll count by sending this revision.

 It eases for merging if I do not have to make many tiny edits
 myself.

ii) Create more teams or at least more people should commit to be part
of a team and help in reviewing what they know.

For instance, since Sept. (167 days ago) I have been CC in 108
patches submissions.  Most of them are from ’core’ team that I would
qualify as “complex”. :-)

Many patches assigned to ’core’ team are sent by committers.  The
issue is not being a committer or not.  Instead, being more eyes
commenting would increase the confidence.  Thus it would reduce the
workload.

That’s the same for any team, IMHO.

And I do not speak about patches that are not assigned to any team.


Somehow, we need to think how people would feel “accountable” for doing
the collective tasks with low, no direct or personal reward.

As with many non-technical topic

Re: Git-LFS or Git Annex?

[Simon Tournier]

Hi Timothy,

On sam., 27 janv. 2024 at 10:59, Timothy Sample  wrote:

> https://git.ngyro.com/git-annex-remote-clouda/tree/git-annex-remote-clouda/remote.scm

Oh cool, thanks.  Bookmarked.

Cheers,
simon

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

Late to the party. :-)  Processing my backlog…

On jeu., 18 janv. 2024 at 10:25, Sharlatan Hellseher  
wrote:

> I'm currently in review and split some packages from (gnu packages golang) 
> into
> (gnu packages golang-crypto) to simplify the maintenance. I try to play with
> that option and see which packages are missed to satisfy passing all tests.

What is the status of this?  Is all fine?

Cheers,
simon

Re: Committers available for Patch hacking/review meet-up?

[Simon Tournier]

Hi,

On mar., 13 févr. 2024 at 14:48, Steve George  wrote:

> At Guix Days we said we'd organise some patch review sessions.

Cool!


> Anyone available? If you are and can put your name down for a particular 
> date that would be brilliant!

I will do.  Thanks for the initiative!


> Q2: Does anyone have permission on 
> https://libreplanet.org/wiki/Group:Guix to give a user the right to 
> create new pages? I want to document the sessions and how-to's.

Hum, I do not know who did it and how… there is this webpage:

https://libreplanet.org/wiki/Group:Guix/PatchReviewSessions2024


> https://www.timeanddate.com/worldclock/meetingtime.html?iso=20240213&p1=5416&p2=136&p3=179

Is Annecy time the same as Paris time? ;-)


Cheers,
simon

Google Season of Docs 2024

[Simon Tournier]

Hi,

( It is when my plate is full that I add more in. :-) )

Google is announcing the Season of Docs.  Somehow, it is similar of
Google Summer of Code but for… wait for it… documentation!

https://opensource.googleblog.com/2024/02/announcing-google-season-of-docs-2024.html

I would like that the Guix project applies as an Organization.  Hence my
message.  Before jumping in the boring paperwork, the first questions
are:

 1. Who does volunteer for being potential mentor?
 2. Would one of you readers be interested by being technical writer?
 3. Any for improving the documentation?

Well, shoot any ideas for #3.  It will help in all cases. :-)

It can range from a one line idea to more elaborated idea; as shown here
[1, 2, 3].

Depending on the feedback, I will create a LibrePlanet webpage or else
and go further in the process. :-)

Last, keep in mind the deadline is less than 10 days from now.

1: https://developers.google.com/season-of-docs/docs/project-ideas
2: https://developers.google.com/season-of-docs/docs/case-study-example
3: https://developers.google.com/season-of-docs/docs/2022/participants

Cheers,
simon

Re: bug#63775: git describe on current master says: v1.3.0-38775-g6192acf8b7

[Simon Tournier]

Hi,

On sam., 03 févr. 2024 at 19:43, Giovanni Biscuolo  wrote:

> This is a git bug, not an issue with our repo, and for this reason (I
> hope) I'm closing this bug; please see below.

Here the explanation of the bug of “git describe”:

https://lore.kernel.org/git/20191008123156.gg11...@szeder.dev/

  $ git describe d1a251a1fa
  v2.23.0-135-gd1a251a1fa
  $ git log --oneline v2.23.0..d1a251a1fa | wc -l
  59

Uh-oh, 59 != 135.

This is happening because:

  - Git is too fast ;) and the committer date has a one second
granularity, so scripts can easily create subsequent commits with
the same committer date.  Case in point are the two subsequent
merge commits f3c19f85c5 and 4a3ed2bec6 at the bottom of this
simplified history snippet (kind of a hand-edited 'git log --graph
--format="%h %cd %s"'):

*   d1a251a1fa 2019-09-09 12:26:36 -0700 Merge branch 
'en/checkout-mismerge-fix'
|\
* | ... a big chunk of history simplified away ...
| * acb7da05ac 2019-08-16 09:58:00 -0700 checkout: remove duplicate code
* | a5e4be2f68 2019-04-25 16:41:15 +0900 Merge branch 
'ab/commit-graph-fixes'
* | f3c19f85c5 2019-04-25 16:41:14 +0900 Merge branch 'ab/gc-reflog'
|/
*   4a3ed2bec6 2019-04-25 16:41:14 +0900 Merge branch 'nd/checkout-m'

  - 'git describe' implements its own history traversal: it iterates
over all parents of a commit, adds any yet unseen parents to a
commit list ordered by date, and then continues with the first,
i.e. most recent commit from that list.  While doing so it uses
several bits in 'commit->object.flags' to track reachability
information from several candidate tags at once, and copies these
flags from each commit to its parents; this is important to
compute the correct number of additional commits.  Another
important thing is the implementation detail that
commit_list_insert_by_date() inserts a new commit after all other
commits with the same date that are already in the list.


Thanks Giovanni for pointing this out.

Cheers,
simon

Re: ice-9 match penalty depending on pattern?

[Simon Tournier]

Hi,

On mer., 07 févr. 2024 at 10:41, Carlo Zancanaro  wrote:

>> Why not?  Do I miss something in the implementation of ’match’?
>
> The only reason I can think of would be if these matches are sometimes
> provided improper lists, which need to fail these match conditions. That
> seems unlikely to me, but it should be clear from looking at the other
> match clauses in each case.

Well, I have not pruned the list returned by just grepping. :-)  And I
have just grepped with the term ’head’, ’tail’ and ’\.\.\.’

Somehow, my question is twofold:

1. Is the “expensive” check worth for such case:

  (match paths
((head tail ...)
 (if (visited? head)
 (loop tail visited result)
 (call-with-values
 (lambda ()
   (loop (references store head)
 (visit head)
 result))
   (lambda (visited result)
 (loop tail
   visited
   (cons head result))
(()
 (values visited result)

seen in ’topologically-sorted’ procedure from (guix store) module.

2. Is the “expensive” check worth for such multi-cases:

  (match sexp
((? string? str)
 (let ((prefix "swh:1:dir:"))
   (if (string-prefix? prefix str)
   (cons (string-drop str (string-length prefix)) ids)
   ids)))
((head tail ...)
 (loop tail (loop head ids)))
(_ ids))

seen in ’lookup-disarchive-spec’ from (guix lint).

Well, I am not saying to rely on ’car’ and ’cdr’.  Instead, I am asking
what is the idiomatic Guile pattern matching for Guile?

My main concern is about chasing the unnecessary checks for making Guix
a bit faster. :-)


Cheers,
simon

ice-9 match penalty depending on pattern?

[Simon Tournier]

Hi,

>From Ludo’s mastodon message [1]:

Re ‘match’ penalty: when using ellipses in patterns, the generated
code checks for “proper lists”, which is O(n).  The trick is to
instead match a pair:

✔ (match lst ((head . tail) …))
❎ (match lst ((head tail ...) …))

Therefore I have confused by some patterns.

--8<---cut here---start->8---
28 candidates:
./gnu/services/monitoring.scm:249:((head tail ...)
./gnu/system/file-systems.scm:249:  ((head1 tail1 ...)
./gnu/system/file-systems.scm:251: ((head2 tail2 ...)
./gnu/packages.scm:116:  ((_ file head tail ...)
./gnu/build/activation.scm:85:  ((head tail ...)
./gnu/build/linux-boot.scm:225:  ((head tail ...)
./guix/http-client.scm:255:  ((head tail ...)
./guix/records.scm:604:  ((_ record field offset ((head normal) tail ...))
./guix/records.scm:607:  ((_ record field offset ((head delayed) tail ...))
./guix/records.scm:610:  ((_ record field offset ((head thunked) tail ...))
./guix/read-print.scm:482:  ((head tail ...)
./guix/read-print.scm:750:  ((head tail ...)
./guix/lint.scm:1614:((head tail ...)
./guix/scripts/package.scm:809: ((head tail ...) head
./guix/self.scm:160:  ((head tail ...)
./guix/store.scm:1556:((head tail ...)
./guix/store.scm:1764:  ((head tail ...)
./guix/ui.scm:295:  ((head tail ...)
./guix/ui.scm:2258:   ((head tail ...) head)))
./guix/docker.scm:261:  (((head ...) (tail ...) id)
./guix/build/store-copy.scm:72:((head tail ...)
./guix/build/gremlin.scm:336:  ((head tail ...)
./guix/build/graft.scm:330:  ((head tail ...)
./guix/build/utils.scm:236:  ((head tail ...)
./guix/build/utils.scm:405:  ((head tail ...)
./guix/utils.scm:910:  ((head1 tail1 ...)
./guix/utils.scm:913: ((head2 tail2 ...)
./build-aux/compile-all.scm:71:   ((head tail ...)
--8<---cut here---end--->8---

Maybe for some of them, it changes nothing for the user-visible
performances.  Maybe it does. :-)

Well, I do not know the length of each match.  However, some are part of
some loop.  Therefore, if we could save some cycles by simply replacing
the ellipsis, as

((head tail ...) stuff that use head or tail)

by

((head . tail) stuff that use head or tail)

Why not?  Do I miss something in the implementation of ’match’?

Cheers,
simon

1:
https://social.sciences.re/@civo...@toot.aquilenet.fr/111885442823194970

Re: [fr] Moment de convivialité Guix@Paris en février

[Simon Tournier]

salut,

On jeu., 01 févr. 2024 at 20:51, Tanguy LE CARROUR  wrote:

> Ce jeudi 8 février 2024 à 19h, se tiendra la cinquième édition de Guix@Paris
> ouverte au public.
> Comme les fois précédentes, il sera possible de participer à distance
> (*cf* ci-dessous).

Chouette !  Ayant raté la journée de jeudi des Guix Days, je suis très
intéressé par un petit résumé. :-D  Peut-être pas le seul
d’ailleurs. ;-)

à très tantôt,
simon

[post Guix Days] Guix Common Document (was: Request-For-Comment process)

[Simon Tournier]

Hi all,

I hope that the discussion we had yesterday (Friday 2nd) in Guix Days
has clarified the idea behind this proposal.

I am waiting Ludo’s notes in order to refine this proposal, integrate
many comments and/or ideas, and polish.

Thanks all participants.


The aim of the proposal is to have a process to document our processes
with the least bureaucracy as possible.  Well, Debian project is often
cited as an example (social contract, voting system, etc.).  Indeed,
however there is more bureaucracy in Debian than in French State. ;-)

Instead, let just formalize what we are already doing.

Currently, we are just adding more and more sections to the manual and
for other parts the structure for making decisions is not clear.  For
sure, it works… until now but I think it does not scale and we are
touching the limits about what can be done with this informal structure.

Let me clarify my attempt behind this “RFC proposal”.  First,
pukkamustard proposed the name “Guix Common Document” echoing “greatest
common divisor“ (gcd): the greatest common divisor of two or more
integers is the largest positive integer that divides each of the
integers – other said, that’s the larger integer in common with all.

I like it because it captures well the idea; although such different
name could be confusing from the outside.   Anyway.  That’s an
implementation detail. ;-)

Second, from my point of view, the core components of the proposal are:

 + consensus;
 + co-supporter.

Consensus, because it is how we already collaborate.  Somehow, it
changes almost nothing for our daily operations but having an explicit
formalization will help outsiders.  The definition of “consensus” is
twofold:

 1. can live with;
 2. concerns are actively resolved.

Other said, the definition wording of “consensus” specifies how to avoid
being blocked by disagreements: when one wish to block a proposal then
one bears a special responsibility for finding alternatives, proposing
ideas/code or explaining the rationale for the status quo.

And to make it clear, the first idea for making decision is “voting” but
then we need to define “who” votes.  Well, this appears to me a
counter-measure against something that would be rare and this solution
does not trust in the values of our community (being welcoming,
inclusive, taking care of each other, etc. well as least, trying as much
as possible :-)).

For me, the counter-measure against an hostile takeover is somehow
captured the point #2 above.


Co-supporter, because similarly as the manual section « (guix) Reviewing
the Work of Others » [1], the aim is to cross the final line, make
progress by incremental focused improvements.  Therefore, a proposal
needs the help of someone committed to the project (long-standing
contributor, committer, etc.).

I agree that “contributor sufficiently familiar” is maybe too vague and
needs more specific examples as “contributor sufficiently familiar
(committers or people with X commits)”.  Well, that’s part refining the
proposal. :-)


Last, I think that the time-frame for discussing needs to be bounded.
Somehow this bound will help in the incremental improvement and will
avoid the trap of the perfect-as-the-first-try.


Well, let recover from these awesome Guix Days and from FOSDEM and then
resume this proposal.

Cheers,
simon

1: https://guix.gnu.org/manual/devel/en/guix.html#Reviewing-the-Work-of-Others

Mechanism for helping in multi-channels configuration

[Simon Tournier]

Hi,

Well, using Guix bdab356 from a little bit more than one month old, then
associating the channel guix-science 0b3d4a2f last week, I get the
failure:

--8<---cut here---start->8---
$ guix build /gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv
The following derivation will be built:
  /gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv
building /gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv...
(repl-version 0 1 1)
WARNING: (guix-science build bazel-build-system): imported module (guix build 
utils) overrides core binding `delete'
(exception unbound-variable (value #f) (value "Unbound variable: ~S") (value 
(python-nr-stream)) (value #f))
builder for `/gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv' 
failed to produce output path 
`/gnu/store/qzgj4vig3vklbznz1i0pgy11nr3z4rv9-guix-science'
build of /gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv failed
View build log at 
'/var/log/guix/drvs/g3/aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv.gz'.
guix build: error: build of 
`/gnu/store/g3aa5rh7bs5pyxd3q1gvhwz1s9z1vh3z-guix-science.drv' failed
--8<---cut here---end--->8---

Well, that’s expected!  Guix bdab356 does not contain python-nr-stream
introduced by commit 7dfe41aa71a4a4a9d6065a44e9c6271717215b3e.

The wishlist is: provide a machine-readable description on guix-science
channel side in order to help in finding the good overlap between
commits of different channels.

It could be nice if instead of an hard error, “guix pull” could say:
« the channel ’guix’ needs to be at least at commit 1234abc ».

WDYT?

Cheers,
simon

Re: Request-For-Comment process: concrete implementation

[Simon Tournier]

Hi Ricardo,

On mer., 20 déc. 2023 at 12:49, Ricardo Wurmus  wrote:

> I just got back from travels and finally caught up with important email.
> I read the proposal and it looks good to me.  Thank you for working on
> this!
>
> This would be the first project I contribute to that has an RFC process,
> so I don’t know what to look out for.  My concerns may thus be
> completely out of touch with reality, so beware as you read on.

Thank you for the feedback.  Much appreciated! :-)

I will address your question in another broad message.

Cheers,
simon

Re: Git-LFS or Git Annex?

[Simon Tournier]

Hi Kyle,

On jeu., 25 janv. 2024 at 21:20, Kyle Meyer  wrote:

> Fwiw I think someone could do that outside Haskell, if they preferred,
> via a custom backend:
>
>   https://git-annex.branchable.com/design/external_backend_protocol/
>
> Special remotes can also be written in other languages:
>
>   https://git-annex.branchable.com/design/external_special_remote_protocol/

Thanks!  I did not know.  Indeed, it could be a nice GSoC to implement
some ‘git-annex-backend-nar’. :-)

Cheers,
simon

Re: Git-LFS or Git Annex?

[Simon Tournier]

Hi Ludo, all,

On mer., 24 janv. 2024 at 16:22, Ludovic Courtès  wrote:

> The question boils down to: Git-LFS or Git Annex?

Some months ago, I gave a look for managing some datasets.  My
conclusion is Git-Annex.  The main drawback of Git-LFS is that the
server needs to support the protocol.  On Git-Annex side, the main
drawback is Haskell.

Haskell could seem a detail but it is not when considering other
architectures than x86_64.  Give a look to CI filtering with ’ghc-’:

http://ci.guix.gnu.org/eval/1074397/dashboard?system=i686-linux

Here I pick i686 as an example for making the point of the Haskell
support of non-x86_64.  Aside, I do not speak about the resources that
Haskell requires for being compiled.

Do not take me wrong: it does not mean that’s a roadblock but let keep
that in mind: Git-Annex comes with limitations because of Haskell.

That’s said, Git-Annex seems adapted for the workflow you describe:
backup large files between various servers.  And it would be a bridge
between content and address.  However, the content still needs to be
stored on some servers, IMHO.  Git-Annex supports “special remotes” [1]
but it is not clear for me if the aim is to distribute the workload
between the two main servers or if the aim is just to ease the
maintenance of backups.

Last, you speak about content-addressed and this part is not clear for
me.  In Git-Annex, you have in one hand the Git content-addressed system
and in the other hand the “key-value backends“ [2].  Somehow, Git-Annex
stores the key in a file that is stored in Git itself and the value is
somehow stored outside Git itself.

Recently, support of Git-LFS had been added to git-download with
a4db19d8e07eeb26931edfde0f0e6bca4e0448d3.  In that context, with
content-addressed in mind, are you speaking to add Git-Annex support and
thus distribute the videos as substitutes; probably also easing the
maintenance of backups.  Or is the question unrelated?

On a side note, depending on the size of the videos, it is only possible
to use non-cryptograpgically backends as URL.

All that said, let fix the ideas: a simple example, sync content between
machine-A and machine-B where original content is also kept elsewhere.

Let create a Git repository with a file annexed.

--8<---cut here---start->8---
machine-A$ mkdir example && cd example
machine-A$ git init && git annex init

machine-A$ $ git annex addurl -b MD5 --file sources.json \
 https://guix.gnu.org/sources.json
addurl https://guix.gnu.org/sources.json 
(to sources.json) ok
(recording state in git...)

machine-A$ file sources.json
sources.json: symbolic link to 
.git/annex/objects/jx/1j/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a

machine-A$ git annex add .
machine-A$ git commit -am 'Add sources.json'
[master (root-commit) bdf6bca] Add sources.json
 1 file changed, 1 insertion(+)
 create mode 12 sources.json
--8<---cut here---end--->8---

Let’s backup.

--8<---cut here---start->8---
machine-B$ $ git clone file:///tmp/example backup && cd backup/

machine-B$ file sources.json 
sources.json: broken symbolic link to 
.git/annex/objects/jx/1j/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a
--8<---cut here---end--->8---

As you see, here nothing is really copied.  It is only a symbolic link
pointing to some content outside what Git trackes.

--8<---cut here---start->8---
machine-B$ guix hash -rx .
0x8kiaprmjq6f02pdq155wlznxhzi871mk0la6sp944q854pcpn5

machine-B$ git annex get sources.json
get sources.json (from origin...) 
ok
(recording state in git...)

machine-B$ guix hash -rx .
0x8kiaprmjq6f02pdq155wlznxhzi871mk0la6sp944q854pcpn5

machine-B$ file sources.json
sources.json: symbolic link to 
.git/annex/objects/jx/1j/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a
--8<---cut here---end--->8---

Let’s remove the file on machine-B; for whatever reason.

--8<---cut here---start->8---
machine-B$ git annex drop sources.json
drop sources.json ok
(recording state in git...)

machine-B$ file sources.json
sources.json: broken symbolic link to 
.git/annex/objects/jx/1j/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a
--8<---cut here---end--->8---

And assume that machine-A is now unreachable.   Let’s get again on
machine-B.

--8<---cut here---start->8---
machine-B$ git annex get sources.json
get sources.json (from web...) 
ok
(recording state in git...)

machine-B$ file sources.json
sources.json: symbolic link to 
.git/annex/objects/jx/1j/MD5-s2697572--a2b17d21f5a209b1763ad537a97d9c5a/MD5-s

Re: Collecting Guix talks at FOSDEM

[Simon Tournier]

Hi Steve,

On mer., 17 janv. 2024 at 21:16, Steve George  wrote:

> Hi - initial draft of the post went to Ludo today for review (and commit 
> as I don't have the rights). We'll see what his reaction is to my 
> British English and random punctuation! ;-)

Cool!  Thank you.

For the next time, in order to avoid to put too much on Ludo’s plate, we
have a dedicated email guix-b...@gnu.org.  The emails there are private
and 2-3 people are behind the list.

For example, the recent post [1] had been sent there then a round of
“review“ (mainly typo and links), and published. :-)

1: https://guix.gnu.org/en/blog/2024/building-packages-targeting-psabis/

Cheers,
simon

Re: Collecting Guix talks at FOSDEM

[Simon Tournier]

Hi,

On Tue, 16 Jan 2024 at 15:35, Steve George  wrote:

> We're planning to put up a blog post about Guix (and Guix-related) talks 
> at FOSDEM [0]. I've collected all the talks that that are about Guix (or 
> connected areas). If I've missed any Guix related talks please tell me 
> so I can add them to the blog post:

Thanks for taking care about that.

It could be nice to publish this bog soon.  Is it doable?

Cheers,
simon

Re: Using the pyproject-build-system

[Simon Tournier]

Hi,

On Tue, 16 Jan 2024 at 00:19, Troy Figiel  wrote:

> Although not fully PEP 517-compliant according the documentation, the
> pyproject-build-system does seem to fall back to setuptools.build_meta
> if the pyproject.toml is missing. Contrary to what the name implies to
> me, it can therefore also be used for packages with only a setup.py file.

My understanding is that PEP 517 will be the default buitin way for
building Python project, and pyproject-build-system is an implementation
of such PEP 517.  Somehow, the adoption of PEP 517 by Python projects
will imply more Guix packages using pyproject-build-system.

In addition, one (and tangential) longer term for building Python
packages using Guix is rely on one build-system, namely
pyproject-build-system.  Hence maybe this unclear situation.

Cheers,
simon

Re: An update on ‘core-updates’

[Simon Tournier]

Hi,

On Thu, 11 Jan 2024 at 16:10, Ludovic Courtès  wrote:

> Long story short: I’d like us to freeze and merge the branch ASAP,
> notably because the glibc graft on ‘master’ leads to a bad user
> experience.  I’m happy with the current state of the branch and wouldn’t
> mind postponing remaining upgrades for the next cycle.
>
> Thoughts?

LGTM.  Hoping ASAP will be sooner than later… busy month! :-)

Cheers,
simon

Re: Proposition to streamline our NAR collection to just zstd-compressed ones

[Simon Tournier]

Hi,

On Tue, 09 Jan 2024 at 21:32, Maxim Cournoyer  wrote:

> What do you think?  Should we go ahead and effect the following simple
> change for the Berlin build farm?
>
> --8<---cut here---start->8---
> modified   hydra/modules/sysadmin/services.scm
> @@ -683,7 +683,7 @@ to a selected directory.")
> ;; 
> 
> ;; for the compression ratio/decompression speed
> ;; tradeoffs.
> -   (compression '(("lzip" 9) ("zstd" 19)))
> +   (compression '(("zstd" 19)))
> (cache-bypass-threshold cache-bypass-threshold)
> (workers publish-workers)))
> --8<---cut here---end--->8---

I think it is a good idea but the change is more than just oneline. ;-)

I agree with Ludo: the change requires communication.  Something like:

 1. Blog post.  Something like that [1], a bit extended with a Migration
section.

 2. A news (guix pull --news) announcing the sunset date.  And probably
pointing to the blog post (or elsewhere) for helping the migration.

 3. Optionally emit a warning when the daemon is “too” old.


I agree that the extra space can be annoying.  In the same time, user
experience matters more, IMHO.

Cheers,
simon

1: https://guix.gnu.org/en/blog/2022/sunsetting-gzip-substitutes-availability/

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

CC:
$ ./etc/teams.scm list-members go
Katherine Cox-Buday 
Sharlatan Hellseher 


On Sun, 14 Jan 2024 at 22:12, Troy Figiel  wrote:

> --8<---cut here---start->8---
> (define* (check #:key tests? import-path #:allow-other-keys)
>   "Run the tests for the package named by IMPORT-PATH."
>   (when tests?
> (invoke "go" "test" (string-append import-path "/...")))
>   #t)
> --8<---cut here---end--->8---
>
> For example, if the -v flag is added (which might be a better default?)
> to the check phase of go-github-stretchr-testify, it can be seen that
> only `TestImports' runs, none of the tests in assert, http, etc.
> However, the source code in these subdirectories is still recursively
> copied to out during the install phase.
>
> Is this desired behaviour? I assumed it isn't, because it looks like we
> are skipping a lot of tests during the check phase. However, I might
> also simply be overlooking something here as I am new to packaging
> Golang with Guix.

>From your description, it seems a good idea.  What do Go “experts“
think?

Cheers,
simon

Re: Commit Access: Sharlatan Hellseher

[Simon Tournier]

Hi Oleg,

On Thu, 11 Jan 2024 at 19:56, Sharlatan Hellseher  wrote:

> I am happy to have been granted commit access

Cool!  Welcome.

> If anyone has a good patch review workflow using Emacs, Gnus, and Magit,
> I would appreciate it ;-)

Well, nothing more than what had been already suggested.  Well, somehow
pipe the messages with “git am -3s” and apply the patches to a Git
worktree (named cooking), then rebase this cooking on the top of
master.

Cheers,
simon

Re: $EDITOR and “guix edit”

[Simon Tournier]

On Fri, 12 Jan 2024 at 20:46, Liliana Marie Prikler
 wrote:

> PS: I should probably just write the patch myself at this point, but I
> feel like it'll be misunderstood either way.

Sorry but I do not understand how your proposal would work in tandem
with the current "guix edit".  So yes, please send a patch for helping
me (us?) to understand what you suggest.

Cheers,
simon

RE: [swh-devel] Call for public review - SWH Nix/GNU Guix stack

[Simon TOURNIER]

Hi,

> The initial NixGuix loader (currently in production) lists and loads
> origins from a manifest, ignoring the specific origins mentioned above. The
> new stack will be able to ingest those origins. It will also optionally
> associate, if present, a NAR hash (specific intrinsic identifier to Nix and
> Guix) to what’s called an ExtID (SWH side).

Cool!  Thank you.

> Regarding the SWH API reading side of the ExtID though is a work to be done.

In short, currently Guix relies on SWH API for resolving from
“something” to SWHID, where “something” can be:

 + Git label tag + url
 + Git commit hash
 + plain url

Well, the situation is in good shape IMHO – I do not have recent
numbers, say all is fine for 75% of all Guix packages and for 90% of
Guix packages coming from some Git repositories – but still, we have
examples where “Git label tag + url” fails.  For one instance, see [1]
pointed by [2].

The information – history of history – is there in SWH but it would
require on Guix side to parse the snapshot information and extract as
best as possible; trying several SWH snapshots until a match.  Something
like that.  Chance of success until completion?  Weak. :-)

Moreover, what about the missing 25%?  They are Guix packages coming
from Mercurial repositories or from Subversion repositories or some
others.

Back on October 2020, we had discussion [3] for sending a save request
for packages using SVN checkouts but at the time we did not have a clear
path for retrieving.  Then on March 2023, maybe an path for retrieving
with this discussion [4]… but still many hacks are required [5].

Again, the information is there in SWH but it would require on Guix side
to parse the snapshot information and extract as best as possible;
trying several SWH snapshots until a match.  Something like that.
Chance of success until completion?  Weak. :-)

If only one source is missing, all the castle potentially falls down.  Somehow,
a dictionary from ExtID as nar hash to SWHID would help to have the
castle more robust. :-)

The SWH archive coverage of Guix packages would not be 75% because we, on
Guix side, are not able to know or retrieve these missing 25%.  Such dictionary
could reinforce the bridge between reproducible computational environment 
and archiving, IMHO.

So yeah, we are looking forward to some ExtID interface.  :-)

Cheers,
simon


1: https://issues.guix.gnu.org/66015#0-lineno53
2: 
https://gitlab.softwareheritage.org/swh/devel/swh-loader-git/-/issues/4751#note_148587
3: https://issues.guix.gnu.org/43442#9
4: https://sympa.inria.fr/sympa/arc/swh-devel/2023-03/msg9.html
5: https://issues.guix.gnu.org/43442#13

Re: $EDITOR and “guix edit”

[Simon Tournier]

Hi,

On Fri, 12 Jan 2024 at 18:39, Liliana Marie Prikler
 wrote:

> > Well, I see how to write specific Scheme wrapper around $EDITOR; as I
> > did in [1].
> >
> > Or, I see how to tweak guix/scripts/edit.scm for running specific
> > launcher depending on $EDITOR.
> >
> > Liliana, could you provide a proof-of-concept about « the shell-esque
> > "${LINE}" and "${FILE}" that would get replaced by Scheme code
> > looking for those strings »?  Because I do not see what you mean.
>
> (let* ((editor (getenv "GUIX_EDITOR"))
>(editor (string-replace-substring editor "${FILE}" the-file))
>(editor (string-replace-substring editor "${LINE}" the-line)))
>   editor)
>
> with the-file and the-line being placeholders for the actual variable
> names.  You could obviously do smarter things with gash, but let's not
> go there at the moment.

I do not understand how it is different from the wrapper I already did:

https://gitlab.com/zimoun/advanced-packages-2023/-/blob/main/vscode-wrapper?ref_type=heads#L70-99

Cheers,
simon

Re: $EDITOR and “guix edit”

[Simon Tournier]

Hi,

On Mon, 20 Nov 2023 at 20:33, Liliana Marie Prikler  
wrote:

>>  2. Do we put this code in some etc/vscode-wrapper that user can
>>  install?  (or that we could automatically installl) Or maybe revamp
>> it
>>  for calling this code via some shell function?
>
> With VSCode et al. not being Guix packages, I see little point in
> providing these wrappers through Guix itself.

I do not want to address here where to keep VSCode support and instead I
would like to address $EDITOR which does not follow the good ol’
fashion.

Well, I see how to write specific Scheme wrapper around $EDITOR; as I
did in [1].

Or, I see how to tweak guix/scripts/edit.scm for running specific
launcher depending on $EDITOR.

Liliana, could you provide a proof-of-concept about « the shell-esque
"${LINE}" and "${FILE}" that would get replaced by Scheme code looking
for those strings »?  Because I do not see what you mean.

Cheers,
simon

1: 
https://gitlab.com/zimoun/advanced-packages-2023/-/blob/main/vscode-wrapper?ref_type=heads

PS:

About VSCode.  Somehow, it is a chicken-or-the-eggs problem.  “We“
cannot complain with lengthy threads about the lack of contributor
diversity, or that many Guix tools are Emacs-centric, etc. and in the
same time say no because VSCode is not packaged in Guix proper.

We like it or not – I do not like it and do not use it! – for sure,
VSCode is currently one of the most used editor around.  Being friendly
with VSCode users would help to have more contributions from them.

Re: Building and caching old Guix derivations for a faster time machine

[Simon Tournier]

Hi Maxim,

On Thu, 30 Nov 2023 at 08:28, Maxim Cournoyer  wrote:

> I'd like to have a single archive type as well in the future, but I'd
> settle on Zstd, not lzip, because it's faster to compress and
> decompress, and its compression ratio is not that different when using
> its highest level (19).

When running an inferior (past revision), some past Guile code as it was
in this past revision is launched.  Hum, I have never checked: the
substitution mechanism depends on present revision code (Guile and
daemon) or on past revision?

Other said, what are the requirements for the backward compatibility?
Being able to run past Guix from a recent Guix, somehow.


>>  1. Keep for as longer as we can all the requirements for running Guix
>>  itself, e.g., “guix time-machine”.  Keep all the dependencies and all
>>  the outputs of derivations.  At least, for all the ones the build farms
>>  are already building.
>>
>>  2. Keep for 3-5 years all the outputs for specific Guix revision, as
>>  v1.0, v1.1, v1.2, v1.3, v1.4.  And some few others.
>
> That'd be nice, but not presently doable as we can't fine tune retention
> for a particular 'derivation' and its inputs in the Cuirass
> configuration, unless I've missed it.

That’s an implementation detail, a bug or a feature request, pick the
one you prefer. ;-)

We could imagine various paths for these next steps, IMHO.  For
instance, we could move these outputs to some specific stores
independent of the current ones (ci.guix and bordeaux.guix).  For
instance, we could have “cold” storage with some cooking bakery for
making hot again, instead of keeping all hot.  For instance, we could
imagine etc. :-)

Well, I do not have think much and I just speak loud: Cuirass (and Build
Coordinator) are the builders, and I would not rely on them for some NAR
“archiving“, instead maybe “we” could put some love into the tool
nar-herder.  Somehow, extract specific NAR that the project would like
to keep longer than the unpredictable current mechanism.

Cheers,
simon