Re: rust-build-system from antioxidant

[Maxim Cournoyer]

Hi Nicolas,

Nicolas Graves  writes:

> Hi all,
>
> Thanks for this discussion that I didn't expected to happen ;)
>
> I'll try and finish a version of the rust-build-system but I'd like to
> know if there are reasons to not want a direct and complete rewrite of
> all Rust packages before putting more time into this. My rationale is
> that since we have channels, users won't really be affected in any
> meaningful way because adding a "snapshot of cargo packages" in a
> channel and then adding the channel is straightforward.

Thank you!

My impression is that the Rust packaging in Guix is so problematic that
if a way would mean less work for you and motivate you to push this work
to the finish line it, that may well be the better way :-).

> I've merged the rust-build-system and rust-workspace-build-system into a
> single one, and made some UX improvements (a dozen patches on top of
> Maxime's work).
>
> If atomicity / readability of changes is the issue, I can try to cut the
> packages rewrites into patches (although that would make more than 1k
> patches total I think).

That would match our convention better, if it's not too difficult to
accomplish (is this going to be automated?  I've written some imperfect
but useful automatic rewriting tool in the past to remove the Python 2
packages [0].  Perhaps it could provide some ideas)

[0]  
https://notabug.org/apteryx/guix-api-examples/src/master/purge-python2-packages.scm

-- 
Thanks,
Maxim

Re: rust-build-system from antioxidant

[Development of GNU Guix and the GNU System distribution.]

Hi all,

Thanks for this discussion that I didn't expected to happen ;)

I'll try and finish a version of the rust-build-system but I'd like to
know if there are reasons to not want a direct and complete rewrite of
all Rust packages before putting more time into this. My rationale is
that since we have channels, users won't really be affected in any
meaningful way because adding a "snapshot of cargo packages" in a
channel and then adding the channel is straightforward.

I've merged the rust-build-system and rust-workspace-build-system into a
single one, and made some UX improvements (a dozen patches on top of
Maxime's work).

If atomicity / readability of changes is the issue, I can try to cut the
packages rewrites into patches (although that would make more than 1k
patches total I think).

-- 
Best regards,
Nicolas Graves

Re: rust-build-system from antioxidant

[Maxime Devos]

Op 12-06-2023 om 15:05 schreef Maxim Cournoyer:

Hi Maxime,

Maxime Devos  writes:


Op 12-06-2023 om 03:17 schreef Maxim Cournoyer:
[...]
Yes.  Overruling is a form of blocking, and blocking by authority
(whether de facto or de jure) is overruling.


There should not be a notion of 'overruling' in our
contribution processes (unless the Guix co-maintainers have to step in
as a last resort) if all participants strive to build consensus, as
mentioned in info '(guix) Commit Access':
 It is expected from all contributors, and even more so from
committers,
 to help build consensus and make decisions based on consensus.  To learn
 what consensus decision making means and understand its finer details,
 you are encouraged to read
 .
I thought I knew what consensus meant myself, but the above link
helped
me to re-frame a few things in a way that is more conducive to building
consensus.




[...]


For example, the thread of the patch you sent at
 is a good example of this, pretty
much everyone (except Ludo') agreed that the provided patch is good.


Let's avoid directly criticising ourselves and try to discuss what I
think has more value, which is coming to a better understanding of the
situation and how the perceived deadlock could be undone.
Consensus is not a majority vote; all parties have to walk the extra mile

>to reach a

common ground.


This quote is taken out of context -- nowhere I stated or assumed that 
it was a majority vote -- (n - tiny)/n is higher than 50%, and I also 
demonstrated that some other criteria were met.


Please don't ignore the explanation I wrote below all that, let's avoid 
taking quotes out-of-context, and let's avoid ignoring the concerns I 
pointed out in my previous e-mail.


> [...] try to discuss what I think has more value,

which is coming to a better understanding of the situation, [...]


I did this in my previous e-mail.  You can bring new discussion from 
another POV if you want to, but why are you ignoring my discussion on 
this matter?


> [...] all parties have to walk the extra mile to reach a common
> ground. [...]

This sentence would have more weight if you explained somewhere how this 
wasn't the case in .



I think the object there was from a semantic point of
view: we'd have a 'garbage collection' command (guix gc) which wouldn't
collect any garbage!  It's a valid objection, although its importance in

> the narrow use case presented was not agreed by all parties.

Ludo' claimed that the resulting patch wouldn't collect garbage:


I believe the effect is that ‘guix gc -D /gnu/store/…-disk-image’ would
remove nothing: /gnu/store/.links would still contain a copy of that big
disk image, so as a result, you’ve freed zero bytes.


On its own, this is a valid objection, but it is false -- it does 
collect some garbage -- it deletes the /gnu/store/ITEM.  This is 
implicitly referred to in:


> https://issues.guix.gnu.org/51427#4

[...]  Obviously there needs to be a way of __removing
single items from the store__, [...]


> zimoun wrote on 9 Nov 2021 19:10

[...] Even if the phase is drastically speed up, it would be probably still
too slow when using the option ’-D’ __remove only one __; or just
some. [...]


Yet, Ludo' seems to have missed this:

> [Ludovic Courtès wrote on 17 Nov 2021 11:02]
> [...]> No; like I wrote, it would have the effect of not deleting 
anything:

> [...

So I have written it down explicitly:

> https://issues.guix.gnu.org/51427#25
Also, it _does_ collect garbage -- it collects the /gnu/store/... item, 
it just doesn't collect _all_ the garbage (it doesn't collect the 
individual files in the store item or the things in /gnu/store/.links).


There wasn't any response.


A consensus-based outcome could be to add a new option to guix gc,
e.g. '--invalidate', which would be documented as "invalidate
(de-register from the Guix database) rather actually delete from the
store".  If that's still argued semantically unclear we could go with a
dedicated 'guix invalidate', although that seems overkill to me.


"guix gc -D" is already semantically clear -- it deletes a single 
/gnu/store/... item.  IIUC, the patch at 
 fixes the bug where it also 
deletes more than that.


(IMO both deleting and not deleting the relevant links in 
/gnu/store/.links would be acceptable behaviors, as long as it doesn't 
do more than that.  Also IMO deleting more links is technically a bug, 
but harmless __as long as it's efficient.)


(IIUC, it also deregisters the store item, but that's needed for 
consistency of DB->file system, and deregistering can be considered a 
removal of DB entries, so not really a problem.)



This is a bit more work than the 1 line change initially suggested, but
I think we can agree that'd be a more general/better solution.


It's neither more general nor less 

Re: rust-build-system from antioxidant

[Development of GNU Guix and the GNU System distribution.]

Hi Vagrant,

On Mon, Jun 12, 2023 at 8:09 AM Vagrant Cascadian  wrote:
>
> everything kind of has a social component!

Thank you for pointing that out! Many technical discussions about
project governance claim high moral ground but remain incomplete.

Maxime possibly felt some righteous indignation because Maxim's
well-intended reference to consensus does not actually apply in Guix
that often. (I have to be careful to keep both your names in order.)
It would be ignorant to claim that there are no power structures.

Not all of them are official, but they are there. [1] Hierarchies are
human, and they are not always bad.

During the past year, I made some ambitious proposals that did not
resonate with the general public or any of the folks in power. (I am a
member of the general public.) Some committers told me privately that
nothing big happens without the maintainer collective.

Someone in the maintainer collective told me that nothing truly
important (like legal stuff) happens without Ludo'.

Of course, that expectation places an extraordinary burden on Ludo',
who already works very hard. I don't think he (or maybe "they") even
wants all that responsibility. Personally, I find Ludo' measured and
generous.

That being said, everyone makes mistakes. Sometimes, words are
misunderstood. At other times, the merit of an argument is overlooked.
Maybe that's what happened in the bug Maxime cited. Or maybe Maxime
made a mistake with that narrow, unfavorable and critical assessment.
Either way, it's important to have goodwill toward one another.

I know a little bit about how large groups can work together. For the
past nine years, I have been a minor city official in a bustling
community of 230,000 immigrants—mostly from India and China, with 158
languages spoken at home. The key is to keep solving the community's
problems and also, to solve other people's problems in addition to my
own.

In Jewish mysticism, there is an old story about a giant vessel that
once existed when the world was made. Unfortunately, it broke into a
gazillion pieces, and we each ended up with a little shard. We spend
the remainder of our lives looking for other pieces that fit.

Let's be proud of our large and diverse community. Let's focus on the
pieces that fit together. Thank you all for being here!

Kind regards
Felix

[1] https://guix.gnu.org/blog/2022/gnu-guix-maintainer-rotation/

Re: rust-build-system from antioxidant

[Vagrant Cascadian]

On 2023-06-11, Maxim Cournoyer wrote:
> Maxime Devos  writes:
>> Op 02-06-2023 om 20:02 schreef Nicolas Graves:
>>> A few months ago, Maxime Devos worked on a new rust-build-system to
>>> handle a few issues we were experiencing with cargo (see discussions on
>>> antioxidant in guix-devel).
>>> A month ago, we discussed about the possibility of the integration
>>> in
>>> core guix, and the required steps. Maxime and I had a different
>>> approach. Maxime highlighted the possibility to make a smooth transition
>>> but once that would require many gradual changes and deprecation. My
>>> approach was that since we'll have to eventually migrate all packages to
>>> rust-build-system, and since we can freeze all former rust packages in
>>> an archive channel, I would be clearer to make the transition at once.
>>> [...]
>>
>> Actually, I started out with the non-gradual approach, but that was
>> overruled by Ludo', IIRC.
>
> Did you perhaps meant to say that it was disagreed with, or at worst
> "blocked by"?  There should not be a notion of 'overruling' in our
> contribution processes (unless the Guix co-maintainers have to step in
> as a last resort) if all participants strive to build consensus, as
> mentioned in info '(guix) Commit Access':
>
>It is expected from all contributors, and even more so from committers,
>to help build consensus and make decisions based on consensus.  To learn
>what consensus decision making means and understand its finer details,
>you are encouraged to read
>.
>
> I thought I knew what consensus meant myself, but the above link helped
> me to re-frame a few things in a way that is more conducive to building
> consensus.

A possible reframing that might be relevent here too that is maybe not
captured in the seedsforchange link referenced...

In a consensus decision (formal or informal), it is often really
valuable to not get caught up in "so-and-so is blocking X, Y, Z", but
rather more "this issue so-and-so raised is blocking X, Y, Z" or even
"this issue is blocking X, Y, Z". It is a perhaps subtle distinction,
but an important one, as I think it can refocus on a problem-solving
approach rather than finger-pointing-and-blaming approach.

That said, I definitely get the impression the Guix community practices
a very informal ad-hoc form of consensus, which ... while very flexible,
misses many of the strongest benefits of consensus (notably, clarity of
when a decision is actually reached and the next steps to move forward
with it).

A quick attempt at condensing a (more) formal consensus decision making
process boils down to:

* describe a situation, issue, proposal, etc.
* clarify understanding of the above (this step unfortunately
  may get skipped as people are very eagre to...)
* raise concerns
* address concerns (as a distinct step from raising concerns)
(iterate and repeat above steps as necessary)
* call for a decision (all concerns ideally resolved or mostly so.. ?)
* record decision (noting outstanding concerns if any)

I am not suggesting that a more formal process is even appropriate to
guix patch review...

Obviously, with the asyncronous nature of, say, a mailing list
discussion, these things do not necessarily happen in a structured
sequential way, though possibly being aware of the "ideal" formal
process, people can informally attempt to approximate it.

I do want to point out that informal processes are more prone to falling
into difficult patterns (e.g. defaulting to hierarchy or other power
dynamics), at least with the social aspects... I think this is a
collaborative project, so everything kind of has a social component!

We all fall down sometimes or lapse into old habits now and then, but
even if there is no formalized clarity of process, it is helpful to at
least reach towards the spirit of consensus building...


live well,
  vagrant


signature.asc
Description: PGP signature

Re: rust-build-system from antioxidant

[Maxim Cournoyer]

Hi Maxime,

Maxime Devos  writes:

> Op 12-06-2023 om 03:17 schreef Maxim Cournoyer:
>> Hi Maxime,
>> Maxime Devos  writes:
>> 
>>> Op 02-06-2023 om 20:02 schreef Nicolas Graves:
 A few months ago, Maxime Devos worked on a new rust-build-system to
 handle a few issues we were experiencing with cargo (see discussions on
 antioxidant in guix-devel).
 A month ago, we discussed about the possibility of the integration
 in
 core guix, and the required steps. Maxime and I had a different
 approach. Maxime highlighted the possibility to make a smooth transition
 but once that would require many gradual changes and deprecation. My
 approach was that since we'll have to eventually migrate all packages to
 rust-build-system, and since we can freeze all former rust packages in
 an archive channel, I would be clearer to make the transition at once.
 [...]
>>>
>>> Actually, I started out with the non-gradual approach, but that was
>>> overruled by Ludo', IIRC.
>> Did you perhaps meant to say that it was disagreed with, or at worst
>> "blocked by"?
>
> Yes.  Overruling is a form of blocking, and blocking by authority
> (whether de facto or de jure) is overruling.
>
>> There should not be a notion of 'overruling' in our
>> contribution processes (unless the Guix co-maintainers have to step in
>> as a last resort) if all participants strive to build consensus, as
>> mentioned in info '(guix) Commit Access':
>> It is expected from all contributors, and even more so from
>> committers,
>> to help build consensus and make decisions based on consensus.  To learn
>> what consensus decision making means and understand its finer details,
>> you are encouraged to read
>> .
>> I thought I knew what consensus meant myself, but the above link
>> helped
>> me to re-frame a few things in a way that is more conducive to building
>> consensus.
>

[...]

> For example, the thread of the patch you sent at
>  is a good example of this, pretty
> much everyone (except Ludo') agreed that the provided patch is good.

Let's avoid directly criticising ourselves and try to discuss what I
think has more value, which is coming to a better understanding of the
situation and how the perceived deadlock could be undone.  Consensus is
not a majority vote; all parties have to walk the extra mile to reach a
common ground.  I think the object there was from a semantic point of
view: we'd have a 'garbage collection' command (guix gc) which wouldn't
collect any garbage!  It's a valid objection, although its importance in
the narrow use case presented was not agreed by all parties.

A consensus-based outcome could be to add a new option to guix gc,
e.g. '--invalidate', which would be documented as "invalidate
(de-register from the Guix database) rather actually delete from the
store".  If that's still argued semantically unclear we could go with a
dedicated 'guix invalidate', although that seems overkill to me.

This is a bit more work than the 1 line change initially suggested, but
I think we can agree that'd be a more general/better solution.  Such is
the trade-off of consensus-based decision making (requires more
effort/slower moving but with a higher quality outcome).

-- 
Thanks,
Maxim

Re: rust-build-system from antioxidant

[Maxime Devos]

Op 12-06-2023 om 03:17 schreef Maxim Cournoyer:

Hi Maxime,

Maxime Devos  writes:


Op 02-06-2023 om 20:02 schreef Nicolas Graves:

A few months ago, Maxime Devos worked on a new rust-build-system to
handle a few issues we were experiencing with cargo (see discussions on
antioxidant in guix-devel).
A month ago, we discussed about the possibility of the integration
in
core guix, and the required steps. Maxime and I had a different
approach. Maxime highlighted the possibility to make a smooth transition
but once that would require many gradual changes and deprecation. My
approach was that since we'll have to eventually migrate all packages to
rust-build-system, and since we can freeze all former rust packages in
an archive channel, I would be clearer to make the transition at once.
[...]


Actually, I started out with the non-gradual approach, but that was
overruled by Ludo', IIRC.


Did you perhaps meant to say that it was disagreed with, or at worst
"blocked by"?


Yes.  Overruling is a form of blocking, and blocking by authority 
(whether de facto or de jure) is overruling.



There should not be a notion of 'overruling' in our
contribution processes (unless the Guix co-maintainers have to step in
as a last resort) if all participants strive to build consensus, as
mentioned in info '(guix) Commit Access':

It is expected from all contributors, and even more so from committers,
to help build consensus and make decisions based on consensus.  To learn
what consensus decision making means and understand its finer details,
you are encouraged to read
.

I thought I knew what consensus meant myself, but the above link helped
me to re-frame a few things in a way that is more conducive to building
consensus.


In my experience, Ludovic often doesn't actually practice that, though.
For example, the thread of the patch you sent at 
 is a good example of this, pretty 
much everyone (except Ludo') agreed that the provided patch is good.


People pointed out how Ludo' multiple time missed the point of the 
patch.  Yet, Ludo' kept missing the point, e.g. consider:


> [Liliana Marie prikler wrote on 18 jul 2022 19:03]

Am Montag, dem 18.07.2022 um 15:57 +0200 schrieb Ludovic Courtès:
Toggle quote (7 lines)

Hello,

With commit 472a0e82a52a3d5d841e1dfad6b13e26082a5750 (Nov. 2021),
partially fixing , there is
hopefully less pressure to skip the remove-unused-links phase.

Should we close this issue?

As a hard disk user, I'm leaning towards "no".  In fact, I recently
encountered a case where I think I might want to skip it even if not
deleting "specific items".  [...


> [Ludovic]

I agree that we should strive to have good performance on that kind of
hardware too, but I don’t know how to get there.


That's what the patch is for:

> [Liliana]
> [...]

I agree that we should strive to have good performance on that kind
of hardware too, but I don’t know how to get there.

I don't think deleting links will ever be fast on that disk.  But what
I've been saying the whole time is that I don't always need the links
deleted.  I think adding "expert" switches to skip these phases might
actually be enough – after all, if I ever do want to run a full GC, the
information ought to be the same, no?


[A remainder by me that Ludovic is missing the point]
[...] The point isn't to work-around slow "deleting unused links" 
implementation, but rather to avoid inherit slowness of deleting 
everything when deleting a few things suffice [...]

> [...]

Apologies for being elliptic.  My point here, as has been discussed
earlier in this thread, is that we can’t just skip that phase or we’d
simply leave files around without actually deleting them.


As repeatedly explained in different ways previously, we can, in fact, 
just do this and these consequences are unproblematic.  This is again 
explained in different ways in a response by Liliana and me.


Given that a few of the participants that wanted the patch in some form, 
are actually committers yet didn't commit it even though there was 
consensus, and that de facto Ludo' has authority and disagreed with the 
patch, the only conclusion I can draw from this, is that Ludo' 
effectively overruled the consensus with their (de facto) authority.


Whether this was intentional or not, the effect was the same.

(Technically ‘https://www.seedsforchange.org.uk/consensus#block’ allows 
for an unilateral block, but it later writes ‘However it provides a 
safety net for situations where a proposal would seriously hurt the 
group of people in it’, which doesn't apply at all here.)


This is not the antioxidant stuff, but it should serve as an explanation 
on why I do not believe that Ludovic practices consensus.  There are 
also a few other examples/threads that look suspect to me, but they are 
very ambiguous/not clear-cut.


Greetings,
Maxime.

Re: rust-build-system from antioxidant

[Maxim Cournoyer]

Hi Maxime,

Maxime Devos  writes:

> Op 02-06-2023 om 20:02 schreef Nicolas Graves:
>> A few months ago, Maxime Devos worked on a new rust-build-system to
>> handle a few issues we were experiencing with cargo (see discussions on
>> antioxidant in guix-devel).
>> A month ago, we discussed about the possibility of the integration
>> in
>> core guix, and the required steps. Maxime and I had a different
>> approach. Maxime highlighted the possibility to make a smooth transition
>> but once that would require many gradual changes and deprecation. My
>> approach was that since we'll have to eventually migrate all packages to
>> rust-build-system, and since we can freeze all former rust packages in
>> an archive channel, I would be clearer to make the transition at once.
>> [...]
>
> Actually, I started out with the non-gradual approach, but that was
> overruled by Ludo', IIRC.

Did you perhaps meant to say that it was disagreed with, or at worst
"blocked by"?  There should not be a notion of 'overruling' in our
contribution processes (unless the Guix co-maintainers have to step in
as a last resort) if all participants strive to build consensus, as
mentioned in info '(guix) Commit Access':

   It is expected from all contributors, and even more so from committers,
   to help build consensus and make decisions based on consensus.  To learn
   what consensus decision making means and understand its finer details,
   you are encouraged to read
   .

I thought I knew what consensus meant myself, but the above link helped
me to re-frame a few things in a way that is more conducive to building
consensus.

-- 
Thanks,
Maxim

Re: rust-build-system from antioxidant

[Maxime Devos]

Op 02-06-2023 om 20:02 schreef Nicolas Graves:


A few months ago, Maxime Devos worked on a new rust-build-system to
handle a few issues we were experiencing with cargo (see discussions on
antioxidant in guix-devel).

A month ago, we discussed about the possibility of the integration in
core guix, and the required steps. Maxime and I had a different
approach. Maxime highlighted the possibility to make a smooth transition
but once that would require many gradual changes and deprecation. My
approach was that since we'll have to eventually migrate all packages to
rust-build-system, and since we can freeze all former rust packages in
an archive channel, I would be clearer to make the transition at once.

> [...]

Actually, I started out with the non-gradual approach, but that was 
overruled by Ludo', IIRC.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature