Re: Transform options should error on nonexistant targets

[zimoun]

Hi Ryan,

On Wed, 18 Aug 2021 at 03:57, Ryan Prior  wrote:
> I learned today that Guix will chug happily along applying a transform to a 
> nonexistent package.
>
> For example, I can run:
> guix environment --with-latest=not-exist --ad-hoc which
>
> This shows no warning or errors. I think it would be beneficial to show an 
> error & bail if the target of a transformation is a package that doesn't 
> exist, as this is likely indicative of an error.

Indeed. :-)  The issue comes from guix/transformations.scm 
(options->transformation):

--8<---cut here---start->8---
  (let ((new (transform obj)))
(when (eq? new obj)
  (warning (G_ "transformation '~a' had no effect on ~a~%")
   name
   (if (package? obj)
   (package-full-name obj)
   obj)))
new)
--8<---cut here---end--->8---

and the warning is not reach because guix/packages.scm (package-mapping):

--8<---cut here---start->8---
(else
 ;; Return a variant of P with PROC applied to P and its explicit
 ;; dependencies, recursively.  Memoize the transformations.  
Failing
 ;; to do that, we would build a huge object graph with lots of
 ;; duplicates, which in turns prevents us from benefiting from
 ;; memoization in 'package-derivation'.
 (let ((p (proc p)))
   (package
…
--8<---cut here---end--->8---

In the case of “guix build hello --with-latest=foo”, then ’proc’ has no
effect, i.e., ’p’ is identical to ’(proc p)’ but a new package is
allocated which leads that ’new’ and ’obj’ are not ’eq?’.

Well, I have tried the attached patch.  But then ’tests/packages.scm’
fails.  Hum, I need an input because I do not know if I miss something
or if the test is also inaccurate.

--8<---cut here---start->8---
(test-assert "package-input-rewriting/spec"
  (let* ((dep (dummy-package "chbouib"
(native-inputs `(("x" ,grep)
 (p0  (dummy-package "example"
(inputs `(("foo" ,coreutils)
  ("bar" ,grep)
  ("baz" ,dep)
 (rewrite (package-input-rewriting/spec
   `(("coreutils" . ,(const sed))
 ("grep" . ,(const findutils)))
   #:deep? #f))
 (p1  (rewrite p0))
 (p2  (rewrite p0)))
(and (not (eq? p1 p0))
 (eq? p1 p2)  ;memoization
--8<---cut here---end--->8---

I miss why ’p1’ should not be the same as ’p0’.

> What do you think?

Maybe, it could be worth to open a report for that.  Feel free. ;-)


Cheers,
simon

>From e1cd54f4cccad37f7134b342c8dee9da9fa28588 Mon Sep 17 00:00:00 2001
From: zimoun 
Date: Wed, 25 Aug 2021 17:32:58 +0200
Subject: [PATCH 1/1] packages: 'package-mapping' does not allocate unwritten
 package.

Reported by Ryan Prior .

* guix/packages.scm (package-mapping): Do not allocate a new package if the
procedure has no effect.
---
 guix/packages.scm | 31 +--
 1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index c825f427d8..15aa67fe0a 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -6,6 +6,7 @@
 ;;; Copyright © 2017, 2019, 2020 Efraim Flashner 
 ;;; Copyright © 2019 Marius Bakke 
 ;;; Copyright © 2021 Chris Marusich 
+;;; Copyright © 2021 Simon Tournier 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1058,20 +1059,22 @@ applied to implicit inputs as well."
  ;; to do that, we would build a huge object graph with lots of
  ;; duplicates, which in turns prevents us from benefiting from
  ;; memoization in 'package-derivation'.
- (let ((p (proc p)))
-   (package
- (inherit p)
- (location (package-location p))
- (build-system (if deep?
-   (build-system-with-package-mapping
-(package-build-system p) rewrite)
-   (package-build-system p)))
- (inputs (map rewrite (package-inputs p)))
- (native-inputs (map rewrite (package-native-inputs p)))
- (propagated-inputs (map rewrite (package-propagated-inputs p)))
- (replacement (and=> (package-replacement p) replace))
- (properties `((,mapping-property . #t)
-   ,@(package-properties p)
+ (let ((new (proc p)))
+   (if (eq? new p)
+   p
+   

Re: Xiden/Guix design comparison

[Sage Gerard]

Thank you so much for the wealth of resources, and for reading the materials!

I see how you got the workflow aspect, although I hesitate to sum up Xiden as a 
workflow engine. When I wrote Polyglot (a static site generator), it also 
included an abstraction for arbitrary workflows. In my mind, Xiden and Polyglot 
have workflow engines, because I always hated it when I used a library or 
framework, only for it to pressure me into doing things the " way." I 
strongly believe that tools don't get opinions, so whenever I write anything 
over 1000 lines of code, I tend to add on a way to override my design-level 
decisions within my project.

I'm aware of what you're saying re: yogurt, and since Xiden is cross-platform 
it necessarily requests different levels of your trust in system binaries. I'd 
want the best possible scenario on GNU/Linux, hence my other thread. But for 
Windows, I think Xiden's development is basically done.

On 8/25/21 6:50 AM, zimoun wrote:

> Hi Sage,
>
> On Tue, 24 Aug 2021 at 20:33, Sage Gerard
> [](mailto:s...@sagegerard.com)
> wrote:
>
>> This is an offshoot Q&A based off Devos' questions in the "How did you
>> handle making a GNU/Linux distribution?" thread. It pertains to how
>> Guix compares to Xiden on some points.
>>
>> Given the length of both of our emails, I don't expect everyone to
>> read this. Guix is mentioned for discussion purposes, but I use the
>> -devel list only because the questions did. Apologies to the mods if
>> that's not an excuse, since I'm not trying to distract from the
>> purpose of the list.
>>
>> Read on only if you care.
>
> I have read your emails and your Reddit post, and also watched your
> presentation at RacketCon, in addition to the White Paper and some docs.
> Well, I must admit that I lack context, probably from Racket ecosystem,
> to get all the points of what Xiden is.  To answer to your first email,
> IIUC, as I said earlier, you should use the Guix binaries––probably via
> the Ryan’s proposal––to explore your ideas behind Xiden.
>
> Because it is guix-devel, my attempt here is to explain what Guix is and
> maybe you will find how to use it with Xiden. ;-)
>
> From my understanding, Xiden seems like a workflow engine; see for
> example:
> [](https://guixwl.org/)
> Evil is in details but GWL is somehow less strict than Guix itself.  GWL
> defines ’process’es which do whatever stuff using programs and then
> ’workflow’ which is a graph of processes.  On the other hand, Guix
> defines ’package’s which do more or less “only”: ’fetch-from-Internet &&
> ./configure && make && make check && make install’, so a lot of things
> have to be strict, for instance version, method for fetching, etc.
>
> And, again, from my understanding, the launcher looks like a flexible
> and declarative way to describe how the content flows.  Maybe give a
> look at Common Workflow Language [0] or if you have not yet to Dataflow
> [1] stuff.  I do not know if it is relevant, for sure I miss a lot of
> stuff with Xiden. :-) In Guix, nothing really flows but things are just
> built.  All the other stuff Guix does come from the nice properties that
> these builds have.
>
> 0:
> [](https://www.commonwl.org/)
> 1:
> [](https://en.wikipedia.org/wiki/Dataflow_architecture)
> Reading your piece of writings, I thought to topics you might be
> interested in.  Speaking about build systems, a good framework to
> compare them is exposed in “Build à la carte“ [2]; it is a really good
> reading, IMHO.
>
> Guix is now a lot of things, but, from my point of view, at first, Guix
> is somehow a (glue of) build system: it transforms a graph of
> dependencies into binaries items; vertexes are package definitions and
> edges are inputs.  Note that Guix consider 2 kind of inputs: explicit
> and implicit.  Roughly speaking and to stay short, the implicit inputs
> are compilers and various utilities.  For instance, the Guix package
> ’racket-minimal’ [3] requires a C compiler which is not listed in the
> ’inputs’ field but is implicit by ’gnu-build-system’.
>
> 2:
> [](https://www.microsoft.com/en-us/research/publication/build-systems-a-la-carte/)
> 3:
> [](https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/racket.scm?id=ae3e6ba506f57a4353f491242a3f5e4d419f9aa9#n128)
> Now, I would like to point issues about trust.  Today, when you build
> Racket from source (milk), you need compilers (yogourt).  These
> compilers come from source (other milk) compiled by compilers (other
> yogourt).  And so on.  The question is the size of the initial yogourt
> you are ready to eat (trust).  In case you missed the pedestrian
> explanations of Trusting Trust by Carl, please watch [4].  I do not know

Re: Xiden/Guix design comparison

[Sage Gerard]

> Did you discover guix after writing Xiden?

I did, although when I did discover Guix, I incorporated what I learned from 
Guix with the limitations in Racket's approach to package management.

> Why start yet another package manager from scratch?

Do I need a reason? I like programming, I do it to relax, and I think this 
domain is interesting and fun.

Normally when I duplicate an existing effort, it is because I wish to learn how 
it works well enough to be self-sufficient within the domain. This adds more 
effort on my part, but that's the path that led me from scripting web front 
ends to maintaining a homelab, and hopefully write an operating system from 
scratch. I like learning by doing, and that necessarily leads to duplication of 
existing efforts somewhere.

> Are there any features that couldn't just be implemented in Guix itself?

Of course not. This is all FOSS. I could just as easily ask if there are any 
missing features that couldn't just be implemented in Xiden. Ditto for 
low-level system call/cc and Racket. We all eventually deal with C with the 
same audiences, so I see our journeys as equifinal.

The only difference I see is that if we both start from an new user, I 
hypothesize I can teach them to use Xiden faster than you can teach them to use 
Guix. A race could be fun! That would help us both identify usability pain 
points.

Finding things I'm missing is actually part of why I'm here, because I already 
know Xiden is not bootstrapped and is necessarily missing a sufficiently tall 
tower of turtles. But I don't see that as sufficient reason to use Guix, 
because I'm here to learn by doing, not to stop just because I'm given an 
opportunity to stop thinking.

> It is turtles all the way down. I could demo this in private if you are 
> interested. [...] it would be most beneficial at large to integrate some of 
> the most useful elements of your research into Guix, rather than try to 
> reproduce the work to bootstrap the world and also redefine or translate the 
> thousands of package definitions.

I believe you, and I agree. Ryan Prior's email showed me a nice way to layer 
Xiden on top of one of said turtles, and I'm satisfied with that for now. If 
Xiden grows enough to feasibly bootstrap with volunteers, then I'd rather it be 
on its own than dependent on a middleman. I'm not too worried about 
interoperability problems, because that's something I require Xiden to solve 
early on.

And if I may: I think I see one thing that is missing from Guix as an ecosystem 
as opposed to just a tool. Technical superiority can never address the 
subjective (and sometimes silly) reasons that help users pick a tool. Even if I 
acknowledge Guix as technically superior, that doesn't mean I want to use it. 
The time it takes for me to learn Guix well is lower than the time it takes for 
me to replicate the functionality I want at the time using Xiden (see The Lisp 
Curse).

Now, I also want Xiden to be adaptable enough to simplify for laypersons, 
without forcing them to depend on me as a middleman. To me, that's the main 
difference between the user's experience with Guix, and with Xiden.

I don't know how well that will play out yet. Time will tell.

> ---
> From: Sage Gerard [](mailto:s...@sagegerard.com)
> Sent: August 24, 2021 1:33:55 PM PDT
> To: guix-devel@gnu.org
> Subject: Xiden/Guix design comparison
>
> This is an offshoot Q&A based off Devos' questions in the "How did you handle 
> making a GNU/Linux distribution?" thread. It pertains to how Guix compares to 
> Xiden on some points.
>
> Given the length of both of our emails, I don't expect everyone to read this. 
> Guix is mentioned for discussion purposes, but I use the -devel list only 
> because the questions did. Apologies to the mods if that's not an excuse, 
> since I'm not trying to distract from the purpose of the list.
>
> Read on only if you care.
>
> --
>
>> Packages in Guix can specify which version of a package they use. [...] Is 
>> something missing from Guix here?
>
> In Xiden, package versions and the notations used to express them are 
> user-defined. If you use semver and I use dates, we can still map our 
> dependency lists to the same package definitions. Xiden leaves it to us to 
> agree on a scheme for some set of packages.
>
>> SHA-384 and Keccak are separate things? [...] By definition, implementations 
>> of SHA-384 have no reason to use Keccak.
>
> Correct. I'm just saying that Alice's decision to prefer a SHA-384 
> implementation for using Keccak is not our business. Ditto for Bob and SHA-1. 
> Alice can force use of Keccak for software arriving on her system with Xiden, 
> without needing to discuss that with anyone.
>
> She's in charge of how a package manager works on her system, and we aren't. 
> Xiden approaches CHFs similarly to OpenSSL in that implementations are 
> selected by providers, except in Xiden a knowledgeable user may assume th

Re: Xiden/Guix design comparison

[Ryan Sundberg]

Hello Sage,

Why start yet another package manager from scratch? Are there any features that 
couldn't just be implemented in Guix itself? I prefer development in Racket 
over guile too, but Guile with it's low level system call/c compatibility plays 
to its strengths for the use cases involved in systems programming a package 
manager.

Did you discover guix after writing Xiden? I'm not understanding the use cases 
Xiden proposes that would be difficult to implement in Guix (for example, GPG 
signature verification for package sources). I don't think Racket as a platform 
is compelling enough to justify the effort.

You also compare Xiden to Racket as if it is a package manager to write other 
package managers with. But you can also just call into Guix modules directly 
and integrate guix into any other guile program and build a new package manager 
on top of it. I personally have used Racket to generate guile code to call guix 
modules to spawn processes for deterministic/generated virtual machines. It is 
turtles all the way down. I could demo this in private if you are interested.

Anyways it sounds like there is some valuable research in Xiden, but I think it 
would be most beneficial at large to integrate some of the most useful elements 
of your research into Guix, rather than try to reproduce the work to bootstrap 
the world and also redefine or translate the thousands of package definitions. 
The maintenance alone on the myriad package definitions is enough to keep up 
with for all of us put together.

My $0.02! Thanks for sharing.

Sincerely,

Ryan Sundberg


 Original Message 
From: Sage Gerard 
Sent: August 24, 2021 1:33:55 PM PDT
To: guix-devel@gnu.org
Subject: Xiden/Guix design comparison

This is an offshoot Q&A based off Devos' questions in the "How did you handle 
making a GNU/Linux distribution?" thread. It pertains to how Guix compares to 
Xiden on some points.

Given the length of both of our emails, I don't expect everyone to read this. 
Guix is mentioned for discussion purposes, but I use the -devel list only 
because the questions did. Apologies to the mods if that's not an excuse, since 
I'm not trying to distract from the purpose of the list.

Read on only if you care.

--

> Packages in Guix can specify which version of a package they use. [...] Is 
> something missing from Guix here?

In Xiden, package versions and the notations used to express them are 
user-defined. If you use semver and I use dates, we can still map our 
dependency lists to the same package definitions. Xiden leaves it to us to 
agree on a scheme for some set of packages.

> SHA-384 and Keccak are separate things? [...] By definition, implementations 
> of SHA-384 have no reason to use Keccak.

Correct. I'm just saying that Alice's decision to prefer a SHA-384 
implementation for using Keccak is not our business. Ditto for Bob and SHA-1. 
Alice can force use of Keccak for software arriving on her system with Xiden, 
without needing to discuss that with anyone.

She's in charge of how a package manager works on her system, and we aren't. 
Xiden approaches CHFs similarly to OpenSSL in that implementations are selected 
by providers, except in Xiden a knowledgeable user may assume the role of a CHF 
implementation provider for their own system. I suspect we'd agree that is a 
terrible idea, which is why I use zero-trust defaults for everything to 
mitigate risks.

I allow for danger because most software distribution problems I've seen in the 
field boil down to smart people dealing with the lasting consequences of 
outdated or rushed decisions. I'm not assuming competency, but I am assuming 
that even those with bad practices still need to solve their distribution 
problems.

>> unconditionally prefers cached installations, such that every defined 
>> artifact
>> has exactly one stored copy on disk.
>
> Coming from Guix, I don't know what this means. Does this mean only one 
> version
> of a package can be in the store at the same time, and building a newer 
> version
> deletes the older version? How does this compare with ‘content deduplication’?

Xiden has a cache for package inputs, and a cache for installed outputs. The 
user has more control over the latter. Depending on the configuration you use 
in a launcher, you can map all inputs to the same content-addressable 
filesystem, implying deduplication. However, you can also decide to defeat the 
installed output cache for the sake of how SxS installations in Xiden. That way 
you can store any number of any revision of any package in one store (Although 
I use the word "state" in the docs).

>> but only for artifacts from services authenticated by his operating system's 
>> certificates
>
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>
> (source (origin
> (method url-fetch)
> (uri (string-append ["mirror://gnu/hello/hello-"](mirror://gnu/hello/hello-) 
> version
> ".tar.

Re: Xiden/Guix design comparison

[zimoun]

Hi Sage,

On Tue, 24 Aug 2021 at 20:33, Sage Gerard  wrote:

> This is an offshoot Q&A based off Devos' questions in the "How did you
> handle making a GNU/Linux distribution?" thread. It pertains to how
> Guix compares to Xiden on some points. 
>
> Given the length of both of our emails, I don't expect everyone to
> read this. Guix is mentioned for discussion purposes, but I use the
> -devel list only because the questions did. Apologies to the mods if
> that's not an excuse, since I'm not trying to distract from the
> purpose of the list. 
>
> Read on only if you care.

I have read your emails and your Reddit post, and also watched your
presentation at RacketCon, in addition to the White Paper and some docs.
Well, I must admit that I lack context, probably from Racket ecosystem,
to get all the points of what Xiden is.  To answer to your first email,
IIUC, as I said earlier, you should use the Guix binaries––probably via
the Ryan’s proposal––to explore your ideas behind Xiden.

Because it is guix-devel, my attempt here is to explain what Guix is and
maybe you will find how to use it with Xiden. ;-)

>From my understanding, Xiden seems like a workflow engine; see for
example:

  

Evil is in details but GWL is somehow less strict than Guix itself.  GWL
defines ’process’es which do whatever stuff using programs and then
’workflow’ which is a graph of processes.  On the other hand, Guix
defines ’package’s which do more or less “only”: ’fetch-from-Internet &&
./configure && make && make check && make install’, so a lot of things
have to be strict, for instance version, method for fetching, etc.

And, again, from my understanding, the launcher looks like a flexible
and declarative way to describe how the content flows.  Maybe give a
look at Common Workflow Language [0] or if you have not yet to Dataflow
[1] stuff.  I do not know if it is relevant, for sure I miss a lot of
stuff with Xiden. :-) In Guix, nothing really flows but things are just
built.  All the other stuff Guix does come from the nice properties that
these builds have.

0: 
1: 

Reading your piece of writings, I thought to topics you might be
interested in.  Speaking about build systems, a good framework to
compare them is exposed in “Build à la carte“ [2]; it is a really good
reading, IMHO.

Guix is now a lot of things, but, from my point of view, at first, Guix
is somehow a (glue of) build system: it transforms a graph of
dependencies into binaries items; vertexes are package definitions and
edges are inputs.  Note that Guix consider 2 kind of inputs: explicit
and implicit.  Roughly speaking and to stay short, the implicit inputs
are compilers and various utilities.  For instance, the Guix package
’racket-minimal’ [3] requires a C compiler which is not listed in the
’inputs’ field but is implicit by ’gnu-build-system’.

2:

3: 


Now, I would like to point issues about trust.  Today, when you build
Racket from source (milk), you need compilers (yogourt).  These
compilers come from source (other milk) compiled by compilers (other
yogourt).  And so on.  The question is the size of the initial yogourt
you are ready to eat (trust).  In case you missed the pedestrian
explanations of Trusting Trust by Carl, please watch [4].  I do not know
about Microsoft, and about Mac the initial yogourt is really a huge
piece. :-) The initial yogourt used by Guix is detailed here [5,6].

4: 
5: 
6: 


My point here is if Xiden works on Mac and the launcher is zero-trust,
it could be worth to specify against which attack Xiden is trying to
protect.  IIUC, not against a Trusting Trust attack. ;-)

Guix tries to get a straight path from a minimal set of binaries to
modern compilers.  The aim [7] is to have human-auditable binaries,
somehow.  That’s said, this path is running using a (binary) kernel on
the top of hardware, so still a chicken-egg problem. ;-) The question,
as always, is against what we protect.  Note that another path is tried:
formal verification. Somehow, the audit is done by proof-assistant
[8,9]; another long story. :-)

7: 
8: 
9: 

Last, about preserving semantics, i.e., do not rebuild, it is a
difficult task.  For instance, I remember the idea exposed in this blog
post [10].  And about functional principles applied to deployment,
’propellor’ [11,12] by Joey Hess might be interesting.  Or not. :-)

10:

11: