Re: How did you handle making a GNU/Linux distribution?

[Christine Lemmer-Webber]

Philip McGrath  writes:

> On 9/12/21 11:09 PM, Christine Lemmer-Webber wrote:
>> Philip McGrath  writes:
>>> Christine Lemmer-Webber had floated the idea at some point of trying
>>> to integrate Racket and Guile.
>>>
>>> IIRC, I think what she's had in mind was trying to make a Guile
>>> backend for Racket along the lines of the Chez Scheme backend (or the
>>> BC backend, or experimental backends like Pycket).
>> Yes that's what I had in mind :)
>
> A few stray thoughts, all with the caveat that I haven't actually
> tried any of this ...
>
>>> there are two things that have struck me
>>> as downsides:
>>>
>>>   1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
>>>  Report)" (§6, p. 13) that, "If our task were to compile Racket to an
>>>  existing target, then we would not have achieved such a high degree
>>>  of compatibility. … we have taken the liberty of modifying Chez
>>>  Scheme to make it an easier target for Racket."
>>>
>>>  https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf
>>>
>>>  Presumably a Racket-on-Guile project would face the same trade-off,
>>>  where modifications to Guild, if Racket CS is a guide, could require
>>>  hard work over many years, and lesser compatibility would make the
>>>  result less useful.
>> At one point when I spoke to Matthew, he was very optimistic that
>> the
>> work on porting on top of Chez would open Racket to running on top of
>> many other backends.  But yes... since there have been so many "custom"
>> modifications to Chez, it's easier to be skeptical about that these
>> days...
>
> I think there a few possible senses for what "running on top of many
> other backends" could mean. My impression overall is that it has
> gotten easier, but may not necessarily be easy.
>
> The most clearly demonstrated is that it seems to be easier to port
> Chez Scheme to new architectures than to port Racket BC's VM. Racket
> CS supports the Apple M1 chip natively (and hopefully will support the 
> Linux kernel on the M1 when that stabilizes), which Racket BC does
> not. Racket CS also fully supports some platforms (ARM in general,
> IIRC) on which Racket BC lacks support for futures, places, or the
> JIT. The most promising route to Racket on WASM also seems to be
> adding a WASM backend to the Chez Scheme compiler. (In fairness, there
> are also some architectures, like ppc64le, to which no one has ported
> Racket CS yet, for which Racket BC can offer at least some support via
> C.)

Who will get to WASM first, Racket or Guile?  :)

> More generally, the "linklet" abstraction[1] seems to provide a much
> more clear boundary between the responsibilities of a backend 
> implementation and the common frontend than existed before Racket
> 7.0. For example, the backend doesn't have to deal with macro
> expansion or even shadowing of variables: it just need to have some
> way to instantiate linklets (likely with further backend-specific
> compilation) and to provide a whole bunch of primitives. I believe
> I've heard Sam Tobin-Hochstadt say that linklets have made it possible
> for Pycket (the experimental Racket implementation in Reticulated
> Python) to run most Racket code.
>
> [1]: https://docs.racket-lang.org/reference/linklets.html
>
> Another benefit of linklets is that they've defined a clear path for
> implementing parts of Racket in Racket itself, like the regexp 
> implementation on Racket CS. This seems like it could help a lot with
> supplying that very large number of primitives.
>
> So I expect it would be entirely possible to implement a linklet-based
> Racket backend in Guile. I do suspect that getting production-quality 
> performance and compatibility could be a lot of work. But my bigger
> concern is ...
>
>>>   2. As you probably know, Racket programs can't generally use
>>>  Chez Scheme implemented libraries, because Chez Scheme effectively
>>>  is the "unsafe" layer of the Racket VM. For example, not all Racket
>>>  procedures are Chez Scheme procedures, and Racket's continuations
>>>  wrap Chez Scheme's to implement delimited and composable control,
>>>  threads, parameters, full continuation marks, etc.
>>>
>>>  For Racket CS, this isn't a great loss (there aren't so many
>>>  Chez-specific libraries, and portable libraries can run in Racket's
>>>  R6RS language), but, for a hypothetical Racket-on-Guile,
>>>  bidirectional interoperability would be a big attraction: imagine
>>>  Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
>>>  working together.
>
> I don't think a Racket-on-Guile that worked like Racket-on-Chez would
> achieve the most interesting benefit (IMO) of bringing the two
> languages together: safe, bidirectional interoperability.
>
> Probably there are many ways to achieve that goal. In principle, one
> could write a Racket-on-Guile implementation where arbitrary Guile
> could run without breaking Racket's 

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

On 9/12/21 11:09 PM, Christine Lemmer-Webber wrote:

Philip McGrath  writes:

Christine Lemmer-Webber had floated the idea at some point of trying
to integrate Racket and Guile.

IIRC, I think what she's had in mind was trying to make a Guile
backend for Racket along the lines of the Chez Scheme backend (or the
BC backend, or experimental backends like Pycket).


Yes that's what I had in mind :)


A few stray thoughts, all with the caveat that I haven't actually tried 
any of this ...



there are two things that have struck me
as downsides:

  1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
 Report)" (§6, p. 13) that, "If our task were to compile Racket to an
 existing target, then we would not have achieved such a high degree
 of compatibility. … we have taken the liberty of modifying Chez
 Scheme to make it an easier target for Racket."

 https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf

 Presumably a Racket-on-Guile project would face the same trade-off,
 where modifications to Guild, if Racket CS is a guide, could require
 hard work over many years, and lesser compatibility would make the
 result less useful.


At one point when I spoke to Matthew, he was very optimistic that the
work on porting on top of Chez would open Racket to running on top of
many other backends.  But yes... since there have been so many "custom"
modifications to Chez, it's easier to be skeptical about that these
days...


I think there a few possible senses for what "running on top of many 
other backends" could mean. My impression overall is that it has gotten 
easier, but may not necessarily be easy.


The most clearly demonstrated is that it seems to be easier to port Chez 
Scheme to new architectures than to port Racket BC's VM. Racket CS 
supports the Apple M1 chip natively (and hopefully will support the 
Linux kernel on the M1 when that stabilizes), which Racket BC does not. 
Racket CS also fully supports some platforms (ARM in general, IIRC) on 
which Racket BC lacks support for futures, places, or the JIT. The most 
promising route to Racket on WASM also seems to be adding a WASM backend 
to the Chez Scheme compiler. (In fairness, there are also some 
architectures, like ppc64le, to which no one has ported Racket CS yet, 
for which Racket BC can offer at least some support via C.)


More generally, the "linklet" abstraction[1] seems to provide a much 
more clear boundary between the responsibilities of a backend 
implementation and the common frontend than existed before Racket 7.0. 
For example, the backend doesn't have to deal with macro expansion or 
even shadowing of variables: it just need to have some way to 
instantiate linklets (likely with further backend-specific compilation) 
and to provide a whole bunch of primitives. I believe I've heard Sam 
Tobin-Hochstadt say that linklets have made it possible for Pycket (the 
experimental Racket implementation in Reticulated Python) to run most 
Racket code.


[1]: https://docs.racket-lang.org/reference/linklets.html

Another benefit of linklets is that they've defined a clear path for 
implementing parts of Racket in Racket itself, like the regexp 
implementation on Racket CS. This seems like it could help a lot with 
supplying that very large number of primitives.


So I expect it would be entirely possible to implement a linklet-based 
Racket backend in Guile. I do suspect that getting production-quality 
performance and compatibility could be a lot of work. But my bigger 
concern is ...



  2. As you probably know, Racket programs can't generally use
 Chez Scheme implemented libraries, because Chez Scheme effectively
 is the "unsafe" layer of the Racket VM. For example, not all Racket
 procedures are Chez Scheme procedures, and Racket's continuations
 wrap Chez Scheme's to implement delimited and composable control,
 threads, parameters, full continuation marks, etc.

 For Racket CS, this isn't a great loss (there aren't so many
 Chez-specific libraries, and portable libraries can run in Racket's
 R6RS language), but, for a hypothetical Racket-on-Guile,
 bidirectional interoperability would be a big attraction: imagine
 Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
 working together.


I don't think a Racket-on-Guile that worked like Racket-on-Chez would 
achieve the most interesting benefit (IMO) of bringing the two languages 
together: safe, bidirectional interoperability.


Probably there are many ways to achieve that goal. In principle, one 
could write a Racket-on-Guile implementation where arbitrary Guile could 
run without breaking Racket's invariants, but I expect that would make 
the task even harder: presumably that's why Racket-on-Chez doesn't work 
that way.


But if linklets are as good as an abstraction as they seem to be for 
compiling code in the presence of modules and macros---all of the 
complications of 

Re: How did you handle making a GNU/Linux distribution?

[Christine Lemmer-Webber]

Philip McGrath  writes:

> Hi Sage,
>
> On 8/22/21 5:53 PM, Sage Gerard wrote:
>> Thanks for the detailed answer!
>> It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead
>> of
>> Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
>> someone already tried something like that?
>
> I haven't tried Xiden yet, and I haven't done any concrete work toward
> this (I have been working on managing Racket packages with Guix), but 
> Christine Lemmer-Webber had floated the idea at some point of trying
> to integrate Racket and Guile.
>
> IIRC, I think what she's had in mind was trying to make a Guile
> backend for Racket along the lines of the Chez Scheme backend (or the
> BC backend, or experimental backends like Pycket).

Yes that's what I had in mind :)

> As I said, I haven't actually tried any of this, but, as I've thought
> about what might be involved, there are two things that have struck me 
> as downsides:
>
>  1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
> Report)" (§6, p. 13) that, "If our task were to compile Racket to an
> existing target, then we would not have achieved such a high degree
> of compatibility. … we have taken the liberty of modifying Chez
> Scheme to make it an easier target for Racket."
>
> https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf
>
> Presumably a Racket-on-Guile project would face the same trade-off,
> where modifications to Guild, if Racket CS is a guide, could require
> hard work over many years, and lesser compatibility would make the
> result less useful.

At one point when I spoke to Matthew, he was very optimistic that the
work on porting on top of Chez would open Racket to running on top of
many other backends.  But yes... since there have been so many "custom"
modifications to Chez, it's easier to be skeptical about that these
days...

>  2. As you probably know, Racket programs can't generally use
> Chez Scheme implemented libraries, because Chez Scheme effectively
> is the "unsafe" layer of the Racket VM. For example, not all Racket
> procedures are Chez Scheme procedures, and Racket's continuations
> wrap Chez Scheme's to implement delimited and composable control,
> threads, parameters, full continuation marks, etc.
>
> For Racket CS, this isn't a great loss (there aren't so many
> Chez-specific libraries, and portable libraries can run in Racket's
> R6RS language), but, for a hypothetical Racket-on-Guile,
> bidirectional interoperability would be a big attraction: imagine
> Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
> working together.
>
> If I were going to work on this, I'd start by looking at having Racket
> and Guile coexist as siblings with interoperability through their FFIs 
> level. Even better, eventually you could compile Guile to Racket
> linklets, so the two could coexist in the same primitive module
> system. There would probably always need to be something along the
> lines of require/typed to interoperate between the languages, since
> Guile has mutable pairs and an unusual approach to falsehood and
> nullity to let Scheme's #f and '() coexist with Emacs List's nil. (See 
> https://www.gnu.org/software/guile/manual/html_node/Nil.html).
>
 I'm at the point where users are requesting a GNU/Linux distribution for
 Xiden, such that Racket is the primary language for day-to-day
 operation. I'm ignorant of the scope of work, and am unsure if I can do
 it alone.
>
> To me, at least, the scope of the work in creating a new GNU/Linux
> distribution seems daunting. My hope would be that bringing Racket and 
> Guile closer together would let most if not all of the effort be shared.
>
> Hope some of this is useful, and best of luck!
>
> -Philip

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

I'll answer your questions in a different thread to limit the noise in
this one. Look for the "Xiden/Guix design comparison" subject line
within the next 24h.

On 8/24/21 12:42 PM, Maxime Devos wrote:
> Sage Gerard schreef op ma 23-08-2021 om 18:24 [+]:
>> Thank you for the links!
>>
>>> I miss which problem Xiden is solving and how it does.
>> You are not the first to say so, and I'm happy to learn more about why.
>>
>> I'll try to explain in a different way here, so forgive the text wall.
>> I'll incorporate any feedback here to improve the white paper. I'd also like
>> to hear more about Guix's philosophy on what I'm going to talk about, because
>> I had reasons to end up with a different model.
> Maybe the publications at  are 
> informative.
>
>> --
>>
>> If we each use a program to distribute software, then those programs are 
>> probably
>> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses 
>> this by
>> decoupling the subjective elements of software distribution from the 
>> objective ones.
>>That means modeling software distribution as (partly) a semantics 
>> problem. Racket didn't
>>   do this with its own package managers. Which is ironic, because it's hard 
>> to guarantee
>>   that that an arbitrary Racket program will get the dependencies they mean 
>> to use.
>>
>>   I've written about this at length [1][2][3],
> About polyglot: packages in Guix can specify which version of a package they 
> use.
> E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a 
> specific
> version of 'autoconf' (2.13) and 'rust' (1.41)
>
>  (native-inputs
>   `(("autoconf" ,autoconf-2.13)
> ("automake" ,automake)
> ("llvm" ,llvm)   ;for llvm-objdump
> ("perl" ,perl)
> ("pkg-config" ,pkg-config)
> ("python" ,python-3)
> ("rust" ,rust-1.41)
> ("cargo" ,rust-1.41 "cargo")))
>
> Is something missing from Guix here?  About [3]: this doesn't seem a problem 
> in Guix,
> if you're referring to the ability to define multiple versions of a package 
> at the same
> time (it's an ‘official policy’ to usually try to only keep the latest 
> version of a
> package though, but channels can ignore this completely).
>
>>   and spoke about it last year [4]. Xiden has changed a bit since the speech 
>> to be more flexible, and it is no longer limited for use in Racket projects. 
>> Apologies if you see something that looks contradictory.
>> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
>>
>> Alice
>>
>> trusts SHA-384, but only if its implementation uses Keccak.
> SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
> SHA-3: a version of Keccak.  By definition, implementations of SHA-384
> have no reason to use Keccak.
>
>> defines "sha384" as the canonical name of the CHF, and treat variants like 
>> "SHA-384" as aliases.
>> delegates trust in artifact signatures to GPG subprocesses.
> Guix allows changing the hash algorithm used, search for ‘content-hash’ in 
> the guix manual.
> Changing the hash algorithm globally unavoidably requires modifying every 
> package definition
> though, but that could be automated.
>
>> downloads content from her employer's S3 buckets.
> The ‘official’ substitute servers can be replaced:
> "guix build --substitute-urls=http://s3.bucket.employer;
> (after authorising that substitute server).
>
>> unconditionally prefers cached installations, such that every defined 
>> artifact
>> has exactly one stored copy on disk.
> Coming from Guix, I don't know what this means.  Does this mean only one 
> version
> of a package can be in the store at the same time, and building a newer 
> version
> deletes the older version?  How does this compare with ‘content 
> deduplication’?
>
>> All dependents access the stored artifact
>> using symbolic links or Windows junctions.
>
>> prefers declaring versions with edition and revision names.
>> Bob trusts SHA-1,
> See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the 
> first place?
> (See 
> .)
>
>>   but only for artifacts from services authenticated by his operating 
>> system's certificates
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>
>  (source (origin
>(method url-fetch)
>(uri (string-append "mirror://gnu/hello/hello-" version
>".tar.gz"))
>(sha256 ; imagine this was SHA-1 instead.
> (base32
>  "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
>
> Bob got the origin specification from Xiden (or something like a Guix channel
> but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
> certificates’?  I mean, when you let your 

Re: How did you handle making a GNU/Linux distribution?

[Maxime Devos]

Sage Gerard schreef op ma 23-08-2021 om 18:24 [+]:
> Thank you for the links!
> 
> > I miss which problem Xiden is solving and how it does. 
> 
> You are not the first to say so, and I'm happy to learn more about why.
> 
> I'll try to explain in a different way here, so forgive the text wall.
> I'll incorporate any feedback here to improve the white paper. I'd also like
> to hear more about Guix's philosophy on what I'm going to talk about, because
> I had reasons to end up with a different model.

Maybe the publications at  are 
informative.

> --
> 
> If we each use a program to distribute software, then those programs are 
> probably
> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this 
> by
> decoupling the subjective elements of software distribution from the 
> objective ones.
>   That means modeling software distribution as (partly) a semantics problem. 
> Racket didn't
>  do this with its own package managers. Which is ironic, because it's hard to 
> guarantee
>  that that an arbitrary Racket program will get the dependencies they mean to 
> use.
>
>  I've written about this at length [1][2][3],

About polyglot: packages in Guix can specify which version of a package they 
use.
E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a specific
version of 'autoconf' (2.13) and 'rust' (1.41)

(native-inputs
 `(("autoconf" ,autoconf-2.13)
   ("automake" ,automake)
   ("llvm" ,llvm)   ;for llvm-objdump
   ("perl" ,perl)
   ("pkg-config" ,pkg-config)
   ("python" ,python-3)
   ("rust" ,rust-1.41)
   ("cargo" ,rust-1.41 "cargo")))

Is something missing from Guix here?  About [3]: this doesn't seem a problem in 
Guix,
if you're referring to the ability to define multiple versions of a package at 
the same
time (it's an ‘official policy’ to usually try to only keep the latest version 
of a
package though, but channels can ignore this completely).

>  and spoke about it last year [4]. Xiden has changed a bit since the speech 
> to be more flexible, and it is no longer limited for use in Racket projects. 
> Apologies if you see something that looks contradictory.
> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
> 
> Alice
> 
> trusts SHA-384, but only if its implementation uses Keccak.

SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
SHA-3: a version of Keccak.  By definition, implementations of SHA-384
have no reason to use Keccak.

> defines "sha384" as the canonical name of the CHF, and treat variants like 
> "SHA-384" as aliases.
> delegates trust in artifact signatures to GPG subprocesses.

Guix allows changing the hash algorithm used, search for ‘content-hash’ in the 
guix manual.
Changing the hash algorithm globally unavoidably requires modifying every 
package definition
though, but that could be automated.

> downloads content from her employer's S3 buckets.

The ‘official’ substitute servers can be replaced:
"guix build --substitute-urls=http://s3.bucket.employer;
(after authorising that substitute server).

> unconditionally prefers cached installations, such that every defined artifact
> has exactly one stored copy on disk.

Coming from Guix, I don't know what this means.  Does this mean only one version
of a package can be in the store at the same time, and building a newer version
deletes the older version?  How does this compare with ‘content deduplication’?

> All dependents access the stored artifact
> using symbolic links or Windows junctions.


> prefers declaring versions with edition and revision names.
> Bob trusts SHA-1,

See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the 
first place?
(See 
.)

>  but only for artifacts from services authenticated by his operating system's 
> certificates

This is rather vague to me, unless you're referring to substitutes.
What does this mean for origin specifications?

(source (origin
  (method url-fetch)
  (uri (string-append "mirror://gnu/hello/hello-" version
  ".tar.gz"))
  (sha256 ; imagine this was SHA-1 instead.
   (base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"

Bob got the origin specification from Xiden (or something like a Guix channel
but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
certificates’?  I mean, when you let your operating system build that origin
(via guix (or Xiden?), which you could consider part of the OS), it will check
that hash, and it will be considered valid (‘authenticated?’) if the hash 
matches.
There are no certificates at play here.

I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does that
provide to Bob?  (Unless Bob runs a substitute server on their 

Re: How did you handle making a GNU/Linux distribution?

[zimoun]

Hi Philip,

On Mon, 23 Aug 2021 at 13:30, Philip McGrath  wrote:
> On 8/23/21 11:38 AM, zimoun wrote:
>> The bootstrap path of Racket in Guix is not clear to me.  I miss if a
>> Racket interpreter or compiler in its binary format is used (as
>> Haskell for instance) or if all is compiled from source using tools
>> already bootstrapped.  Anyway, I miss what you would like bootstrap?
>> If you mean a trusted seed in order to start a package collection, you
>> could use the current Guix binaires---as a starting point.
>
> I wrote a long comment at the top of gnu/packages/racket.scm that 
> explains the current state of affairs. (For anyone unfamiliar with the 
> Guix source tree: 
> )

Oh sorry!  I did incorrectly my homework… although I remember the
discussion with Ludo when you submitted the patch set. :-)

> In brief, Racket is bootstrapped from source using only a C 
> compiler---including bootstrapping Racket's fork of Chez Scheme without 
> an existing Chez Scheme compiler---with one exception: the "expander" 
> subsystem (which implements the reader, module system, and macro 
> expander) is implemented in Racket, and the older C implementation was 
> completely replaced as of Racket 7.0. (One of the motivations was to fix 
> many problems with the old implementation.) For bootstrapping, an 
> intermediate representation as a "schemified" linklet is checked into 
> the Racket source repository.

Thanks for explaining.  All is clear.

> This impact is mitigated because, by design, the intermediate 
> representation is "human-readable and -editable Scheme code". 
> Anecdotally, when I read diffs of changes to the Racket repository, I 
> sometimes read tens of lines of code before I realize that I'm looking 
> at the generated version of the file, not the source. So the expander is 
> not bootstrappable in the most rigorous sense, but it is readily auditable.
>
> If someone wanted to tie the knot, the task would be to use an existing 
> bootstappable language to write a minimal expander that can expand the 
> canonical Racket implementation: if your output is identical to the 
> checked-in generated files, you've succeeded. (Using Guile could avoid 
> the heavy maintenance burden of the old C expander implementation, and 
> indeed you could write just enough of an adapter to load the Racket 
> source files into Guile, which is the technique Racket uses to bootstrap 
> the Chez Scheme compiler.) I expect Racket would welcome a contribution 
> along those lines; it just hasn't been a priority for anyone yet.

Guile everywhere! ;-)

Thanks again the details.

Cheers,
simon

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

Sounds like a great ROI, thank you for the suggestion!

 Original Message 
On Aug 23, 2021, 3:23 PM, Ryan Prior wrote:

>> [I want to bootstrap] all binaries related to creating a GNU/Linux 
>> distribution, such that I can reproduce an exact OS, Racket installation, 
>> and Xiden instance. I want a trusted initial state on GNU/Linux.
>
> Seems like the easy path for you is to package Xiden for Guix, and then 
> construct a system definition (using Guix System) that includes Racket and 
> Xiden in its default set of packages, and finally use Guix to build 
> system-images for that target system.
>
> To do this does not require that you can run Racket on the Guile VM. The 
> target system-image doesn't even need to have Guile or Guix installed on it 
> at all; it can just serve as a bootstrapped base and you can build from there 
> to your final desired state.
>
> I'm happy to explain or assist with any part of this process, and I imagine 
> others on this list would be as well.
>
> Designing your own bootstrap and rewriting everything in Racket, or waiting 
> for a suitable Racket-on-Guile implementation, do not seem necessary at all 
> to reach your stated goals. Hope this helps!

Re: How did you handle making a GNU/Linux distribution?

[Ryan Prior]

> [I want to bootstrap] all binaries related to creating a GNU/Linux 
> distribution, such that I can reproduce an exact OS, Racket installation, and 
> Xiden instance. I want a trusted initial state on GNU/Linux.

Seems like the easy path for you is to package Xiden for Guix, and then 
construct a system definition (using Guix System) that includes Racket and 
Xiden in its default set of packages, and finally use Guix to build 
system-images for that target system.

To do this does not require that you can run Racket on the Guile VM. The target 
system-image doesn't even need to have Guile or Guix installed on it at all; it 
can just serve as a bootstrapped base and you can build from there to your 
final desired state.

I'm happy to explain or assist with any part of this process, and I imagine 
others on this list would be as well.

Designing your own bootstrap and rewriting everything in Racket, or waiting for 
a suitable Racket-on-Guile implementation, do not seem necessary at all to 
reach your stated goals. Hope this helps!

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

Thank you for the links!

> I miss which problem Xiden is solving and how it does.

You are not the first to say so, and I'm happy to learn more about why.

I'll try to explain in a different way here, so forgive the text wall. I'll 
incorporate any feedback here to improve the white paper. I'd also like to hear 
more about Guix's philosophy on what I'm going to talk about, because I had 
reasons to end up with a different model.

--

If we each use a program to distribute software, then those programs are 
probably incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden 
addresses this by decoupling the subjective elements of software distribution 
from the objective ones. That means modeling software distribution as (partly) 
a semantics problem. Racket didn't do this with its own package managers. Which 
is ironic, because it's hard to guarantee that that an arbitrary Racket program 
will get the dependencies they mean to use. I've written about this at length 
[1][2][3], and spoke about it last year [4]. Xiden has changed a bit since the 
speech to be more flexible, and it is no longer limited for use in Racket 
projects. Apologies if you see something that looks contradictory.

What does that mean? Let's say Alice and Bob each write a Xiden launcher.

Alice

- trusts SHA-384, but only if its implementation uses Keccak.
- defines "sha384" as the canonical name of the CHF, and treat variants like 
"SHA-384" as aliases.
- delegates trust in artifact signatures to GPG subprocesses.
- downloads content from her employer's S3 buckets.
- unconditionally prefers cached installations, such that every defined 
artifact has exactly one stored copy on disk. All dependents access the stored 
artifact using symbolic links or Windows junctions.
- prefers declaring versions with edition and revision names.

Bob

- trusts SHA-1, but only for artifacts from services authenticated by his 
operating system's certificates.
- trusts digest creation and signature verification as implemented by his 
host's dynamically--linked `libcrypto` library for use in the same process.
- downloads content from GitHub packages
- unconditionally prefers SxS installations, such that packages cannot conflict 
in a shared namespace at the cost of defeating an internal cache.
- prefers Semantic Versions

These preferences are valid in Xiden's DSL for writing launchers. The 
invariants for each launcher can differ as much as Cargo and Vortex differ. 
What I wanted to do was allow users to declare their own invariants explicitly, 
but only when those invariants are wanted for subjective reasons. I can't just 
write an overly-opinionated tool in this space because opinions have 
shelf-lives, and they can't interrupt our ability to get the content we need on 
demand, with our own safety guarentees.

Xiden launchers have advantages over shell scripts in that we can both still 
download and install dependencies from the same servers, and create 
reproducible builds in terms of the same (bit-identical) state on disk. We just 
differ on details that shouldn't impact how we work. It also helps us patch 
holes faster, since if (say) a catalog maintainer tells us that a CHF was 
compromised, we can immediately revoke trust in the CHF independently in our 
own clients. The package definitions are all expressed in terms of what the 
launchers allow.

So what problem does this solve? I'm trying to preserve an element of walk-away 
power in any tech community. Maybe your community goes in a questionable 
direction. Maybe your desired software is in a different ecosystem. Maybe the 
maintainers are abusive jerks. Maybe you have a `leftpad`/`event-stream` 
incident and the developer is unable/unwilling to patch the problem. You'd 
probably want a way to escape to a new community or distribution model without 
giving up content or doing complex integration work. If Xiden can do that, 
users don't have to depend on the same middlemen to share work. This is also 
better for developers because it doesn't oblige them to do everything their 
users want, while knowing users will adapt to inevitable distribution problems. 
Everyone's consent becomes explicit, such that Xiden's "real" invariant is that 
consent between any two parties be mutually compatible. I hope to improve user 
freedom from this angle, and my approach is biased by my experience with 
Racket's limitations in this space.

The subjective parts I talk about go further into details like what name 
canons, versioning schemes, trusted certificate chains, and even specific 
approaches to an exact diamond dependency. The objective parts include 
integrity checking, signature verification, safety limits, automatic dependency 
resolution, patches, protocols, and other concerns that a decent package 
manager normally cares about. By keeping these separate, all of the below 
features become cheap to write, without sacrificing trust & safety in the 
abstract.

- Download archived GitHub repositories 

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

Hi Simon,

On 8/23/21 11:38 AM, zimoun wrote:

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.


I wrote a long comment at the top of gnu/packages/racket.scm that 
explains the current state of affairs. (For anyone unfamiliar with the 
Guix source tree: 
)


In brief, Racket is bootstrapped from source using only a C 
compiler---including bootstrapping Racket's fork of Chez Scheme without 
an existing Chez Scheme compiler---with one exception: the "expander" 
subsystem (which implements the reader, module system, and macro 
expander) is implemented in Racket, and the older C implementation was 
completely replaced as of Racket 7.0. (One of the motivations was to fix 
many problems with the old implementation.) For bootstrapping, an 
intermediate representation as a "schemified" linklet is checked into 
the Racket source repository.


This impact is mitigated because, by design, the intermediate 
representation is "human-readable and -editable Scheme code". 
Anecdotally, when I read diffs of changes to the Racket repository, I 
sometimes read tens of lines of code before I realize that I'm looking 
at the generated version of the file, not the source. So the expander is 
not bootstrappable in the most rigorous sense, but it is readily auditable.


If someone wanted to tie the knot, the task would be to use an existing 
bootstappable language to write a minimal expander that can expand the 
canonical Racket implementation: if your output is identical to the 
checked-in generated files, you've succeeded. (Using Guile could avoid 
the heavy maintenance burden of the old C expander implementation, and 
indeed you could write just enough of an adapter to load the Racket 
source files into Guile, which is the technique Racket uses to bootstrap 
the Chez Scheme compiler.) I expect Racket would welcome a contribution 
along those lines; it just hasn't been a priority for anyone yet.


-Philip

Re: How did you handle making a GNU/Linux distribution?

[zimoun]

Hi Sage,

On Sat, 21 Aug 2021 at 22:18, Sage Gerard  wrote:

> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. It is also free software under the GPLv3. The source is
> available at zyrolasting/xiden on Github, pending migration to a new host.

Oh thanks! :-)

> I'm at the point where users are requesting a GNU/Linux distribution for
> Xiden, such that Racket is the primary language for day-to-day
> operation. I'm ignorant of the scope of work, and am unsure if I can do
> it alone.  My understanding is that you had to bootstrap your entire
> toolchain and address Ken Thompson's compiler hack from a different
> angle. Is that right?

Well, from my understanding, Guix (package manager) and Guix System
(OS on the top) are not about bootstrap.  If you give a look at one of
the first public presentations:



then bootstrap the whole toolchain was not on topic (at least not
explicitly listed).  Reducing the bootstrap seed came later and
gradually, for instance see:




> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. I'm bad at both of those things, but if there is any
> opportunity to collaborate on implementation details based on Xiden's
> progress, I am happy to give back.

I have read the White Paper and the doc.  I am sorry but I miss which
problem Xiden is solving and how it does.  For instance, I do not
understand what a "launcher" is and what its job is.  Maybe Xiden is a
"workflow" engine?

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.

Well, from my understanding, as Philip said, compile Guile to Racket
linklets seems something promising. :-)



All the best,
simon

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

> I looked at the Xiden documentation and source code a bit,
> and I couldn't find any actual package definitions, or how to make
> any package definitions.  Where are the 'coreutils' and some text editors
> like 'nano' or 'emacs' or 'vi' packaged for example? [...] Guix and Xiden 
> could copy code and package definitions from each other
> (after making some adaptations for Guile vs Racket and different interfaces),
> though I wouldn't know how realistic that is in advancce.

I put these bits of your email together because I think they are related.

The system-level examples you are looking don't exist because Xiden is not 
fully bootstrapped. What I can do in the short-term is write a launcher that 
follows LFS, then verifies any artifacts used for the purpose. I wasn't 
satisfied with publishing that approach since it involves implicit trust in 
system level dependencies, but you can lean on the zero-trust default config 
and use ACLs on the small attack surface to mitigate risks there.

Until a Guile/Racket bridge is formed as Phillip described, or until GNU Mes is 
adapted to target Racket, I do not feel comfortable distributing system-level 
binaries using Xiden. It's easy for a user to write a launcher to use them, 
though. That's why I put so much focus on writing launchers that define a 
contract between your chosen data sources and your subjective expression of 
dependencies. Package definitions are subservient to this approach, because as 
an abstraction, they sit between the subsystem that fetches and verifies data, 
and the subsystem that understands what a user means when they ask to install a 
dependency. The latter is the launcher's job, and I often use one to generate 
package definitions, find integrity information, trust certs, etc.

The end of the guide directs you to example launchers, which set rules for how 
package definitions work in the context of those launchers. The early examples 
are just toy demos, but later ones handle things like a particular self-hosting 
approach, and managing multiple versions of Racket installations.

> Hyperlinks in the manual aren't working:

Let me know which ones in a Github issue? The Racket community shares a Racket 
installation for online docs, and I occasionally run into problems with the 
generated links. Sometimes its CI related, sometimes its because I didn't type 
a Scribble form exactly right. The docs are all present at least. Entry point 
links follow for convenience.

https://docs.racket-lang.org/xiden-guide/index.html
https://docs.racket-lang.org/xiden-white-paper/index.html
https://docs.racket-lang.org/xiden-reference/index.html

> Is this GPL-version-3-only or GPL-version-3-or-later?

Currently 3. I didn't add "or later" (yet) because I don't understand the 
social and legal consequences of doing so, even when new license versions are 
compatible. Happy to learn more here.

Re: How did you handle making a GNU/Linux distribution?

[Maxime Devos]

Sage Gerard schreef op za 21-08-2021 om 16:43 [+]:
> Hi folks,
> 
> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. [...]

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. [...]

I looked at the Xiden documentation and source code a bit,
and I couldn't find any actual package definitions, or how to make
any package definitions.  Where are the 'coreutils' and some text editors
like 'nano' or 'emacs' or 'vi' packaged for example?

Hyperlinks in the manual aren't working:

https://download.racket-lang.org/releases/8.2/doc/local-redirect/index.html?doc=xiden-reference=index.html=8.2

reads

This page is intended to redirect to the result of a search request. Since 
you’re reading this, it seems that the redirection did not work.

I would recommend working on the documentation.

> It is also free software under the GPLv3

Is this GPL-version-3-only or GPL-version-3-or-later?  If it's the latter,
Guix and Xiden could copy code and package definitions from each other
(after making some adaptations for Guile vs Racket and different interfaces),
though I wouldn't know how realistic that is in advancce.

Greetings,
Maxxime.


signature.asc
Description: This is a digitally signed message part

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

Hi Sage,

On 8/22/21 5:53 PM, Sage Gerard wrote:

Thanks for the detailed answer!

It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of
Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
someone already tried something like that?


I haven't tried Xiden yet, and I haven't done any concrete work toward 
this (I have been working on managing Racket packages with Guix), but 
Christine Lemmer-Webber had floated the idea at some point of trying to 
integrate Racket and Guile.


IIRC, I think what she's had in mind was trying to make a Guile backend 
for Racket along the lines of the Chez Scheme backend (or the BC 
backend, or experimental backends like Pycket).


As I said, I haven't actually tried any of this, but, as I've thought 
about what might be involved, there are two things that have struck me 
as downsides:


 1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
Report)" (§6, p. 13) that, "If our task were to compile Racket to an
existing target, then we would not have achieved such a high degree
of compatibility. … we have taken the liberty of modifying Chez
Scheme to make it an easier target for Racket."

https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf

Presumably a Racket-on-Guile project would face the same trade-off,
where modifications to Guild, if Racket CS is a guide, could require
hard work over many years, and lesser compatibility would make the
result less useful.

 2. As you probably know, Racket programs can't generally use
Chez Scheme implemented libraries, because Chez Scheme effectively
is the "unsafe" layer of the Racket VM. For example, not all Racket
procedures are Chez Scheme procedures, and Racket's continuations
wrap Chez Scheme's to implement delimited and composable control,
threads, parameters, full continuation marks, etc.

For Racket CS, this isn't a great loss (there aren't so many
Chez-specific libraries, and portable libraries can run in Racket's
R6RS language), but, for a hypothetical Racket-on-Guile,
bidirectional interoperability would be a big attraction: imagine
Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
working together.

If I were going to work on this, I'd start by looking at having Racket 
and Guile coexist as siblings with interoperability through their FFIs 
level. Even better, eventually you could compile Guile to Racket 
linklets, so the two could coexist in the same primitive module system. 
There would probably always need to be something along the lines of 
require/typed to interoperate between the languages, since Guile has 
mutable pairs and an unusual approach to falsehood and nullity to let 
Scheme's #f and '() coexist with Emacs List's nil. (See 
https://www.gnu.org/software/guile/manual/html_node/Nil.html).



I'm at the point where users are requesting a GNU/Linux distribution for
Xiden, such that Racket is the primary language for day-to-day
operation. I'm ignorant of the scope of work, and am unsure if I can do
it alone.


To me, at least, the scope of the work in creating a new GNU/Linux 
distribution seems daunting. My hope would be that bringing Racket and 
Guile closer together would let most if not all of the effort be shared.


Hope some of this is useful, and best of luck!

-Philip

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

Thanks for the detailed answer!

It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of
Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
someone already tried something like that?

On 8/21/21 5:18 PM, Leo Famulari wrote:
> On Sat, Aug 21, 2021 at 04:43:34PM +, Sage Gerard wrote:
>> My name is Sage. I wrote a cross-platform Guix-like package manager
>> called Xiden. It applies functional package management to the Racket
>> ecosystem. It is also free software under the GPLv3. The source is
>> available at zyrolasting/xiden on Github, pending migration to a new host.
> Hi! Cool!
>
>> I'm at the point where users are requesting a GNU/Linux distribution for
>> Xiden, such that Racket is the primary language for day-to-day
>> operation. I'm ignorant of the scope of work, and am unsure if I can do
>> it alone.  My understanding is that you had to bootstrap your entire
>> toolchain and address Ken Thompson's compiler hack from a different
>> angle. Is that right?
> At the time that Guix was founded, it was not considered possible to
> bootstrap a GNU/Linux system from source; on which OS would you perform
> the first compilation? How would you bootstrap that OS? Ad infinitum.
> Stage0 and mes and friends are fixing that now. [0]
>
> So, some binary "seeds" were created, which are compiled binaries of
> core tools such as GCC and binutils, and the system was based on those
> seeds. And we gave instructions for reproducing these seeds; you could
> use any commonly available distro as a host. Maybe that sounds icky in
> 2021, since the situation has improved radically, but it's how every OS
> was created since decades ago [1], except that most operating systems (and
> even language ecosystems) did not keep track of their seeds or give
> instructions for reproducing them. So Guix was better than all the rest
> from its beginning :)
>
> https://guix.gnu.org/manual/en/html_node/Bootstrapping.html
>
> I was not involved at that stage of Guix. I'm sure others can give more
> detailed answers.
>
>> I'd greatly appreciate learning how you all built Guix's GNU/Linux
>> distribution so that I can prepare a realistic roadmap and recruit help
>> where necessary. I'm bad at both of those things, but if there is any
>> opportunity to collaborate on implementation details based on Xiden's
>> progress, I am happy to give back.
> [0] https://bootstrappable.org/
> https://bootstrappable.org/projects/mes.html
> https://www.gnu.org/software/mes/
> Also visit the #bootstrappable channel on Libera.chat
>
> [1] It would be interesting to know the last truly bootstrapped OS

Re: How did you handle making a GNU/Linux distribution?

[Leo Famulari]

On Sat, Aug 21, 2021 at 04:43:34PM +, Sage Gerard wrote:
> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. It is also free software under the GPLv3. The source is
> available at zyrolasting/xiden on Github, pending migration to a new host.

Hi! Cool!

> I'm at the point where users are requesting a GNU/Linux distribution for
> Xiden, such that Racket is the primary language for day-to-day
> operation. I'm ignorant of the scope of work, and am unsure if I can do
> it alone.  My understanding is that you had to bootstrap your entire
> toolchain and address Ken Thompson's compiler hack from a different
> angle. Is that right?

At the time that Guix was founded, it was not considered possible to
bootstrap a GNU/Linux system from source; on which OS would you perform
the first compilation? How would you bootstrap that OS? Ad infinitum.
Stage0 and mes and friends are fixing that now. [0]

So, some binary "seeds" were created, which are compiled binaries of
core tools such as GCC and binutils, and the system was based on those
seeds. And we gave instructions for reproducing these seeds; you could
use any commonly available distro as a host. Maybe that sounds icky in
2021, since the situation has improved radically, but it's how every OS
was created since decades ago [1], except that most operating systems (and
even language ecosystems) did not keep track of their seeds or give
instructions for reproducing them. So Guix was better than all the rest
from its beginning :)

https://guix.gnu.org/manual/en/html_node/Bootstrapping.html

I was not involved at that stage of Guix. I'm sure others can give more
detailed answers.

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. I'm bad at both of those things, but if there is any
> opportunity to collaborate on implementation details based on Xiden's
> progress, I am happy to give back.

[0] https://bootstrappable.org/
https://bootstrappable.org/projects/mes.html
https://www.gnu.org/software/mes/
Also visit the #bootstrappable channel on Libera.chat

[1] It would be interesting to know the last truly bootstrapped OS