Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

Sounds like a great ROI, thank you for the suggestion!

 Original Message 
On Aug 23, 2021, 3:23 PM, Ryan Prior wrote:

>> [I want to bootstrap] all binaries related to creating a GNU/Linux 
>> distribution, such that I can reproduce an exact OS, Racket installation, 
>> and Xiden instance. I want a trusted initial state on GNU/Linux.
>
> Seems like the easy path for you is to package Xiden for Guix, and then 
> construct a system definition (using Guix System) that includes Racket and 
> Xiden in its default set of packages, and finally use Guix to build 
> system-images for that target system.
>
> To do this does not require that you can run Racket on the Guile VM. The 
> target system-image doesn't even need to have Guile or Guix installed on it 
> at all; it can just serve as a bootstrapped base and you can build from there 
> to your final desired state.
>
> I'm happy to explain or assist with any part of this process, and I imagine 
> others on this list would be as well.
>
> Designing your own bootstrap and rewriting everything in Racket, or waiting 
> for a suitable Racket-on-Guile implementation, do not seem necessary at all 
> to reach your stated goals. Hope this helps!

Re: How did you handle making a GNU/Linux distribution?

[Ryan Prior]

> [I want to bootstrap] all binaries related to creating a GNU/Linux 
> distribution, such that I can reproduce an exact OS, Racket installation, and 
> Xiden instance. I want a trusted initial state on GNU/Linux.

Seems like the easy path for you is to package Xiden for Guix, and then 
construct a system definition (using Guix System) that includes Racket and 
Xiden in its default set of packages, and finally use Guix to build 
system-images for that target system.

To do this does not require that you can run Racket on the Guile VM. The target 
system-image doesn't even need to have Guile or Guix installed on it at all; it 
can just serve as a bootstrapped base and you can build from there to your 
final desired state.

I'm happy to explain or assist with any part of this process, and I imagine 
others on this list would be as well.

Designing your own bootstrap and rewriting everything in Racket, or waiting for 
a suitable Racket-on-Guile implementation, do not seem necessary at all to 
reach your stated goals. Hope this helps!

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

Thank you for the links!

> I miss which problem Xiden is solving and how it does.

You are not the first to say so, and I'm happy to learn more about why.

I'll try to explain in a different way here, so forgive the text wall. I'll 
incorporate any feedback here to improve the white paper. I'd also like to hear 
more about Guix's philosophy on what I'm going to talk about, because I had 
reasons to end up with a different model.

--

If we each use a program to distribute software, then those programs are 
probably incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden 
addresses this by decoupling the subjective elements of software distribution 
from the objective ones. That means modeling software distribution as (partly) 
a semantics problem. Racket didn't do this with its own package managers. Which 
is ironic, because it's hard to guarantee that that an arbitrary Racket program 
will get the dependencies they mean to use. I've written about this at length 
[1][2][3], and spoke about it last year [4]. Xiden has changed a bit since the 
speech to be more flexible, and it is no longer limited for use in Racket 
projects. Apologies if you see something that looks contradictory.

What does that mean? Let's say Alice and Bob each write a Xiden launcher.

Alice

- trusts SHA-384, but only if its implementation uses Keccak.
- defines "sha384" as the canonical name of the CHF, and treat variants like 
"SHA-384" as aliases.
- delegates trust in artifact signatures to GPG subprocesses.
- downloads content from her employer's S3 buckets.
- unconditionally prefers cached installations, such that every defined 
artifact has exactly one stored copy on disk. All dependents access the stored 
artifact using symbolic links or Windows junctions.
- prefers declaring versions with edition and revision names.

Bob

- trusts SHA-1, but only for artifacts from services authenticated by his 
operating system's certificates.
- trusts digest creation and signature verification as implemented by his 
host's dynamically--linked `libcrypto` library for use in the same process.
- downloads content from GitHub packages
- unconditionally prefers SxS installations, such that packages cannot conflict 
in a shared namespace at the cost of defeating an internal cache.
- prefers Semantic Versions

These preferences are valid in Xiden's DSL for writing launchers. The 
invariants for each launcher can differ as much as Cargo and Vortex differ. 
What I wanted to do was allow users to declare their own invariants explicitly, 
but only when those invariants are wanted for subjective reasons. I can't just 
write an overly-opinionated tool in this space because opinions have 
shelf-lives, and they can't interrupt our ability to get the content we need on 
demand, with our own safety guarentees.

Xiden launchers have advantages over shell scripts in that we can both still 
download and install dependencies from the same servers, and create 
reproducible builds in terms of the same (bit-identical) state on disk. We just 
differ on details that shouldn't impact how we work. It also helps us patch 
holes faster, since if (say) a catalog maintainer tells us that a CHF was 
compromised, we can immediately revoke trust in the CHF independently in our 
own clients. The package definitions are all expressed in terms of what the 
launchers allow.

So what problem does this solve? I'm trying to preserve an element of walk-away 
power in any tech community. Maybe your community goes in a questionable 
direction. Maybe your desired software is in a different ecosystem. Maybe the 
maintainers are abusive jerks. Maybe you have a `leftpad`/`event-stream` 
incident and the developer is unable/unwilling to patch the problem. You'd 
probably want a way to escape to a new community or distribution model without 
giving up content or doing complex integration work. If Xiden can do that, 
users don't have to depend on the same middlemen to share work. This is also 
better for developers because it doesn't oblige them to do everything their 
users want, while knowing users will adapt to inevitable distribution problems. 
Everyone's consent becomes explicit, such that Xiden's "real" invariant is that 
consent between any two parties be mutually compatible. I hope to improve user 
freedom from this angle, and my approach is biased by my experience with 
Racket's limitations in this space.

The subjective parts I talk about go further into details like what name 
canons, versioning schemes, trusted certificate chains, and even specific 
approaches to an exact diamond dependency. The objective parts include 
integrity checking, signature verification, safety limits, automatic dependency 
resolution, patches, protocols, and other concerns that a decent package 
manager normally cares about. By keeping these separate, all of the below 
features become cheap to write, without sacrificing trust & safety in the 
abstract.

- Download archived GitHub repositories (sou

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

Hi Simon,

On 8/23/21 11:38 AM, zimoun wrote:

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.


I wrote a long comment at the top of gnu/packages/racket.scm that 
explains the current state of affairs. (For anyone unfamiliar with the 
Guix source tree: 
)


In brief, Racket is bootstrapped from source using only a C 
compiler---including bootstrapping Racket's fork of Chez Scheme without 
an existing Chez Scheme compiler---with one exception: the "expander" 
subsystem (which implements the reader, module system, and macro 
expander) is implemented in Racket, and the older C implementation was 
completely replaced as of Racket 7.0. (One of the motivations was to fix 
many problems with the old implementation.) For bootstrapping, an 
intermediate representation as a "schemified" linklet is checked into 
the Racket source repository.


This impact is mitigated because, by design, the intermediate 
representation is "human-readable and -editable Scheme code". 
Anecdotally, when I read diffs of changes to the Racket repository, I 
sometimes read tens of lines of code before I realize that I'm looking 
at the generated version of the file, not the source. So the expander is 
not bootstrappable in the most rigorous sense, but it is readily auditable.


If someone wanted to tie the knot, the task would be to use an existing 
bootstappable language to write a minimal expander that can expand the 
canonical Racket implementation: if your output is identical to the 
checked-in generated files, you've succeeded. (Using Guile could avoid 
the heavy maintenance burden of the old C expander implementation, and 
indeed you could write just enough of an adapter to load the Racket 
source files into Guile, which is the technique Racket uses to bootstrap 
the Chez Scheme compiler.) I expect Racket would welcome a contribution 
along those lines; it just hasn't been a priority for anyone yet.


-Philip

Re: How did you handle making a GNU/Linux distribution?

[zimoun]

Hi Sage,

On Sat, 21 Aug 2021 at 22:18, Sage Gerard  wrote:

> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. It is also free software under the GPLv3. The source is
> available at zyrolasting/xiden on Github, pending migration to a new host.

Oh thanks! :-)

> I'm at the point where users are requesting a GNU/Linux distribution for
> Xiden, such that Racket is the primary language for day-to-day
> operation. I'm ignorant of the scope of work, and am unsure if I can do
> it alone.  My understanding is that you had to bootstrap your entire
> toolchain and address Ken Thompson's compiler hack from a different
> angle. Is that right?

Well, from my understanding, Guix (package manager) and Guix System
(OS on the top) are not about bootstrap.  If you give a look at one of
the first public presentations:



then bootstrap the whole toolchain was not on topic (at least not
explicitly listed).  Reducing the bootstrap seed came later and
gradually, for instance see:




> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. I'm bad at both of those things, but if there is any
> opportunity to collaborate on implementation details based on Xiden's
> progress, I am happy to give back.

I have read the White Paper and the doc.  I am sorry but I miss which
problem Xiden is solving and how it does.  For instance, I do not
understand what a "launcher" is and what its job is.  Maybe Xiden is a
"workflow" engine?

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.

Well, from my understanding, as Philip said, compile Guile to Racket
linklets seems something promising. :-)



All the best,
simon

Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard]

> I looked at the Xiden documentation and source code a bit,
> and I couldn't find any actual package definitions, or how to make
> any package definitions.  Where are the 'coreutils' and some text editors
> like 'nano' or 'emacs' or 'vi' packaged for example? [...] Guix and Xiden 
> could copy code and package definitions from each other
> (after making some adaptations for Guile vs Racket and different interfaces),
> though I wouldn't know how realistic that is in advancce.

I put these bits of your email together because I think they are related.

The system-level examples you are looking don't exist because Xiden is not 
fully bootstrapped. What I can do in the short-term is write a launcher that 
follows LFS, then verifies any artifacts used for the purpose. I wasn't 
satisfied with publishing that approach since it involves implicit trust in 
system level dependencies, but you can lean on the zero-trust default config 
and use ACLs on the small attack surface to mitigate risks there.

Until a Guile/Racket bridge is formed as Phillip described, or until GNU Mes is 
adapted to target Racket, I do not feel comfortable distributing system-level 
binaries using Xiden. It's easy for a user to write a launcher to use them, 
though. That's why I put so much focus on writing launchers that define a 
contract between your chosen data sources and your subjective expression of 
dependencies. Package definitions are subservient to this approach, because as 
an abstraction, they sit between the subsystem that fetches and verifies data, 
and the subsystem that understands what a user means when they ask to install a 
dependency. The latter is the launcher's job, and I often use one to generate 
package definitions, find integrity information, trust certs, etc.

The end of the guide directs you to example launchers, which set rules for how 
package definitions work in the context of those launchers. The early examples 
are just toy demos, but later ones handle things like a particular self-hosting 
approach, and managing multiple versions of Racket installations.

> Hyperlinks in the manual aren't working:

Let me know which ones in a Github issue? The Racket community shares a Racket 
installation for online docs, and I occasionally run into problems with the 
generated links. Sometimes its CI related, sometimes its because I didn't type 
a Scribble form exactly right. The docs are all present at least. Entry point 
links follow for convenience.

https://docs.racket-lang.org/xiden-guide/index.html
https://docs.racket-lang.org/xiden-white-paper/index.html
https://docs.racket-lang.org/xiden-reference/index.html

> Is this GPL-version-3-only or GPL-version-3-or-later?

Currently 3. I didn't add "or later" (yet) because I don't understand the 
social and legal consequences of doing so, even when new license versions are 
compatible. Happy to learn more here.

Re: How did you handle making a GNU/Linux distribution?

[Maxime Devos]

Sage Gerard schreef op za 21-08-2021 om 16:43 [+]:
> Hi folks,
> 
> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. [...]

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. [...]

I looked at the Xiden documentation and source code a bit,
and I couldn't find any actual package definitions, or how to make
any package definitions.  Where are the 'coreutils' and some text editors
like 'nano' or 'emacs' or 'vi' packaged for example?

Hyperlinks in the manual aren't working:

https://download.racket-lang.org/releases/8.2/doc/local-redirect/index.html?doc=xiden-reference&rel=index.html&version=8.2

reads

This page is intended to redirect to the result of a search request. Since 
you’re reading this, it seems that the redirection did not work.

I would recommend working on the documentation.

> It is also free software under the GPLv3

Is this GPL-version-3-only or GPL-version-3-or-later?  If it's the latter,
Guix and Xiden could copy code and package definitions from each other
(after making some adaptations for Guile vs Racket and different interfaces),
though I wouldn't know how realistic that is in advancce.

Greetings,
Maxxime.


signature.asc
Description: This is a digitally signed message part