Re: [GSoC] Porting GuixSD to GNU Hurd draft
Hi, Quoting Manolis Ragkousis (2016-03-24 14:55:31) > On 03/24/16 15:22, Ludovic Courtès wrote: > >>> The main question is whether you should implement build isolation in > >>> guix-daemon, in which case that would leave little time for the GuixSD > >>> parts. I think I would rather let you focus on the GuixSD stuff as you > >>> wrote, but I’d like to hear what the Hurd folks think. > >> > >> I consider isolation more important. > > > > OK. > > Isolation first it is then. > > > So, Manolis, what about reframing the agenda such that porting > > guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the > > programming effort), followed by GuixSD stuff? > > > > Current objectives then: > 1) Achieve build isolation in the daemon on the Hurd. > 2) Modify Guix so it can produce a working image, while isolating any > cases of Linux assumptions. > 3) Boot to GuixSD Well, I didn't ment to stomp over your priorities like that, it is just that I personally consider it more important and from my point of view it should be easier to achieve. Justus
Re: [GSoC] Porting GuixSD to GNU Hurd draft
Hey On 03/24/16 15:22, Ludovic Courtès wrote: > So, let’s say PID isolation will be optional, and we can always adjust > later on. Sounds good? > > Manolis, make sure to read about how the various Hurd servers provides > these parts of POSIX personality: file systems, UIDs, PIDs, networking, > and so on. Already started. :-) > As far as code integration code, I think we won’t bother syncing this > work with nix-daemon; guix-daemon has already diverged, and for instance > it does not have OS X sandbox support. > > You’ll have to arrange to have the Hurd-specific bits in a separate > file, so that ‘#ifdef HURD’ are not scattered all over the place. > > This is C(++) as you know. WDYT, Manolis? I have to start studying the daemon's code more. From what I know the part that handles builds is in libstore/build.cc. I will start there. I think I can do it. >>> The main question is whether you should implement build isolation in >>> guix-daemon, in which case that would leave little time for the GuixSD >>> parts. I think I would rather let you focus on the GuixSD stuff as you >>> wrote, but I’d like to hear what the Hurd folks think. >> >> I consider isolation more important. > > OK. Isolation first it is then. > So, Manolis, what about reframing the agenda such that porting > guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the > programming effort), followed by GuixSD stuff? > Current objectives then: 1) Achieve build isolation in the daemon on the Hurd. 2) Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions. 3) Boot to GuixSD Ludo, Justus do you agree with this? Manolis
Re: [GSoC] Porting GuixSD to GNU Hurd draft
Hi Justus, Justus Winter <4win...@informatik.uni-hamburg.de> skribis: > Quoting Ludovic Courtès (2016-03-23 14:40:38) >> > 2. The Project >> > >> > The project consists of four main stages >> > >> > 1. Modify Guix so it will be able to create and mount the file-system >> > needed to boot into a system with Hurd at its core. >> > 2. Modify Guix so it can produce a working image, while isolating any >> > cases of Linux assumptions. >> > 3. Successfully boot into one such system using GNU Shepherd with pid 1. >> > 4. Modify the new Guix system to take advantage of Hurd specific >> > mechanisms. > > For me, 4. is the most important bit, so we can build packages in > isolation. I guess you mean isolation in guix-daemon. >> This is more important, and already non-trivial work. If libc provided >> ‘mount’ with support for MS_BIND (which I think it should), it would >> help a bit, but there’s still the problem of the other Linux name spaces >> that are used (the CLONE_NEW* clone(2) flags.) >> >> Thus it may make more sense to write this functionality in guix-daemon >> using directly the Hurd interfaces. Separate PID name spaces, UID name >> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s >> “just” a matter of giving the child process ports to separate proc, >> auth, etc. translators. >> >> In itself this is also a bit of work. I wonder what the Hurd folks >> think about priorities here? > > I'd go for specializing guix-daemon over trying too hard to implement > Linux-compatible interfaces in the libc. So this would become maybe half of the GSoC coding part maybe. WDYT? > I consider the filesystem-isolation part easy, UID isolation > relatively easy, PID isolation is a bit tricky. Currently one cannot > simply start another proc server and expect it to work as it needs > privileged kernel interfaces. I created an RPC to allow nested proc > servers for unprivileged subhurds. That should do the trick, it is > merely a matter of making it actually work I guess. “Merely”, hmm… :-) So, let’s say PID isolation will be optional, and we can always adjust later on. Sounds good? Manolis, make sure to read about how the various Hurd servers provides these parts of POSIX personality: file systems, UIDs, PIDs, networking, and so on. As far as code integration code, I think we won’t bother syncing this work with nix-daemon; guix-daemon has already diverged, and for instance it does not have OS X sandbox support. You’ll have to arrange to have the Hurd-specific bits in a separate file, so that ‘#ifdef HURD’ are not scattered all over the place. This is C(++) as you know. WDYT, Manolis? >> This in itself requires some thought: currently (guix system vm) relies >> on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we >> instead need to boot a complete image with GRUB, like ‘guix system >> vm-image’ does. You’ll have to investigate how to port this. > > qemu can boot multiboot operating systems. Great, didn’t know that. >> > Deliver a working GuixSD system image with Hurd as the kernel. > > Hurd is not a kernel. I think we use “kernel” to mean “the thing(s) that libc talks to.” >> The main question is whether you should implement build isolation in >> guix-daemon, in which case that would leave little time for the GuixSD >> parts. I think I would rather let you focus on the GuixSD stuff as you >> wrote, but I’d like to hear what the Hurd folks think. > > I consider isolation more important. OK. So, Manolis, what about reframing the agenda such that porting guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the programming effort), followed by GuixSD stuff? Thanks, Ludo’.
Re: [GSoC] Porting GuixSD to GNU Hurd draft
On 03/24/16 13:38, Justus Winter wrote: > Otoh if we could properly build packages, we could provide the > substitutes, so people could try Guix on the Hurd without going > through the 12h+ bootstrap procedure. Here the problem is not isolation but the fact that we don't have a substitutes server. The machine I had at my house crashed and I cannot bring it back up remotely. When I get back to my home island and revive it, this will not be an issue. > Here is an overview of the early server bootstrap in the Hurd. It is > slightly outdated, but still the best description that I know of: > > http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/ > It definitely is!! Nice one!! > Hmmm, so one still needs a filesystem, right? That's going to be a > bit tricky too, since whatever tool you use for that purpose, it > surely does not support creating hurdish passive translator records. > Without passive translator records things get indeed more interesting. > I have a patch for some tool for creating ext2 filesystems that could > help, or we create all the passive translator records on first boot > similar to how Samuels Debian/Hurd live cds deal with that. Or I > finish my bootshell work that can boot from filesystems without > passive translator records. Where can I find more info on those? Manolis
Re: [GSoC] Porting GuixSD to GNU Hurd draft
Quoting Manolis Ragkousis (2016-03-24 12:18:25) > >>> The project consists of four main stages > >>> > >>> 1. Modify Guix so it will be able to create and mount the file-system > >>> needed to boot into a system with Hurd at its core. > >>> 2. Modify Guix so it can produce a working image, while isolating any > >>> cases of Linux assumptions. > >>> 3. Successfully boot into one such system using GNU Shepherd with pid 1. > >>> 4. Modify the new Guix system to take advantage of Hurd specific > >>> mechanisms. > > > > For me, 4. is the most important bit, so we can build packages in > > isolation. > > I think our priority should be to first get GuixSD working and then > concentrate to achieving isolation. From what I understand none of the > two is trivial but in the long run the ability to spawn GNU/Hurd system > vms on the fly will make it easier to work on it. Otoh if we could properly build packages, we could provide the substitutes, so people could try Guix on the Hurd without going through the 12h+ bootstrap procedure. > Currently I am working on familiarizing myself with both how Hurd and > GuixSD booting works. I am already using GuixSD as a system and I am > going though the source code so I can better understand what is going on. Here is an overview of the early server bootstrap in the Hurd. It is slightly outdated, but still the best description that I know of: http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/ > >> For debugging purposes, it would be very helpful to say the least to > >> have a working ‘guix system vm’: it would allow you to test your changes > >> very quickly, without rebooting and so on. > >> > >> This in itself requires some thought: currently (guix system vm) relies > >> on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we > >> instead need to boot a complete image with GRUB, like ‘guix system > >> vm-image’ does. You’ll have to investigate how to port this. > > > > qemu can boot multiboot operating systems. > > Justus is correct here. I found this > https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which > explains how to make qemu start with gnumach. This way guix system vm > can work, just with the proper modifications. Hmmm, so one still needs a filesystem, right? That's going to be a bit tricky too, since whatever tool you use for that purpose, it surely does not support creating hurdish passive translator records. Without passive translator records things get indeed more interesting. I have a patch for some tool for creating ext2 filesystems that could help, or we create all the passive translator records on first boot similar to how Samuels Debian/Hurd live cds deal with that. Or I finish my bootshell work that can boot from filesystems without passive translator records. Justus
Re: [GSoC] Porting GuixSD to GNU Hurd draft
Hello Justus, Hello Ludo
On 03/23/16 18:55, Justus Winter wrote:
> Quoting Ludovic Courtès (2016-03-23 14:40:38)
>>> 2. The Project
>>>
>>> The project consists of four main stages
>>>
>>> 1. Modify Guix so it will be able to create and mount the file-system
>>> needed to boot into a system with Hurd at its core.
>>> 2. Modify Guix so it can produce a working image, while isolating any cases
>>> of Linux assumptions.
>>> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
>>> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
>
> For me, 4. is the most important bit, so we can build packages in
> isolation.
I think our priority should be to first get GuixSD working and then
concentrate to achieving isolation. From what I understand none of the
two is trivial but in the long run the ability to spawn GNU/Hurd system
vms on the fly will make it easier to work on it.
>>> Currently the tools Guix uses to interact with the filesystem exist inside
>>> the (guix build syscalls) module.
>>> This module provides bindings to libc's syscall wrappers, which are only
>>> available on a GNU/Linux system.
>>> In order to offer the same functionality on a GNU/Hurd system we must first
>>> write Guile bindings for the
>>> relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
>>> for example.
>>
>> Note that technically the ‘file_set_translator’ function is in libc:
>>
>> --8<---cut here---start->8---
>> scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
>> $1 = #
>> --8<---cut here---end--->8---
>>
>>> This module will be called (guix build hurd). This allows us to
>>> re-implement the functionality of the 'settrans' command, as described
>>> here[1], in Scheme and use that in place of 'mount()', where
>>> applicable.
>
> Right. In fact, what settrans (or mount) does isn't that hard to
> reproduce, though I wouldn't mind moving the c implementation to
> libhurdutil or the like and binding to that.
Then I think it will be wiser to move the c implementation of settrans
to libhurdutil and binding to that. Then we will only need a (guix
build hurd) module which will include that binding. I will update the
proposal if you agree.
>> I think it’s important to think about:
>>
>> 1. How functionality equivalent to linux-{initrd,boot} will be
>> implemented on the Hurd.
>>
>> In my experience the equivalent thing is simpler on the Hurd,
>> esp. if relying on passive translators (see daemons/runsystem.sh in
>> the Hurd), though here we’ll probably explicitly activate
>> translators at boot time.
>
> Currently, there is not really a reason to have an initrd-like
> solution on the Hurd. Yes, one has to start the default pager
> explicitly, but that's about it.
Then it seems we only need a hurd-boot module which will take care of
activating translators and booting the system.
> 2. How to structure GuixSD code in a way that abstracts the
> underlying OS kernel.
>
> For instance, we could have a (guix build os) module providing an
> API that works for both kernels, probably close to the Linux one,
> and that would dispatch to the right implementation, the Linux or
> Hurd one.
This module sounds like a great idea. I will add it to the proposal.
>>> Finally, once GuixSD is successfully ported, we can cater to the last stage
>>> of taking advantage of Hurd specific
>>> mechanisms.
>>> This includes but is not limited to:
>>> 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a
>>> GNU/Hurd system. Subhurds can offer
>>> isolation similar to Linux containers as described here[3].
>>
>> This is really super optional IMO. This module is only used by ‘guix
>> environment --container’, which is an optional feature.
>
> Yes, subhurds are more on the experimental side imho.
So Adding subhurd support -> if there is time.
>
>>> 2)Modify the guix-daemon to run without root privileges by utilizing the
>>> Hurd architecture. The daemon needs root
>>> privileges in order to setup chrooted environments for the builds. In the
>>> Hurd this can be done by setting up a
>>> translator as described here[4].
>>
>> This is more important, and already non-trivial work. If libc provided
>> ‘mount’ with support for MS_BIND (which I think it should), it would
>> help a bit, but there’s still the problem of the other Linux name spaces
>> that are used (the CLONE_NEW* clone(2) flags.)
>>
>> Thus it may make more sense to write this functionality in guix-daemon
>> using directly the Hurd interfaces. Separate PID name spaces, UID name
>> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
>> “just” a matter of giving the child process ports to separate proc,
>> auth, etc. translators.
>>
>> In itself this is also a bit of work. I wonder what the Hurd folks
>> think about priorities here?
