Re: [GSoC] Porting GuixSD to GNU/Hurd Update

[Ludovic Courtès]

Hello Manolis,

Manolis Ragkousis  skribis:

> On Hurd though, because of the lack of mount(), the above could not
> work. But with the help of my libhurdutil library, which I wrote at the
> beginning of this project, I created 2 wrappers inside
> nix/libutil/call.(hh.cc) for mount() and umount2(), called nixMount()
> and nixUmount2(), and depending on the system the implementation
> changes. On Hurd I am using /hurd/firmlink to offer the same
> functionality to bind-mounts.
>
> It seems to work but I am testing it thoroughly so we don't have any
> unpleasant surprises in the future. We will have fully isolated builds
> on Hurd as soon as I am sure that my code works as expected and it's
> merged in Guix upstream.

Neat!  You did the right thing.

> Now on the GuixSD side, I will start on a big issue I had. If you check
> the daemon, and specifically nix/libstore/build.cc:2205-2228, you will
> see that when, for example, guix tries to create a 32 bit vm/image from
> a 64 bit machine, the daemon actually builds the 32 bit binaries on 32
> bit personality mode.

Indeed; hopefully we can address that.

> As a result it is not possible to build GuixSD/Hurd vm/images from
> Linux, for now. But this is not a big problem because we can do it from
> Guix running on Hurd :-). The approach I used, was to add a second hard
> drive on my Hurd vm, mount it, and try to directly deploy a GuixSD
> system on that drive.  Currently Guix can build most of the binaries for
> the system but it still needs work to actually boot into one. You can
> see the result of this work on the currect core-updates (and on my
> github repo for some hacky commits).

Woow!  We knew booting GuixSD on GNU/Hurd was an ambitious goal, so what
you describe is already impressive.

> I have also created a new module called (guix build syscalls-tools)
> which contains some of the code from (guix build syscalls) which will be
> used by a (guix build hurd) module, which will contain call wrappers for
> some Hurd libraries. This work is still in my github repo because it
> needs some work.

Nice stuff again.

> There was one more problem that appeared after we started using
> C_INCLUDE_PATH in our cross-builds. As it seems MiG needs glibc in order
> to be built.  That's why for now I patch cross-mig with the
> glibc-headers so I don't have to depend on the Linux ones. But I will
> talk more on this in a different email.
>
> I think that's enough for now. I avoid talking about things discussed in
> other mails, but if you want please feel free to ask me :-)
>
> To sum things up, we almost have build isolation working but GuixSD
> still needs some work to become bootable. From here on I will
> finish/test my guix-daemon code, and then I will continue on finishing
> the low-level call wrappers for the Hurd libraries, in order to get the
> bootable system. We are close :-).
>
> The repos which you can check any code not yet in upstream Guix or Hurd are:
> https://github.com/Phant0mas/Guix-on-Hurd
> https://github.com/Phant0mas/Hurd

Now we need to chew that, both on the Guix and Hurd sides.  Make sure to
ping us with sizable chunks of this code!  :-)

This is again an impressive piece of work, especially given its breadth:
From cross-compilation toolchain issues, to low-level Linux- and
Hurd-specific programming, and to high-level Scheme hacking.  Kudos!

Thank you Manolis!

Ludo’.


signature.asc
Description: PGP signature

Re: [GSoC] Porting GuixSD to GNU/Hurd Update

[Samuel Thibault]

Manolis Ragkousis, on Sat 20 Aug 2016 21:31:43 +0300, wrote:
> On 08/20/16 20:47, Samuel Thibault wrote:
> > Ok, but the Linux part uses glibc 2.22, and you'd rather use it, right?
> 
> Of course :-)

Ok, I have pushed it, along the debian glibc 2.22 fixup patches, it
seems to be working fine with the i586 build.

About the i686 build, we actually had to make it work in Debian to have
issues with departing from other ports. It seems that the issue we have
is with ifunc, and so we are using

export libc_cv_ld_gnu_indirect_function=no

to just disable ifunc for now.  You'll probably want to do the same.

Samuel

Re: [GSoC] Porting GuixSD to GNU/Hurd Update

[Manolis Ragkousis]

On 08/20/16 20:47, Samuel Thibault wrote:
> Ok, but the Linux part uses glibc 2.22, and you'd rather use it, right?

Of course :-)

Re: [GSoC] Porting GuixSD to GNU/Hurd Update

[Samuel Thibault]

Manolis Ragkousis, on Sat 20 Aug 2016 20:42:25 +0300, wrote:
> Actually the Hurd part of Guix uses 2.19 as it is in
> ftp://alpha.gnu.org/gnu/hurd/
> 
> The glibc tarballs use 2.19 as their version, which is the one I am
> using as the version in our glibc/hurd recipe.

Ok, but the Linux part uses glibc 2.22, and you'd rather use it, right?

Samuel

Re: [GSoC] Porting GuixSD to GNU/Hurd Update

[Manolis Ragkousis]

Hey Samuel,

Actually the Hurd part of Guix uses 2.19 as it is in
ftp://alpha.gnu.org/gnu/hurd/

The glibc tarballs use 2.19 as their version, which is the one I am
using as the version in our glibc/hurd recipe.

Check (gnu packages base) glibc/hurd.

Manolis

On 08/20/16 19:24, Samuel Thibault wrote:
> Hello,
> 
> Which glibc version is guix based on?  It's very high time we upgrade
> our glibc repository, because we see people contribute patch against
> it and it makes use rebase the work in downstreams.  I'd thus say we
> should make our glibc repository based on the minimum version used by
> the supported distros (i.e. debian & guix). Debian is currently at 2.23
> and will soon move to 2.24, a quick grep on the Guix-on-Hurd repository
> seems to be showing glibc 2.22, so we could use 2.22 for now?
> 
> Samuel
> 

[GSoC] Porting GuixSD to GNU/Hurd Update

[Manolis Ragkousis]

Hello Guix, Hello Hurd,

As GSoC is coming to an end I think it's time to sum up my work till
now, report issues I had and then talk about what our next steps are.

First I would like to mention what were the two main objectives of the
project and also what was the status of the port when the project
started, so I can give you a better understanding of the progress.

The objectives were:
1) Achieve build isolation in the daemon on the Hurd.
2) Modify Guix so it can produce a working image, while isolating any
cases of Linux assumptions.
3) Boot to GuixSD/Hurd

When the project started the guix-daemon could work on GNU/Hurd but the
builds were being affected by the system state. Normally the daemon
creates a chrooted environment to make the builds, inside /tmp, and bind
mounts the parts of the system it needs. This way it achieves isolation.

On Hurd though, because of the lack of mount(), the above could not
work. But with the help of my libhurdutil library, which I wrote at the
beginning of this project, I created 2 wrappers inside
nix/libutil/call.(hh.cc) for mount() and umount2(), called nixMount()
and nixUmount2(), and depending on the system the implementation
changes. On Hurd I am using /hurd/firmlink to offer the same
functionality to bind-mounts.

It seems to work but I am testing it thoroughly so we don't have any
unpleasant surprises in the future. We will have fully isolated builds
on Hurd as soon as I am sure that my code works as expected and it's
merged in Guix upstream.

Now on the GuixSD side, I will start on a big issue I had. If you check
the daemon, and specifically nix/libstore/build.cc:2205-2228, you will
see that when, for example, guix tries to create a 32 bit vm/image from
a 64 bit machine, the daemon actually builds the 32 bit binaries on 32
bit personality mode.

As a result it is not possible to build GuixSD/Hurd vm/images from
Linux, for now. But this is not a big problem because we can do it from
Guix running on Hurd :-). The approach I used, was to add a second hard
drive on my Hurd vm, mount it, and try to directly deploy a GuixSD
system on that drive.  Currently Guix can build most of the binaries for
the system but it still needs work to actually boot into one. You can
see the result of this work on the currect core-updates (and on my
github repo for some hacky commits).

I have also created a new module called (guix build syscalls-tools)
which contains some of the code from (guix build syscalls) which will be
used by a (guix build hurd) module, which will contain call wrappers for
some Hurd libraries. This work is still in my github repo because it
needs some work.

There was one more problem that appeared after we started using
C_INCLUDE_PATH in our cross-builds. As it seems MiG needs glibc in order
to be built.  That's why for now I patch cross-mig with the
glibc-headers so I don't have to depend on the Linux ones. But I will
talk more on this in a different email.

I think that's enough for now. I avoid talking about things discussed in
other mails, but if you want please feel free to ask me :-)

To sum things up, we almost have build isolation working but GuixSD
still needs some work to become bootable. From here on I will
finish/test my guix-daemon code, and then I will continue on finishing
the low-level call wrappers for the Hurd libraries, in order to get the
bootable system. We are close :-).

The repos which you can check any code not yet in upstream Guix or Hurd are:
https://github.com/Phant0mas/Guix-on-Hurd
https://github.com/Phant0mas/Hurd

I think that's it for now. I want to thank all of you for your help and
support and for answering my questions, and thank my two mentors Justus
and Ludo for their invaluable help (you guys are awesome :-)).

I would also like to state that thanks to the latest work from the Hurd
guys on Gnumach and Hurd, debian/hurd is more stable than ever!!

I am sure I forgot some things so feel free to ask anything or correct
me. :-)

Thank you,
Manolis

Re: [GSoC] Porting GuixSD to GNU/Hurd Update 1

[Ludovic Courtès]

Hi Manolis!

Manolis Ragkousis  skribis:

> We already knew that the guix-daemon was not working properly on the
> Hurd.  I looked more into it and I found that if you do a build with
> Guix and it fails, and then you retry again, the next one will create a
> new build directory in /tmp/ as it should, but it will still use the
> first one.

Are you sure about this?  There’s a nice feature from a reproducibility
viewpoint, when using chroot builds, which is that the build directory
inside the chroot is always the same (always
“/tmp/guix-build-foobar.drv-0”), even if its name outside the chroot
needs to be different (say, “/my/tmp/guix-build-foobar.drv-12”).  See
Guix commit cb9601029ea164b86bdf997f7160d494c15d344b.

> I also found out that guix publish may sometimes crash if a client asks
> for a big substitute.  I will investigate this one as soon as possible.

If you can find a way to reproduce it, please send it to
bug-g...@gnu.org.

I saw on IRC that you publish GNU/Hurd binaries from your machine, and I
find it pretty cool!

> Now on the coding part, in Guix I updated the gnumach/mig/hurd packages
> to the latest version and worked on getting wip-hurd in a state which
> can eventually merge to master.  A part of wip-hurd is already on master
> and after this core-updates cycle is merged to master, we will be able
> to get the rest of wip-hurd merged as well. Package glibc/hurd will also
> be updated then as well.

Great, I’m willing to make progress on this front!

> On debian/hurd I am using Guix every day and in my github wip-hurd I
> have some patches which I need to clean up and apply to the Guix repo.
> These patches are for various packages of various importance.

Awesome.

> And on the Hurd side, I created a new library called libhurdutil and
> moved the settrans implementation there.  For implementation info please
> read this email[1]. I am currently improving that library after
> Guillem's and Justus' input. (Thank you :-))  On the Guix side I will
> just write a wrapper to use this call and we will be able to control
> translators really easily.  This way the non existing mount() is not a
> problem anymore :-).

Nice work!  Perhaps the library should be called “libhurdtrans” or
something like that to better reflect what it’s about?

It may be that the functionality that was extracted from the ‘settrans’
command needs be nicely “repackaged” to make for a nice library
interface (for instance, by removing global variables and by untangling
UI concerns from actual functionality), but I’ll leave it up to the Hurd
people to comment on it.  :-)

Thanks for the update!

Ludo’.

[GSoC] Porting GuixSD to GNU/Hurd Update 1

[Manolis Ragkousis]

Hello everyone,

One week passed since the beginning of the coding phase of GSoC, so I
think it's time to give you an update on my progress.

In the previous weeks I tried to get me prepared as much as possible and
get a better idea of what needs to be done.  I concentrated on getting
familiar with the parts of Guix and Hurd that are more relevant to this
project and I found out some interesting things.

We already knew that the guix-daemon was not working properly on the
Hurd.  I looked more into it and I found that if you do a build with
Guix and it fails, and then you retry again, the next one will create a
new build directory in /tmp/ as it should, but it will still use the
first one.  What this means is that the status of the previous build
leaks into the new one and there is no isolation.  With this in mind
build isolation in the daemon is currently my first priority.

I also found out that guix publish may sometimes crash if a client asks
for a big substitute.  I will investigate this one as soon as possible.

Now on the coding part, in Guix I updated the gnumach/mig/hurd packages
to the latest version and worked on getting wip-hurd in a state which
can eventually merge to master.  A part of wip-hurd is already on master
and after this core-updates cycle is merged to master, we will be able
to get the rest of wip-hurd merged as well. Package glibc/hurd will also
be updated then as well.

On debian/hurd I am using Guix every day and in my github wip-hurd I
have some patches which I need to clean up and apply to the Guix repo.
These patches are for various packages of various importance.

And on the Hurd side, I created a new library called libhurdutil and
moved the settrans implementation there.  For implementation info please
read this email[1]. I am currently improving that library after
Guillem's and Justus' input. (Thank you :-))  On the Guix side I will
just write a wrapper to use this call and we will be able to control
translators really easily.  This way the non existing mount() is not a
problem anymore :-).

I think that's enough for now.  If you have any questions/suggestions
please feel free to tell me. :-)

Thank you,
Manolis

[1]https://lists.gnu.org/archive/html/bug-hurd/2016-05/msg00041.html