Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
May 2, 2023 3:59 PM, "Sergey Bugaev" wrote: > Even just starting a video blogging channel showing off the Hurd would > go a long way towards getting others more interested. (Maybe this is > something Joshua would be interested in doing?) I was thinking I could > live-stream some of my own hacking, seeing how this format is popular, > and how some people have expressed interest; but I cannot do that > currently for various reasons. > Thanks for the compliment. I suppose I could record videos of me playing with the Hurd...I should probably start actually using the Hurd that I have installed more often.
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mer. 03 mai 2023 01:31:53 +0300, a ecrit: > On Wed, May 3, 2023 at 1:20 AM Samuel Thibault > wrote: > > Actually, even what depends on it. That's the whole thing the > > rebootstrap script I mentioned is about, and it's making progress, it > > should be able to produce essentially what is needed to run > > debootstrap/crosshurd. > > Ah, I don't mean the troubles due to cross-compiling specifically (I > don't think the Hurd libraries are any more problematic compared to > the usual cross-compiling issues?), but simply the fact that they > FTBFS on x86_64-gnu unless you apply my changes -- that only exist on > my machine, for now. And there are a lot of them, mostly strategically > changing size_t to mach_msg_type_number_t in many places, but there > are subtler ones too. Ok, I'll just apply the patches :) Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Wed, May 3, 2023 at 1:20 AM Samuel Thibault wrote: > Actually, even what depends on it. That's the whole thing the > rebootstrap script I mentioned is about, and it's making progress, it > should be able to produce essentially what is needed to run > debootstrap/crosshurd. Ah, I don't mean the troubles due to cross-compiling specifically (I don't think the Hurd libraries are any more problematic compared to the usual cross-compiling issues?), but simply the fact that they FTBFS on x86_64-gnu unless you apply my changes -- that only exist on my machine, for now. And there are a lot of them, mostly strategically changing size_t to mach_msg_type_number_t in many places, but there are subtler ones too. Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mer. 03 mai 2023 01:16:29 +0300, a ecrit: > I see [0] -- is that it? Cool! > > [0]: https://salsa.debian.org/glibc-team/glibc/-/commits/hurd-amd64 It's part of it yes. > How does this work -- how does the Debian tooling know what > 'hurd-amd64' means? Is the full list of Debian package architectures > defined somewhere? dpkg knows it: dpkg-architecture -ahurd-amd64 > Are you doing anything else other than glibc as well? The toolchain, basically. > I guess it should be possible to cross-build just about anything that > does not depend on hurd-libs. Actually, even what depends on it. That's the whole thing the rebootstrap script I mentioned is about, and it's making progress, it should be able to produce essentially what is needed to run debootstrap/crosshurd. > How is this meant to be used -- will I be able to point apt somewhere > (where?) and just download the built package(s)? Yes, see the mail I sent about adding hurd-amd64 to debian-ports. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 11:43 PM Samuel Thibault wrote: > > Sergey Bugaev, le mar. 02 mai 2023 22:58:24 +0300, a ecrit: > > redoing bootstrap sounds rather interesting and challenging. It's not > > the most interesting thing right now though, which is why I haven't > > started really doing it yet. > > On that side, I'm working on it for the Debian part. It's almost ready > actually, it just needs fixing here and there (and that's actually > basically debian packaging concerns anyway, since you all did all the > upstream parts already ;) ) ). Ah, I've meant bootstrap as in servers bootstrap, bootstrapfs, all that. Getting Hurd proper to run on x86_64 *is* the most interesting thing right now, which is why I'm doing it :) I see [0] -- is that it? Cool! [0]: https://salsa.debian.org/glibc-team/glibc/-/commits/hurd-amd64 How does this work -- how does the Debian tooling know what 'hurd-amd64' means? Is the full list of Debian package architectures defined somewhere? Are you doing anything else other than glibc as well? I guess it should be possible to cross-build just about anything that does not depend on hurd-libs. How is this meant to be used -- will I be able to point apt somewhere (where?) and just download the built package(s)? Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 22:58:24 +0300, a ecrit: > redoing bootstrap sounds rather interesting and challenging. It's not > the most interesting thing right now though, which is why I haven't > started really doing it yet. On that side, I'm working on it for the Debian part. It's almost ready actually, it just needs fixing here and there (and that's actually basically debian packaging concerns anyway, since you all did all the upstream parts already ;) ) ). Samuel
Re: [hurd,commited] hurd: Enable x86_64 build script
Thanks so much to all people involved in making the 64bit port a real
thing now!
Samuel
Samuel Thibault, le mar. 02 mai 2023 22:24:24 +0200, a ecrit:
> This now passes crossbuilds.
> ---
> NEWS | 5 +
> README | 2 +-
> scripts/build-many-glibcs.py | 3 +++
> 3 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/NEWS b/NEWS
> index 40964d2ee0..054d81fc81 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -24,6 +24,11 @@ Major new features:
> * A new tunable, glibc.pthread.stack_hugetlb, can be used to disable
>Transparent Huge Pages (THP) in stack allocation at pthread_create.
>
> +* Support for x86_64 running on Hurd has been added. This port requires
> + as least binutils 2.40 and GCC 13:
> +
> +- x86_64-gnu
> +
> Deprecated and removed features, and other changes affecting compatibility:
>
> * In the Linux kernel for the hppa/parisc architecture some of the
> diff --git a/README b/README
> index 63f3a1bebf..9532b97986 100644
> --- a/README
> +++ b/README
> @@ -12,7 +12,7 @@ implement the operating system behavior seen by user
> applications.
> In GNU/Hurd systems, it works with a microkernel and Hurd servers.
>
> The GNU C Library implements much of the POSIX.1 functionality in the
> -GNU/Hurd system, using configurations i[4567]86-*-gnu.
> +GNU/Hurd system, using configurations i[4567]86-*-gnu and x86_64-gnu.
>
> When working with Linux kernels, this version of the GNU C Library
> requires Linux kernel version 3.2 or later.
> diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
> index 97b6862167..95726c4a29 100755
> --- a/scripts/build-many-glibcs.py
> +++ b/scripts/build-many-glibcs.py
> @@ -465,6 +465,9 @@ class Context(object):
> 'ccopts': '-m32 -march=i486'},
>{'arch': 'i586',
> 'ccopts': '-m32 -march=i586'}])
> +self.add_config(arch='x86_64',
> +os_name='gnu',
> +gcc_cfg=['--disable-multilib'])
>
> def add_config(self, **args):
> """Add an individual build configuration."""
> --
> 2.39.2
Prospectives (Was: hurd: Add expected abilist files for x86_64)
Sergey Bugaev, le mar. 02 mai 2023 22:58:24 +0300, a ecrit: > On Tue, May 2, 2023 at 10:06 PM Samuel Thibault > wrote: > > > > Sergey Bugaev, le mar. 02 mai 2023 21:59:40 +0300, a ecrit: > > > I don't see how network/disk/USB are relevant > > > > What I mean is that we don't have much workforce. If we keep > > reimplementing things that already work fine enough, we'll not be making > > useful progress. > > That's true; but I might offer a different perspective here. In > SerenityOS, we have this principle (quoting from FAQ): > > > # Will SerenityOS support `$THING`? > > Maybe. Maybe not. There is no plan. > > # When will you implement `$THING`? > > Maybe someday. Maybe never. If you want to see something happen, you can do > > it yourself! > > There's no plan; everyone just hacks on whatever they feel like! The > end result is not that important, what's important is that we all have > fun. > > Same here: I haven't seen any roadmap for what the Hurd must achieve. Yep, sure! But at some point the less funny pieces also need to be done, otherwise the system becomes irrelevant. For instance, we *do* need rust implemented, otherwise we'll lose libgtk support, and probably soon enough also libxml2 and whatnot re-implemented in rust. To put another perspective: I have been doing that less funny part for a long time already. I can't do everything alone. > As you must be aware by now, I... know very little about systems > development, actually. That is absolutely no problem. > I know nothing about hardware. I can't write drivers. I barely know > what an IRQ is! So you can't really expect me to work on writing USB > drivers, if that's what you mean. I'm *not* meaning to write USB drivers, but to fix building rump to use its USB drivers. The PCI/IRQ/etc. part is already working for AHCI, the USB stack is probably not that much more work, and not technical work. > The point being, if you consider USB drivers a priority It's less a priority than e.g. 64bit support and SMP support, but people do have USB CD readers and USB sticks that we really want to support. > On another note: maybe it would be beneficial to get more people > interested / involved in the Hurd? Of course. > Again, look at how Serenity has grown from someone's personal project > our own [...] > our own [...] > our own [...] > [...] > I could go on and on. > We could be making this same kind of progress on the Hurd side! We have already done that actually. And that's actually part of the problem: it just works *a lot* nowadays, so the small fun parts are done. I have put some smallish items on https://darnassus.sceen.net/~hurd-web/contributing/#index1h2 but that doesn't seem to be picked up, mostly. Larger items are quickly very hairy, e.g. SMP support is still an upcoming very large beast. I won't continue repeating myself, see my 4-year-old-already talk at FOSDEM: https://archive.fosdem.org/2019/schedule/event/roadmap_for_the_hurd/ > There's not enough publicity that the Hurd still exists, is developed, > and actually works quite well. People think that it is an abandoned, > failed project. Yes. It's yet another "it's a matter of"... reviving the quarters of the Hurd. That is missing somebody taking up the task. (And no, I cannot do that. Basically, anything that is "Samuel could do XXX" won't work: I'm already completely overloaded, I just cannot do more, and especially not publicity or community maintenance, I'm just completely no good at that). Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 10:06 PM Samuel Thibault wrote: > > Sergey Bugaev, le mar. 02 mai 2023 21:59:40 +0300, a ecrit: > > I don't see how network/disk/USB are relevant > > What I mean is that we don't have much workforce. If we keep > reimplementing things that already work fine enough, we'll not be making > useful progress. That's true; but I might offer a different perspective here. In SerenityOS, we have this principle (quoting from FAQ): > # Will SerenityOS support `$THING`? > Maybe. Maybe not. There is no plan. > # When will you implement `$THING`? > Maybe someday. Maybe never. If you want to see something happen, you can do > it yourself! There's no plan; everyone just hacks on whatever they feel like! The end result is not that important, what's important is that we all have fun. Same here: I haven't seen any roadmap for what the Hurd must achieve. Today I might feel like hacking on x86_64, and hey, we got somewhere, didn't we? Tomorrow (well not literally tomorrow) I may go back to 9pfs, or finally finish the epoll server rewrite and ensure that Wayland still works. Is that useful progress? Maybe, but what matters is that I have fun while doing it. As you must be aware by now, I... know very little about systems development, actually. (In the words of Socrates: "I know that I know nothing"). I know nothing about hardware. I can't write drivers. I barely know what an IRQ is! So you can't really expect me to work on writing USB drivers, if that's what you mean. Yes, I can learn. Look at me, who could have thought, a few years ago, that I would be contributing to the Hurd -- and now to glibc! But I have tried approaching driver development several times now and it never went anywhere. (On the other hand, it also took me three attempts to get into Rust, before it became my favorite language; so maybe there is still some hope for me and drivers.) The point being, if you consider USB drivers a priority -- I can't really help with that, someone else will have to work on it. And among the things I *can* hack on, redoing bootstrap sounds rather interesting and challenging. It's not the most interesting thing right now though, which is why I haven't started really doing it yet. On another note: maybe it would be beneficial to get more people interested / involved in the Hurd? Again, look at how Serenity has grown from someone's personal project (just a few years ago) to the large community that it is now. We have written a whole Unix desktop system, from absolute scratch -- not only the Kernel, but our own LibC, Lib everything else, all the command line utils, our own little Valgrind (UserspaceEmulator), our own IPC system, our own display server, our own GUI toolkit, our own set of apps, our own Browser with our own browser engine, our own media player... I could go on and on. We could be making this same kind of progress on the Hurd side! There's not enough publicity that the Hurd still exists, is developed, and actually works quite well. People think that it is an abandoned, failed project. Even just starting a video blogging channel showing off the Hurd would go a long way towards getting others more interested. (Maybe this is something Joshua would be interested in doing?) I was thinking I could live-stream some of my own hacking, seeing how this format is popular, and how some people have expressed interest; but I cannot do that currently for various reasons. Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 21:59:40 +0300, a ecrit: > I don't see how network/disk/USB are relevant What I mean is that we don't have much workforce. If we keep reimplementing things that already work fine enough, we'll not be making useful progress. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 8:56 PM Samuel Thibault wrote: > > Sergey Bugaev, le mar. 02 mai 2023 19:34:02 +0300, a ecrit: > > On Tue, May 2, 2023 at 7:08 PM Samuel Thibault > > wrote: > > > > > > Sergey Bugaev, le mar. 02 mai 2023 18:47:50 +0300, a ecrit: > > > > On Tue, May 2, 2023 at 6:20 PM Samuel Thibault > > > > wrote: > > > > > > > > What I'm really interested in doing is 'bootstrapfs'... which is kind > > > > of the same thing as the boot shell, but with a different focus. > > > > > > Not like hurd/boot? > > > > You mean boot(1), which is at boot/ in the Hurd repo? > > Yes: it's precisely doing "Super ideally, gnumach wouldn't even know how > to parse bootscripts and load ELFs; that, too, should be done by some > pre-init task". It can also be used for sub-hurds by supporting device > ports etc., but whatever can do a lot can also do a little. Ah in that sense yes, exactly like what boot(1) does. These are two separate ideas that have to be reconciled: one, that it would be nice to have bootscript parsing and execution and ELF loading to happen outside of the kernel, and the other, that the part of the bootstrap that's already done in userspace today could be radically improved. I think/hope that they could fit nicely together, maybe both belong in the same bootstrap task (and maybe I need to learn more about serverboot and bootshell to understand how), but they are independent. Sorry if I wasn't clear enough about this; I just said that the first one should be considered in the context of the second one and then jumped to describing the second one, and interpreted your question in the context of that and not the first one. > Overall it's nice to have crazy ideas about making the boot flexible > and all that, but as it is now it does work, and we also need to have a > system that is relevant, i.e. that has some of the basic support that > people would expect, such as broad network/disk support, USB etc. Yes, I understand. I'm not proposing we break anything that works. If I ever get to implementing this, of course I'll ensure that it works at least as well as what we have now before proposing it for upstreaming. I don't see how network/disk/USB are relevant -- but it should be easier, not harder to get new combinations of servers (as may be required for booting from specific hardware) working under my plan. You wouldn't need to adapt each and every server to run as an early task, port to libmachdev, accept/handle special bootstrap argv options, etc etc, you'd more or less just recompile with -static and edit the bootscript. Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 21:46:14 +0300, a ecrit: > On Tue, May 2, 2023 at 7:48 PM Samuel Thibault > wrote: > > This will be just the fourth time that this part of the Hurd gets > > reimplemented? > > > > I mean, I agree that some pieces can be added to the picture and things > > be improved, but I see that part getting reimplemented by people having > > great ambitions for it, and then it gets reimplemented again and again. > > Then perhaps you could tell me more about the history, so we can avoid > repeating the mistakes of the past :) Basically, overengineering = never really finished and integrated. > I'm vaguely aware that /hurd/startup was a full-blown init system some > time ago ? No, before it got renamed to startup, it was indeed called init, but wasn't doing much more than what startup does: basically start the bootstrap processes and watch them a bit, and give hand to the runsystem shell script which would just run the init rc scripts. > and then it was made to only do what is required to bring up enough of > a Unix environment to then run a third-part init system. I believe it's better to separate the hurdish pieces from the unixish pieces, yes, so distributions can change the unixish part as desired. > I'm aware of the bootshell proposal, but it seems to have never made > it into the mainline Hurd. That letter also mentions that there was > once a 'serverboot' which apparently did all the bootscript things > that gnumach and boot(1) do now? I don't remember. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 7:48 PM Samuel Thibault wrote: > You'll need to store the /dev directory etc. And possibly some temporary > conf files for some translators, etc. Really, an initrd doesn't seem too > horrible a thing. Better maintain few powerful tools than a series of > small tools that then get lesser care. There's not exactly much to store, you'd have a script that creates the few nodes we need (/servers/acpi, /servers/bus/pci, /dev/wd0) programmatically and starts/resumes the tasks meant to translate on them. But I'm not opposed to an idea of a full initrd either, it indeed could be useful if something required config, though I don't think there are currently any servers that do. But say resolving a hostname for network boot may require some config and nss modules and whatnot. > This will be just the fourth time that this part of the Hurd gets > reimplemented? > > I mean, I agree that some pieces can be added to the picture and things > be improved, but I see that part getting reimplemented by people having > great ambitions for it, and then it gets reimplemented again and again. Then perhaps you could tell me more about the history, so we can avoid repeating the mistakes of the past :) I'm vaguely aware that /hurd/startup was a full-blown init system some time ago (which explains some of its peculiarities), and then it was made to only do what is required to bring up enough of a Unix environment to then run a third-part init system. I'm aware of the bootshell proposal, but it seems to have never made it into the mainline Hurd. That letter also mentions that there was once a 'serverboot' which apparently did all the bootscript things that gnumach and boot(1) do now? Sergey
Re: [PATCH v3 0/6] The remaining x86_64-gnu patches
On Tue, 2 May 2023, Sergey Bugaev via Libc-alpha wrote: > It would also probably make sense to mention my other changes, of > which there have been many, in the NEWS (would a simple "many fixes > and improvements in the Hurd port" suffice?) That may well be an appropriate way to describe them (if you haven't opened individual bug reports in Bugzilla; if a bug is reported in Bugzilla, then marked FIXED after the commit with the target milestone set to the first mainline release with the fix, it will automatically be listed in the list of fixed bugs in NEWS). > Am I supposed to write a patch for this, or will somebody else do it? The general expectation is that someone adding a significant new feature also does the NEWS update, and someone adding a new ABI does the build-many-glibcs.py update. > Will adding it to build-many-glibcs.py automatically enable some sort > of server-side CI for this configuration? My build-many-glibcs.py bots automatically run the compilation parts of the testsuite for all configurations included in that script. -- Joseph S. Myers jos...@codesourcery.com
Re: [PATCH v3 0/6] The remaining x86_64-gnu patches
On Tue, 2 May 2023, Samuel Thibault via Libc-alpha wrote: > Joseph Myers, le mar. 02 mai 2023 14:03:16 +, a ecrit: > > On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote: > > > > > If these patches are pushed, it should be possible for anyone to build > > > x86_64-gnu glibc just out of Git master, without having to dig through > > > the mailing list archive for uncommited patches. > > > > In that case I think there should be a patch to build-many-glibcs.py to > > add an x86_64-gnu configuration, > > I have it pending, just waiting for gcc 13. It's enough for the x86_64-gnu support to be in GCC mainline when the patch to build-many-glibcs.py goes in; it doesn't need to be in the release branch (though being in the release branch may be helpful for users). -- Joseph S. Myers jos...@codesourcery.com
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 19:34:02 +0300, a ecrit: > On Tue, May 2, 2023 at 7:08 PM Samuel Thibault > wrote: > > > > Sergey Bugaev, le mar. 02 mai 2023 18:47:50 +0300, a ecrit: > > > On Tue, May 2, 2023 at 6:20 PM Samuel Thibault > > > wrote: > > > > > > What I'm really interested in doing is 'bootstrapfs'... which is kind > > > of the same thing as the boot shell, but with a different focus. > > > > Not like hurd/boot? > > You mean boot(1), which is at boot/ in the Hurd repo? Yes: it's precisely doing "Super ideally, gnumach wouldn't even know how to parse bootscripts and load ELFs; that, too, should be done by some pre-init task". It can also be used for sub-hurds by supporting device ports etc., but whatever can do a lot can also do a little. Overall it's nice to have crazy ideas about making the boot flexible and all that, but as it is now it does work, and we also need to have a system that is relevant, i.e. that has some of the basic support that people would expect, such as broad network/disk support, USB etc. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 19:34:02 +0300, a ecrit: > > > it starts before anyone else, it implements a very basic in-memory fs > > > > tmpfs can be used for that. > > It could, yes, but I also think this might be an overkill. I don't > think we'd need to store any file data, You'll need to store the /dev directory etc. And possibly some temporary conf files for some translators, etc. Really, an initrd doesn't seem too horrible a thing. Better maintain few powerful tools than a series of small tools that then get lesser care. > So I'm thinking it would be something simple, implemented with netfs. I'm just thinking that you might not need to reimplement netfs. > - get rid of /hurd/startup for good, or at least subsume as much of > its functionality as possible This will be just the fourth time that this part of the Hurd gets reimplemented? I mean, I agree that some pieces can be added to the picture and things be improved, but I see that part getting reimplemented by people having great ambitions for it, and then it gets reimplemented again and again. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 7:08 PM Samuel Thibault wrote: > > Sergey Bugaev, le mar. 02 mai 2023 18:47:50 +0300, a ecrit: > > On Tue, May 2, 2023 at 6:20 PM Samuel Thibault > > wrote: > > > > What I'm really interested in doing is 'bootstrapfs'... which is kind > > of the same thing as the boot shell, but with a different focus. > > Not like hurd/boot? You mean boot(1), which is at boot/ in the Hurd repo? In that case, no, not like that at all. boot(1) is all about emulating gnumach for the purpose of starting up a subhurd. It both provides Mach devices (as passthrough, or Mach console device as its stdio) and reimplements what Mach does when loading and starting the bootstrap tasks. But after that it's off to the usual bootstrap dance (exet2fs discovers it's the first task, does special stuff, helps exec start up, spawns /hurd/startup, and so on). > > it starts before anyone else, it implements a very basic in-memory fs > > tmpfs can be used for that. It could, yes, but I also think this might be an overkill. I don't think we'd need to store any file data, nor involve the default pager or whatever. It'd only need to be usable as a "board" (or: "telephone book" -- was the metaphor used somewhere in the Hurd docs I think?) on which port names can be placed and looked up. So I'm thinking it would be something simple, implemented with netfs. These are the main ideas: - provide / maintain "fs as a name service" during early boot, reattach the same translators to the real root fs when it's up - present a mostly normal looking environment to early bootstrap tasks so they don't need any special handling, and more servers can be repurposed as early boot tasks (note that once the bootstrap as done, they keep running as just normal tasks), this would enable network boot for example - concentrate as much of early startup knowledge / specifics in the bootstrapfs, stop/undo spreading it over everything - get rid of /hurd/startup for good, or at least subsume as much of its functionality as possible - use the regular msg.defs facilities to publish services (rootfs, proc, auth) as they come up -- again, no special modification (fsys_init...) needed, this is already supported by glibc - make all of this scriptable and hackable, using lisp if y'all so prefer But given you're not yet screaming that this is a horrible idea, I feel encouraged :D Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 18:47:50 +0300, a ecrit: > On Tue, May 2, 2023 at 6:20 PM Samuel Thibault > wrote: > > What I'm really interested in doing is 'bootstrapfs'... which is kind > of the same thing as the boot shell, but with a different focus. Not like hurd/boot? > it starts before anyone else, it implements a very basic in-memory fs tmpfs can be used for that. Samuel
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Tue, May 2, 2023 at 6:20 PM Samuel Thibault wrote:
What I'm really interested in doing is 'bootstrapfs'... which is kind
of the same thing as the boot shell, but with a different focus.
Uhh, I need to put in the time to actually write one of those long
letters detailing what I have in mind; but here's an "elevator pitch":
it starts before anyone else, it implements a very basic in-memory fs
(*not* an initrd) which is sufficient to host filesystem nodes on
which translators can be installed (probably only /dev and /servers),
and it also implements some proc RPCs (maybe even the I/O RPCs to give
everyone working std{in,out,err}). It then starts everyone else, and
those all see a much less spartan environment than they do today, i.e.
they have a root filesystem port (to bootstrapfs), and they do things
much as they would normally do on a full booted-up Hurd system, not
with special early boot-up codepaths (libmachdev...).
Filesystem-as-name-service is maintained, in particular ext2fs can
just look up /dev/hd0 (or /dev/wd0), and everyone can lookup
/servers/bus/pci, and 9pfs will be able to look up /servers/socket/2
if you want boot-over-network (but you'll have to statically link
pfinet and add it to the boot script). When the real root fs is up,
bootstrapfs re-attaches all of the translators there and then
"publishes" the root fs port by sending msg_set_init_port to everyone.
Then it starts auth and proc and also publishes their ports in the
same way. In the end everything's connected properly and bootstrapfs
can quietly go away (or exec sysv init perhaps).
And hopefully all of this is scriptable and hackable and nice, because
all of the tricky logic is in one place (bootstrapfs), and everyone
else sees a mostly regular environment and may not even 'notice'
they're running as an early bootstrap task.
There are some subtle details here, and again, I need to write the
full version, but that's the idea.
Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
Sergey Bugaev, le mar. 02 mai 2023 17:10:17 +0300, a ecrit: > On Mon, May 1, 2023 at 8:43 PM Samuel Thibault > wrote: > > > How do we proceed? I don't know enough about rump to get it building; > > > > We can easily cross-build debian packages thanks to the rebootstrap > > scripts from Helmut: > > > > https://salsa.debian.org/helmutg/rebootstrap.git > > I didn't just mean the build system; I'm not familiar with rump at > all. I hardly understand what it is, other than it's some code > extracted from NetBSD and intended to somehow be embeddable in a broad > range of environments. I've skimmed their whitepapers, but I still > don't have any idea how they accomplish what they are claiming to. > > So I'm clearly not the right person to port rump to x86_64-gnu. Unless > you're saying that it's largely architecture-independent and will just > build. It should be largely arch-independent (or ported). > I was secretly hoping that the Linux drivers won't be fully abandoned > with the rise of rump and lwip; They are more than oldish. We really don't want to keep them. > but that Linux and non-Linux implementations will both stay as > available alternatives. And that someone would upgrade the bundled > Linux code (6.3 is out now...) -- perhaps a future project for myself. Really, don't spend time on that, Linux is a *way* too evolving target. Been there, seen it tried both. Don't try to do that (TM). > > Now, that being said, you can also use an initrd, see debian's > > 50_initrd.patch (which nobody has yet taken the time to clean according > > the discussion at the time). That's enough to run a full Hurd system, > > you can for instance take the initrd files from the debian installation > > CD. > > Yes! An initrd sounds exactly like what I need, thank you! > > Well, I'm unlikely to need whatever Debian is putting there, but the > idea of a ramdisk, I mean. Yes, and a working example. > As for the best way to implement this... I have always thought gnumach > should have a bootscript directive to expose a multiboot module as a > memobj. Then there would be a task, /hurd/ramdisk, that would expose > it over the device (and possibly I/O) protocols: That was the idea discussed at the time... That nobody took the time to implement so far. Samuel
Re: [PATCH v3 0/6] The remaining x86_64-gnu patches
Joseph Myers, le mar. 02 mai 2023 14:03:16 +, a ecrit: > On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote: > > > If these patches are pushed, it should be possible for anyone to build > > x86_64-gnu glibc just out of Git master, without having to dig through > > the mailing list archive for uncommited patches. > > In that case I think there should be a patch to build-many-glibcs.py to > add an x86_64-gnu configuration, I have it pending, just waiting for gcc 13. Samuel
Re: [PATCH v3 0/6] The remaining x86_64-gnu patches
Hello, On Tue, May 2, 2023 at 5:03 PM Joseph Myers wrote: > On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote: > > > If these patches are pushed, it should be possible for anyone to build > > x86_64-gnu glibc just out of Git master, without having to dig through > > the mailing list archive for uncommited patches. > > In that case I think there should be a patch to build-many-glibcs.py to > add an x86_64-gnu configuration, as well as updates to NEWS and README to > reflect the new support for such a configuration. yes, that would be great! It would also probably make sense to mention my other changes, of which there have been many, in the NEWS (would a simple "many fixes and improvements in the Hurd port" suffice?) Am I supposed to write a patch for this, or will somebody else do it? Will adding it to build-many-glibcs.py automatically enable some sort of server-side CI for this configuration? Sergey
Re: [PATCH v3 4/6] hurd: Add expected abilist files for x86_64
On Mon, May 1, 2023 at 8:43 PM Samuel Thibault wrote:
> > How do we proceed? I don't know enough about rump to get it building;
>
> We can easily cross-build debian packages thanks to the rebootstrap
> scripts from Helmut:
>
> https://salsa.debian.org/helmutg/rebootstrap.git
I didn't just mean the build system; I'm not familiar with rump at
all. I hardly understand what it is, other than it's some code
extracted from NetBSD and intended to somehow be embeddable in a broad
range of environments. I've skimmed their whitepapers, but I still
don't have any idea how they accomplish what they are claiming to.
So I'm clearly not the right person to port rump to x86_64-gnu. Unless
you're saying that it's largely architecture-independent and will just
build.
> > so enabling the in-kernel Linux drivers seems preferable.
> > Any tips on how I would do that? Are the Linux drivers ancient enough
> > to require their own porting to x86_64, or will they just work?
>
> They will just *not* work with 64bit. The glue code is not ready at all,
> and I think it's really not useful to spend time on that when we have
> the rump drivers which should be working fine.
I was secretly hoping that the Linux drivers won't be fully abandoned
with the rise of rump and lwip; but that Linux and non-Linux
implementations will both stay as available alternatives. And that
someone would upgrade the bundled Linux code (6.3 is out now...) --
perhaps a future project for myself.
> Now, that being said, you can also use an initrd, see debian's
> 50_initrd.patch (which nobody has yet taken the time to clean according
> the discussion at the time). That's enough to run a full Hurd system,
> you can for instance take the initrd files from the debian installation
> CD.
Yes! An initrd sounds exactly like what I need, thank you!
Well, I'm unlikely to need whatever Debian is putting there, but the
idea of a ramdisk, I mean.
As for the best way to implement this... I have always thought gnumach
should have a bootscript directive to expose a multiboot module as a
memobj. Then there would be a task, /hurd/ramdisk, that would expose
it over the device (and possibly I/O) protocols:
module /boot/initrd.img $(initrd-image=vm-object-create)
module /hurd/ramdisk.static ramdisk --vm-object-port=${initrd-image}
$(task-create) $(task-resume)
This is a lot like data-task-create which has been proposed in the
"bootshell" discussion, but without repurposing tasks for this. A
memobj IMO is a much cleaner way to represent a literal uninterpreted
piece of memory provided to us by the bootloader.
Super ideally, gnumach wouldn't even know how to parse bootscripts and
load ELFs; that, too, should be done by some pre-init task; the only
thing we need Mach to do is to pass over the memory regions it has
received from the bootloader to userspace, as memobjects -- and start
the initial task, somehow. From there on, the initial task would parse
the script, parse the ELFs, create the tasks, map the ELF segments
into their memory (with vm_map, since these are memobjects, and not
with a simple copy as it's done now), insert the ports and argv
according to the scripts -- all in userspace.
The question is of course how to start this initial task; but that
needs to be discussed in the context of my broader ideas for how to
reengineer bootstrap...
Anyways, I'll see if I can apply the initrd patch and get it working. Thank you.
Sergey
Re: [PATCH v3 0/6] The remaining x86_64-gnu patches
On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote: > If these patches are pushed, it should be possible for anyone to build > x86_64-gnu glibc just out of Git master, without having to dig through > the mailing list archive for uncommited patches. In that case I think there should be a patch to build-many-glibcs.py to add an x86_64-gnu configuration, as well as updates to NEWS and README to reflect the new support for such a configuration. -- Joseph S. Myers jos...@codesourcery.com
Re: [PATCH 5/5] add setting gs/fsbase
On Tue, May 2, 2023 at 10:04 AM Samuel Thibault wrote: > I don't see any issue on 32bit gnumach: > > $ ./test > Killed > > Perhaps with is only with 64bit gnumach Yes, most likely it's something 64-bit specific. We know from experience that the 32-bit gnumach survives a task_terminate call just fine :) But then maybe you should be checking with only a single bootstrap task trying to terminate itself, not a full Hurd environment, which, again, is surely known to work. Sergey
Re: [PATCH 5/5] add setting gs/fsbase
Hello, Sergey Bugaev, le mer. 26 avril 2023 20:33:20 +0300, a ecrit: > ../kern/ipc_tt.c:395: retrieve_task_self_fast: Assertion > `task->itk_self != IP_NULL' failed.panic ../kern/debug.c:103: > Debugger: Debugger invoked, but there isn't one! > > This is after typing 'quit' in bc, which calls exit () -- I had to fix > up _hurd_exit () in glibc a little to not crash if we don't have > _hurd_ports. From single-stepping, it seems task_terminate () works, > as in it tears down and deallocates the kernel task_t, but then the > syscall (which task_terminate is) just returns back to userspace to > the now-nonexistent task, and it keeps running. It then calls another > syscall and that one breaks with the assertion above. > > You should be able to reproduce this without glibc by just calling > task_terminate (mach_task_self ()). I don't see any issue on 32bit gnumach: $ ./test Killed Perhaps with is only with 64bit gnumach, with the AST questions, I see notably thread_terminate that in the thread==cur_thread case, AST_TERMINATE is used. Samuel
