Re: Possibility of basing a QNX-like OS on NetBSD?
On Mon, May 27, 2024 at 9:13 PM David Holland wrote: > > Barrelfish? Also, compared to the cost of the rest of the system, > dusting off a microkernel that's been on the shelf a bit is cheap. > But IDK, I don't follow these in any detail. > Even the inactive L4-like kernels I'm aware of aren't any closer to QNX Neutrino than seL4 is. Most of them lack capabilities entirely and use threads as IPC destinations rather than having thread-independent endpoints/channels (even the IPC gates in Fiasco.OC and NOVA are bound to threads, whereas the only connection in MCS seL4 is that its endpoints maintain queues of waiting threads). I guess VSTa's kernel would be closer to QNX, but its IPC model isn't fully capability-oriented (QNX's isn't either, which makes security more complex than necessary; this is one of the issues with QNX I want to fix in my own OS) and it only supports x86-32. > I don't see the difference between what you're describing and the > rfork/clone model. But in any event, the problems with pthreads > remain; it's difficult to implement the pthreads behavior of > fork/exec/wait in a model where threads aren't tied to processes. > In the traditional variant of the rfork()/clone() model, there is nothing to visibly group related threads together, and the state containers are anonymous and can't be manipulated directly, only cloned. With the model I'm going to use, a process will be a group of threads (rather than threads being a type of process), but will have basically no state other than a list of threads and a command line/environment, and state containers will be directly visible in procfs. Every thread will belong to one and only one process. fork() (which will be a library function that uses various state manipulation APIs) will only fork a single thread. exec*() will replace an entire process and all threads in it regardless of what state containers are bound to them. wait() will specifically wait for an an entire process, although it will be built on top of a more general API that can wait for individual threads, entire processes, or entire cgroups. > > It only looks like Unix from the outside, and not all that much even > then... > I guess it depends on which definition of "Unix" you consider the most important. Usually I tend to consider "functional Unix" (which is a matter of APIs and commands) more important than "conventional functional Unix" (which is a matter of kernel internals). QNX 4 and Neutrino certainly qualify as "functional Unix" the way I define it, but definitely do not qualify as "conventional functional Unix".
Re: Possibility of basing a QNX-like OS on NetBSD?
On Sat, May 25, 2024 at 07:40:39PM -0600, Andrew Warkentin wrote: > > I think the question you should be asking is what your goal is -- are > > you using seL4 because you specifically want to leverage seL4's > > properties? If so, launching off in another direction seems like the > > wrong move. If not, there are other L4-style microkernels you can use > > that don't have as many restrictions as seL4, and there's a largish > > community of advocates that will each be eager to help you decide to > > use theirs. > > AFAIK the only other active L4-like kernels are NOVA and Fiasco.OC, Barrelfish? Also, compared to the cost of the rest of the system, dusting off a microkernel that's been on the shelf a bit is cheap. But IDK, I don't follow these in any detail. > My OS is going to be multi-server, although it will mostly have a > process-per-subsystem architecture like QNX does, and not a > process-per-component architecture like Genode or L4Re. Protection > domains correspond to subsystems more often than they do to > components, so there's not much point in splitting up components in a > lot of cases. Well, that depends. The whole premise of a microkernel is that you gain something by being in a separate address space and that it makes sense to split up the (traditionally unitary) kernel protection domain to gain that something. How finely you split it seems like a matter of taste. (Or, sometimes, religious doctrine...) > > The primary reason the world's gradually moved from that model of > > threads to a model where threads are second-class within processes > > is... design stupidity in pthreads. If you want to support pthreads, > > especially if you have any concerns about it being fast, don't go in > > this direction. > > I don't think I've seen any Unix-like OS that uses the thread model > I'm planning to use. They pretty much all either use a Plan > 9/Linux-like rfork()/clone() "threads are processes sharing state" > model, or a Mach-like "processes consist of state and threads" model, > rather than a "state is independent of processes and explicitly bound > to specific threads" model. The model I'm planning to use is a close > match to that of seL4, where capability spaces and virtual address > spaces exist independently of each other and threads. I don't see the difference between what you're describing and the rfork/clone model. But in any event, the problems with pthreads remain; it's difficult to implement the pthreads behavior of fork/exec/wait in a model where threads aren't tied to processes. > > > Another thing that I'm not sure about is the real-time performance. In > > > addition to desktop and server use, embedded systems with hard > > > real-time constraints are also an important use case for this system. > > > > In that case you want to stay a long way away from anything that looks > > like Unix. > > QNX is Unix-like and has reasonable performance for real-time systems, > although of course it is quite different from conventional Unix in its > architecture. It only looks like Unix from the outside, and not all that much even then... -- David A. Holland dholl...@netbsd.org
Re: Possibility of basing a QNX-like OS on NetBSD?
On Sat, May 25, 2024 at 3:04 PM David Holland wrote: > > > I think the question you should be asking is what your goal is -- are > you using seL4 because you specifically want to leverage seL4's > properties? If so, launching off in another direction seems like the > wrong move. If not, there are other L4-style microkernels you can use > that don't have as many restrictions as seL4, and there's a largish > community of advocates that will each be eager to help you decide to > use theirs. > AFAIK the only other active L4-like kernels are NOVA and Fiasco.OC, and neither of them have QNX-style long message support. seL4's memory model is probably better for the system I'm working on; NOVA and Fiasco.OC are more designed with strongly hierarchical systems in mind where there are a few subsystem servers that each get their own memory quota and hand out parts of that to child processes, rather than a more traditional Unix-like model (I do plan to support memory quotas and resource trading, although they won't follow a strict hierarchy). There are several things I'd like to add to my fork of seL4, but most of them should be simpler to add than long message support. The only changes I could see myself making on a similar scale to long IPC would be if there are issues with SMP scalability. > > If on the other hand you're specifically interested in making > something QNX-like, you'll probably need to write most of it from > scratch, since the only way to get that level of small and fast is > careful integration and careful design balancing goals and costs. > > In any event, NetBSD isn't a microkernel and turning it into one would > pretty much be a 100% rewrite, the existence of rump notwithstanding. > It would be expensive and probably not come out all that well. > That was pretty much my conclusion as well. > > If you want a single-server Unix on top of a microkernel, rump is > maybe not a bad place to start... but this has been done many times > before and you're probably better off using one of the existing > examples. > My OS is going to be multi-server, although it will mostly have a process-per-subsystem architecture like QNX does, and not a process-per-component architecture like Genode or L4Re. Protection domains correspond to subsystems more often than they do to components, so there's not much point in splitting up components in a lot of cases. > > > I also wish to disaggregate the usual Unix process model into a far > > more thread-centric one where a process is nothing more than a > > collection of threads that share a command line and get replaced on > > exec, with all of the usual process state like virtual address space, > > open file descriptors, and filesystem namespace being separate context > > objects that have to be explicitly shared between threads, and the > > basic process creating primitive just creating a completely blank > > process that the parent explicitly initializes with all the necessary > > state using various APIs (of course, there will be a library that > > implements fork(), spawn(), and pthreads on top of this). > > The primary reason the world's gradually moved from that model of > threads to a model where threads are second-class within processes > is... design stupidity in pthreads. If you want to support pthreads, > especially if you have any concerns about it being fast, don't go in > this direction. > I don't think I've seen any Unix-like OS that uses the thread model I'm planning to use. They pretty much all either use a Plan 9/Linux-like rfork()/clone() "threads are processes sharing state" model, or a Mach-like "processes consist of state and threads" model, rather than a "state is independent of processes and explicitly bound to specific threads" model. The model I'm planning to use is a close match to that of seL4, where capability spaces and virtual address spaces exist independently of each other and threads. > > > Another thing that I'm not sure about is the real-time performance. In > > addition to desktop and server use, embedded systems with hard > > real-time constraints are also an important use case for this system. > > In that case you want to stay a long way away from anything that looks > like Unix. > QNX is Unix-like and has reasonable performance for real-time systems, although of course it is quite different from conventional Unix in its architecture.
Re: Possibility of basing a QNX-like OS on NetBSD?
On Sat, May 25, 2024 at 05:48:56AM -0600, Andrew Warkentin wrote: > I'm currently working on a QNX-like microkernel OS based on a fork of > seL4 and an original root server. Recently I had someone suggest that > I should look at trying to turn NetBSD into a QNX-like microkernel > because seL4's focus is more on static non-Unix-like systems. That's a curious proposition, kind of like suggesting you should harness a dump truck to your horses because chariots are for racing. I think the question you should be asking is what your goal is -- are you using seL4 because you specifically want to leverage seL4's properties? If so, launching off in another direction seems like the wrong move. If not, there are other L4-style microkernels you can use that don't have as many restrictions as seL4, and there's a largish community of advocates that will each be eager to help you decide to use theirs. If on the other hand you're specifically interested in making something QNX-like, you'll probably need to write most of it from scratch, since the only way to get that level of small and fast is careful integration and careful design balancing goals and costs. In any event, NetBSD isn't a microkernel and turning it into one would pretty much be a 100% rewrite, the existence of rump notwithstanding. It would be expensive and probably not come out all that well. If you want a single-server Unix on top of a microkernel, rump is maybe not a bad place to start... but this has been done many times before and you're probably better off using one of the existing examples. > I also wish to disaggregate the usual Unix process model into a far > more thread-centric one where a process is nothing more than a > collection of threads that share a command line and get replaced on > exec, with all of the usual process state like virtual address space, > open file descriptors, and filesystem namespace being separate context > objects that have to be explicitly shared between threads, and the > basic process creating primitive just creating a completely blank > process that the parent explicitly initializes with all the necessary > state using various APIs (of course, there will be a library that > implements fork(), spawn(), and pthreads on top of this). The primary reason the world's gradually moved from that model of threads to a model where threads are second-class within processes is... design stupidity in pthreads. If you want to support pthreads, especially if you have any concerns about it being fast, don't go in this direction. > Another thing that I'm not sure about is the real-time performance. In > addition to desktop and server use, embedded systems with hard > real-time constraints are also an important use case for this system. In that case you want to stay a long way away from anything that looks like Unix. -- David A. Holland dholl...@netbsd.org
Possibility of basing a QNX-like OS on NetBSD?
I'm currently working on a QNX-like microkernel OS based on a fork of seL4 and an original root server. Recently I had someone suggest that I should look at trying to turn NetBSD into a QNX-like microkernel because seL4's focus is more on static non-Unix-like systems. However, I think that would be more difficult than it seems at first glance despite NetBSD being a Unix-like general-purpose OS like QNX. Even though QNX looks superficially similar to conventional Unix-like OSes in a lot of ways and is quite compatible with them, it is really quite different from them in some pretty important ways. One of the biggest of these is the IPC model. QNX's IPC essentially acts as a cross-address space function call, and not a one-way message queue. When a client process sends a message to a server over a channel, the remainder of its timeslice is transferred directly to the server process and a context switch occurs immediately, entirely bypassing the scheduler queue in most cases. The client process is blocked while the server processes the message. Once the server is done, it sends a reply (rather than specifying the channel, it instead specifies the message ID that it got when it received the message), and the same direct context switch happens again in the opposite direction, and the client is unblocked. It is possible for a server to receive further messages even if it has previous messages that it hasn't replied to. A collection of data buffers of arbitrary size and location may be transferred in both directions; these buffers are copied directly from the address space of the sender to that of the receiver with no intermediary buffers in kernel space (specified by a readv()/writev()-style vector). seL4's IPC already has basically identical call/receive/reply with direct context switch semantics to QNX, although it is limited to copying between per-thread single-page buffers rather than arbitrary vectors. In my (currently unnamed) hard fork of seL4 I already have a working preliminary implementation of long IPC with arbitrary vectors, and while dealing with seL4's unconventional preemption semantics was a little tricky, it wasn't especially difficult to add long copying to the existing IPC layer. On the other hand, from what I've seen from looking at the NetBSD sources, implementing QNX-style IPC would require writing a complete IPC layer from scratch and making several modifications to the scheduler and virtual memory manager (trying to port a QNX/L4-style IPC layer from a kernel that already has one probably wouldn't work, since they're pretty tightly integrated). Also, the general architecture is quite different. In order to get something that is QNX-like enough for my liking, the vast majority of subsystems would have to be removed from the kernel and moved into separate user processes. Really it would have to be sort of like an inverse rump kernel where just the inner kernel with the scheduler, IPC, virtual memory management and some parts of the VFS would be left. By the time I'm done, I'm not sure there would be much code left untouched, and I think it might be easier to just continue on my current path of using an seL4 fork and an original root server. I may incorporate more code from other OSes into my root server, since I've already done a little bit of that, but trying to wholesale convert something like NetBSD into the kind of OS I want seems like it might not be worth it. There have been monolithic-to-microkernel conversions done in the past, although all of the ones I'm aware of are either done with monolithic kernels that were designed from the start to be converted (e.g. early Mach kernels), are "serverizations" where the inner parts of the monolithic kernel like scheduling and basic memory management are removed and the remainder is ported to a purpose-built microkernel (e.g. LP49, Lites, and possibly MkLinux), or are just treating the microkernel as a hypervisor (e.g. L4Linux). I'm not aware of any monolithic-to-microkernel conversions where the outer subsystems of the kernel are removed and the inner kernel is retained as a microkernel happening, and I've read a lot on OS history and played around with most of the historical OSes I can get my hands on going back to ones from the 50s. The VFS architecture specifically is rather different between something like QNX and more conventional Unix-like systems. QNX's central VFS or pathname manager (part of the process server) is much simpler than a conventional Unix VFS; it pretty much just matches path prefixes and hands out a connection to the channel ID of the server handling the one that fits the path best; the open request to the server contains the remainder of the path rather than an inode number. Server calls generally bypass the VFS entirely and go straight from the client through the microkernel to the server. The VFS I'm planning to implement will be a little more conventional in order to allow for stronger security (QNX's channel IDs
