On Wed, Sep 2, 2026 at 10:32 AM Edouard Klein <[email protected]> wrote:
> My understanding of the benefits is that by removing interrupts and > context switching you gain a surprising amount of performance for > userspace tasks, each of which gets a basically dedicated core from > start to finish (with all its cache). Is that correct ? > > yes, but there is the further question of avoiding OS noise, which can come up in surprising ways. On the $120M 2048-node ASCI Q machine, when we started using it in 2003, we were only scaling to about 2/3 of the machine. One problem was a 1/30 HZ interference (yes, once every 30 seconds) from ... the line printer daemon, which the vendor (DEC) had left enabled for some reason. LPD woke up, said "I don't have any printers anyway", and went back to sleep. That little loop was expensive! So think about that ... 1/30 HZ. As you scale up, things that would not have been an issue can dominate. 2048 machines is small. Fugaku had 128K nodes. Blue Gene had 64K nodes and one million cores. Careful management of interrupts and OS tasks is critical to preserve your $500M investment. The general rule nowadays seems to gravitate to fat nodes, 16K or so, with many cores, partly to avoid network noise issues. The OS image is heavy, and gets its 4k cores; lots of Linux cpu isolation tricks are used to keep app cores quiet. It's a lot of effort. NIX did the same with far less work. CPU isolation from interrupts is good. Until it's not: https://github.com/rminnich/ftq/blob/master/Documentation/analyses/IPC_induced_bands.md Short form: the better the isolation got, the worse the noise got, due to memory renaming. Memory renaming? See the doc. It was new to me, that's for sure. Look for "isolation makes it worse" -- note that, you really need low variance in the loop, not absolute best performance, and memory renaming messes that up. So it's never simple, but NIX helps a lot with noise. When we first started measuring noise in 2011, the app cores had perfect noise -- the only noise we saw was from the lowest order bit, i.e. we were at the limit of clock precision. Pretty shocking: a general purpose kernel, plan 9, was as good as the special purpose Blue Gene Compute Node Kernel. but, anyway: can somebody take that .img or that kernel in there and get something to boot it? I'd love to see it happen. ron > > Ron Minnich <[email protected]> writes: > > > from a book: > > > > "The NIX project targeted manycore architectures in which only a small > > subset of the cores could run kernel code. Consider N cores on a die > > arranged in a grid, with an on-die network providing a totally > > symmetric network and access to common memory. Let us assume that only > > of the cores can run a kernel and the rest can only run user mode > > code. NIX was intended to provide a kernel for such a system. The > > hypothetical target was a 1,024 core system, as several vendors hinted > > that was reasonable in the 2016 time frame. We picked this number as > > it was large enough to far exceed any possible number of user > > processes: the Plan 9 systems on which NIX was based use only a small > > fraction of this number. Further, the realities of memory bandwidth > > will limit the number of active cores to small single digits. We also > > wanted to have a system with good support for Go, the then-new > > language from Google. At the time, Go was a statically linked language > > and a typical binary was about 1 MiB. This influenced the design of > > the virtual memory system in NIX. The base page size was expanded from > > 4,096 bytes to 2 MiB. We further extended Plan 9’s virtual memory > > model to transparently support GiB pages, but via a simple heuristic > > that made it far less complex than existing systems. The target system > > had a 32:1 ratio of application cores to kernel cores (a close match > > to the Blue Gene/Q system at 16:1), and it was designed for space > > sharing, not time sharing, and had a minimum page size of 2 MiB; > > supported GiB pages; and yet, at the same time, looked in all other > > ways like a standard Plan 9 system." > > > > > > note that the Esperanto system, released in 2017 or so, closely > > matches this model, as do other risc-v systems. I have a few of these > > cards, so may take a look at whether it might work. > > > > On Tue, Sep 1, 2026 at 1:38 PM <[email protected]> wrote: > >> > >> For context, what was NIX? How did it differ from Plan 9 4th edition? > What were the goals? > >> 9fans / 9fans / see discussions + participants + delivery options > Permalink ------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/Ta20bdef5171bc207-Mdec93425c0dc5ebf8b5e3be8 Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
