Hi all, Eelco and I are having a slight disagreement on how to proceed on a PR of mine and we figured we'd ask the community for opinions. If you've dealt with images in Nix and/or virtual machines, please continue reading.
*tl;dr:* please give thumbs up or down on https://github.com/NixOS/nixpkgs/pull/23026 Here's a quick summary of what's going on: - NixOS/nixpkgs has a remarkable amount of image building code spread throughout it, for building EC2/Google/Azure images, cd/dvd images, initramfs, and probably several others. - That code is generally copied and pasted a lot, but almost all of the image builders currently run the filesystem build inside a Linux VM; the usual process is: 1. start Linux VM (because writing filesystems requires root and is mostly "impure") 2. set up partitions & filesystems 3. run nixos-install 4. shut down VM 5. save image to $out - Starting a Linux VM inside a Nix build isn't always possible or efficient. For one example, 95% of "the cloud" doesn't support nested virtualization, so running an image build on EC2 is fully emulated and takes a decimal order of magnitude or more longer than it does on physical hardware, due to the lack of hardware-assisted virtualization. There's no indication that any cloud providers will be be providing nested virtualization support anytime soon. A second example is that nested virtualization support in e.g., VMWare can be pretty flaky on Linux guests: I used to run nested virtualization fine inside NixOS but some recent Linux kernel changes now hang my system when I try it, and there doesn't seem to be any indication that there's active work on remedying such flakiness. - I build NixOS images *a lot* (mostly for EC2, on EC2) and don't have access to any physical Linux hardware. - I have devised a cute process <https://github.com/NixOS/nixpkgs/pull/21943> (using the Linux Kernel Library project) that can quickly and efficiently craft Linux filesystems using the upstream Linux kernel libraries directly, without a VM. Here's what I'm proposing: 1. Make the image builder code throughout NixOS (I'll also refactor and share more code between the image generators over the next few weeks) use LKL and do far less work in a VM, in order to build images efficiently on EC2 (image build time goes from 30+ minutes to <1 minute) and to build them at all on local virtualization. This means *the image builder code no longer calls nixos-install*. 2. To avoid duplicating all the logic in nixos-install, factor out the common elements in the VM-free image builder and nixos-install into a third script, which I've been tentatively calling nixos-prepare-root, and have nixos-install and image builders both call that. Eelco is understandably concerned that by forking the image build and making it take a slightly different path from nixos-install (even if they share most of the core filesystem build logic), we'll make maintenance harder and make it easier to introduce subtle bugs. Although I agree with the gist of that and am striving to minimize divergence from nixos-install, I still need it to work on EC2 and VMWare. Eelco has said he won't veto the PR merge (and my subsequent work) if the community supports it, but doesn't particularly like it himself. That's where you come in :) please check out my PR and let us know what you think. I'd obviously love to merge this, but if the community really doesn't want this change I can maintain some forked logic here for my needs. Thanks, Dan
_______________________________________________ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev