Re: [Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
Two other observations about adding such a feature, where GC roots are added and removed by sending messages to the daemon: - Since the daemon would now know exactly when a GC root was removed, this would allow for a time-based garbage collection policy, where a GC only collects things if they have been unreferenced for at least an hour, or a week, or any other period of time. - I believe this would make our GC approach be an example of "partial tracing" as described in Bacon 2004. "A Unified Theory of Garbage Collection", https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf which would be interestingly novel, since they say "We know of no implementation of partial tracing." ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
Tuomas Tynkkynen writes: > How would deleting indirect roots work? Keep in mind that currently > users can simply delete any result symlinks from nix-build with 'rm' or > any other file management tool to remove the root. That won't work > over NFS because such an indirect root could be created on a different > filesystem (e.g. any local fs) that the machine running nix-daemon can't > access to verify whether the symlink is still there. > > Additionally there's the problem of paths being in use by running processes > on other machines being garbage collected under their feet. Both of these could be handled client-side. Instead of having a per-user roots directory in /nix/var/nix, we would have some truly per-user directory (in /home/user/.var/nix/ or something) where roots are put. We'd also have some kind of state tracking for what roots we've told the nix-daemon about. Then when we one runs nix-store --gc, it would: 1. Clean up broken links in .var/nix/roots 2. Search for runtime roots as the current user (this, incidentally, is a critical feature on its own; my systems run with an unprivileged nix-daemon and hidepid=2, so the nix-daemon (as it should be) cannot determine what programs other users are running) 3. Tell the nix-daemon to remove any roots that were previously registered and are no longer present, excluding roots that are currently runtime roots. Unfortunately, this would mean that if a user runs a program out of the store, without first contacting the daemon and registering a (temporary) root, the store paths for that program could be GC'd out from under it. But I think that makes sense, anyway - the runtime root stuff inherently requires a lot of privileges for the nix-daemon, which are impractical/inadvisable to grant it in containerized/multi-host setups. Much better to move runtime root detection to be as local as possible. This would be an optional mode of operation, of course, global runtime roots are still helpful if you can get them. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
Kevin Cox writes: > So are you also going to stop the user from building new things? You > would also have the problem if the user was inside their quota then > started building something very large. Hmm, in that situation the build should fail, and that failure would be propagated back up to the user. Though that could be difficult to implement, I agree, because builds don't necessarily take place using the user's resource quota. Things can get better if/when we use an intensional store, where the user can perform the build themselves rather than have the nix-daemon perform it on their behalf. > I guess it wouldn't be bad to start allowing this sort of enforcement > but I'm not sure how useful it would be in practice. It's true, thinking it over more, I guess Unix traditionally doesn't control creating new GC roots: You can hardlink anything you can read, and dup() any file descriptor at all. So this kind of store quota policy would be somewhat unnatural in Unix. I don't actually need/want it myself, it was just an idea that I thought would be neat to support. (The other uses of creating-store-roots-over-IPC still stand) > If you have two requests from the same user how do you segment them into > container or some other division? If you can restrict the containers in > some way couldn't you do this yourself by putting the links somewhere > the container can't access? You would pass in a proxy to the container. That is, you'd mount the store into the container, as well as a socket which points to a policy-implementing daemon. That policy-implementing daemon would forward requests on to the real daemon. > Same question as above, these sound like great goals but I don't see how > the Nix daemon is going to be able to divide the requests according to > capability. Again, we could do this through proxying requests through a policy-implementing daemon, which could identify requests based on whatever authentication mechanism it wants. To be clear I'm not suggesting we actually add a ton of policy-logic to nix-daemon itself, sorry for not saying that at the start. But if we added the ability for GC roots to be created through IPC requests, that would allow others to put policy in between, through whatever means they wish. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
Kevin Cox writes: > On 18/06/17 03:49, sba...@catern.com wrote: >> >> It seems to me that it would be useful for the daemon to handle making >> GC roots, and forbid users to directly create GC roots. >> >> The features that come to mind: >> - Allows later implementing policy about GC roots/space consumption >> - Allows avoiding complicated locking around doing GC >> - Allows /nix to be put on network storage transparently >> - Allows /nix to be shared between containers transparently > > I'm not sure just this will solve your problem, however it might be a > good start. For example if the daemon only created indirect roots then > each root is accounted for (based on where it points) and can be cleaned > up by either the system or the user. > > That being said I'm not sure this is necssary. You can already > inspect/delete roots by a given user to see how much space they are > "using". Yes, but that's a post-hoc check which can't really be made to work reliably. What do you do if a user is using too much space? Delete their stuff without any warning? That's unworkable, instead there should be an error at the time that the user tries to consume excess space. (Admittedly the policy around this will be tricky, but currently we don't even have the mechanism to write such policy). > I'm curious what you would want to do to help with containers, > or to help with putting nix on network storage. I don't see how this is > required. It helps because it allows moving away from user-based control over GC roots. Imagine two separate independent services running as the same user in containers built with Nix. The two services have to run as the same user because we don't have privileges on the system to create new users. Those two services shouldn't be allowed to remove each others' GC roots or add GC roots using each others' credentials. We should be able to give the services conditional access to creating GC roots, without full access to a per-user directory; we can do that if GC roots are created through IPC. A similar issue exists for multiple hosts sharing a store. Those hosts shouldn't need to have the exact same set of users, just to share a store. I should be able to publish my store so it is publicly accessible to the world at large, and then allow people to connect in (authenticating in some way) to the nix-daemon managing that store and register new GC roots so they can maintain deployments out of that global store. This is a lot easier than requiring the use of NFS with POSIX user accounts which authenticate via Kerberos (or don't authenticate at all, in traditional NFS). The basic issue is tying control over GC roots to Unix user accounts. Per-user software deployment is great and a big advance over root-only software deployment, but Unix users have two fatal practical flaws: They require privileges to create and they don't straightforwardly work across the network. This hurts using a single store for multiple containers on a single box, as well as using a single store across an untrusting network. And the more fundamental issue is that Unix users aren't compatible with a capability-security model, which hopefully Nix can help push the world towards. I would like to be able to give out a capability for creating GC roots; that's not possible with the current user-based mechanism. Hopefully that makes sense? I am fairly new to Nix, but I'm coming at this from a capability-security mindset, and want to make sure Nix is as compatible as possible with cap-security. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
Wout Mertens writes: > Another issue is that right now, nix does not /require/ the daemon to > work, and this proposal would change that. To clarify I'm just talking about this for multi-user setups, where the daemon is already required. > However, you can totally share /nix between multiple hosts, you just > have to pinkie-promise not to write to it from multiple hosts at the > same time. Yes, this way (if the daemon also had the ability to talk to the database) you don't need that pinkie promise. But this is just one benefit of letting the daemon handle GC roots, I'm interested also in the other benefits. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots
My understanding is that currently GC roots (symlinks in profiles/gcroots) are created and deleted directly by the various Nix tools, even in multi-user configurations. (whether on NixOS or on another Linux distribution) It seems to me that it would be useful for the daemon to handle making GC roots, and forbid users to directly create GC roots. The features that come to mind: - Allows later implementing policy about GC roots/space consumption - Allows avoiding complicated locking around doing GC - Allows /nix to be put on network storage transparently - Allows /nix to be shared between containers transparently The network-storage-/nix use case may be the most important, since there seems to be a lot of people who want to put /nix on NFS. Thoughts? Has this been considered? Thanks for Nix! ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
