Re: [Nix-dev] Multiple machines with nix store shared via NFS

[Renato Alves]

Hi Erik,

This is a different question. As far as I understand it NixOps
replicates a machine configuration. It copies the store. I want the
exact opposite. One store shared across all using NFS.

Anyway just for the record in case someone goes down the same road.

At the time there is no official/supported way to achieve what I wanted.
nix-daemon relies on communication via a socket which is only available
on the machine running that daemon.

What I ended up going for is:
https://gist.github.com/Unode/2e41d8d2766580612d8996aa4f2887f7

Not exactly what I was hoping for but works for now until a better
solution is developed.

Renato


On 03/07/16 22:51, 4levels wrote:
> Hi Renato,
> 
> I'm no expert whatsoever here, but isn't NixOps doing this in a way? At
> least on my nixos deployed servers all stores are kept in sync, only the
> server-specific packages are unique for each machine.
> Don't ask me how it works, I can only confirm that it does ;-)
> 
> Just my 2 cents..
> 
> Kind regards,
> 
> Erik
> 
> 
> On Fri, Jul 1, 2016, 23:16 Renato Alves  > wrote:
> 
> Had a look at the links. The 2011 thread mentions problems that have
> been addressed by either nix-daemon or remote-build nodes.
> 
> There's also some references to socat but not much. Seems like
> forwarding could indeed be a solution but from what I read elsewhere
> it's a fragile setup and requires ssh-agent and passwordless keys.
> 
> There's also some articles about using haproxy and nginx to proxy
> sockets but it's all very much ad-hoc. I'm also not sure how this would
> work given the security model of nix-daemon.
> 
> The 2014 thread is also a good read but focused on using the cluster
> infrastructure to compile things. I don't want to go down that road yet.
> Perhaps in the future if necessary.
> At the moment I'd be happy with having nix-env and nix-shell on all
> machines.
> 
> Is there currently any support for NIX_REMOTE specifying a connection to
> a daemon on a different host? (i.e. literally REMOTE :)
> 
> R
> 
> 
> 
> On 01/07/16 22:34, Layus wrote:
> > I never sait it was a good idea, just soemthing worth investigating.
> >
> > Now, this has been discussed on the ML already
> >
> > See for example
> > [Nix-dev] Sharing the Nix store between systems
> >
>  
> /Rickard
> > Nilsson
> > /[Nix-dev] nix on compute cluster?
> >
> 
> //Andreas
> > Herrmann
> >
> > //And more generally Google search on
> > site:http://lists.science.uu.nl/pipermail/nix-dev/ :-).
> >
> > -- Layus.
> >
> > On 01/07/16 21:55, Renato Alves wrote:
> >> I never tried it with sockets, only tcp ports.
> >>
> >> I need to move the socket to a different location. Currently it also
> >> sits on NFS. I can play with symlinks making it point to a local
> disk.
> >>
> >> The entire setup sounds terribly hackish though.
> >>
> >> If anyone has better alternatives I'd love to hear them.
> >>
> >> Thanks,
> >> R
> >>
> >> On 01/07/16 20:51, Layus wrote:
> >>> Just one idea off the top of my head: Would it be possible to
> forward
> >>> the unix socket of the daemon ?
> >>>
> >>> -- Layus ?
> >>>
> >>> On 01/07/16 15:10, Renato Alves wrote:
>  Hi everyone,
> 
> 
>  Recently I've been trying to setup nix on a multiple-machine shared
>  environment. At the moment I have one machine setup with nix-daemon
>  running from a store in a custom location (/shared/myuser/nix).
> This
>  location is shared via NFS with several other machines. This is all
>  still pretty new to me as I've only recently started using nix and
>  haven't used nixos yet.
> 
>  Before setting up nix-daemon I ran into some problems with database
>  corruption (sqlite) due to concurrent processes. With
> nix-daemon these
>  have gone away.
> 
>  One limitation that I identified recently is that on any of the
> other
>  machines that do not have nix-daemon running (but have read
> access to
>  the store) I can run commands from the store without problems but
>  cannot, for instance, use nix-shell.
>  So I thought of spawning a nix-daemon instance on those
> machines but I'm
>  not sure what is the best way to set it up without risking DB
> corruption
>  again.
> 
>  These are the requirements I'd like to meet:
>   * /shared/myuser/nix is shared across multiple machines with
> the same
>  architecture (x86_64-linux).
>   * All machines should be able to 

Re: [Nix-dev] Multiple machines with nix store shared via NFS

[4levels]

Hi Renato,

I'm no expert whatsoever here, but isn't NixOps doing this in a way? At
least on my nixos deployed servers all stores are kept in sync, only the
server-specific packages are unique for each machine.
Don't ask me how it works, I can only confirm that it does ;-)

Just my 2 cents..

Kind regards,

Erik

On Fri, Jul 1, 2016, 23:16 Renato Alves  wrote:

> Had a look at the links. The 2011 thread mentions problems that have
> been addressed by either nix-daemon or remote-build nodes.
>
> There's also some references to socat but not much. Seems like
> forwarding could indeed be a solution but from what I read elsewhere
> it's a fragile setup and requires ssh-agent and passwordless keys.
>
> There's also some articles about using haproxy and nginx to proxy
> sockets but it's all very much ad-hoc. I'm also not sure how this would
> work given the security model of nix-daemon.
>
> The 2014 thread is also a good read but focused on using the cluster
> infrastructure to compile things. I don't want to go down that road yet.
> Perhaps in the future if necessary.
> At the moment I'd be happy with having nix-env and nix-shell on all
> machines.
>
> Is there currently any support for NIX_REMOTE specifying a connection to
> a daemon on a different host? (i.e. literally REMOTE :)
>
> R
>
>
>
> On 01/07/16 22:34, Layus wrote:
> > I never sait it was a good idea, just soemthing worth investigating.
> >
> > Now, this has been discussed on the ML already
> >
> > See for example
> > [Nix-dev] Sharing the Nix store between systems
> > 
> /Rickard
> > Nilsson
> > /[Nix-dev] nix on compute cluster?
> > 
> //Andreas
> > Herrmann
> >
> > //And more generally Google search on
> > site:http://lists.science.uu.nl/pipermail/nix-dev/ :-).
> >
> > -- Layus.
> >
> > On 01/07/16 21:55, Renato Alves wrote:
> >> I never tried it with sockets, only tcp ports.
> >>
> >> I need to move the socket to a different location. Currently it also
> >> sits on NFS. I can play with symlinks making it point to a local disk.
> >>
> >> The entire setup sounds terribly hackish though.
> >>
> >> If anyone has better alternatives I'd love to hear them.
> >>
> >> Thanks,
> >> R
> >>
> >> On 01/07/16 20:51, Layus wrote:
> >>> Just one idea off the top of my head: Would it be possible to forward
> >>> the unix socket of the daemon ?
> >>>
> >>> -- Layus ?
> >>>
> >>> On 01/07/16 15:10, Renato Alves wrote:
>  Hi everyone,
> 
> 
>  Recently I've been trying to setup nix on a multiple-machine shared
>  environment. At the moment I have one machine setup with nix-daemon
>  running from a store in a custom location (/shared/myuser/nix). This
>  location is shared via NFS with several other machines. This is all
>  still pretty new to me as I've only recently started using nix and
>  haven't used nixos yet.
> 
>  Before setting up nix-daemon I ran into some problems with database
>  corruption (sqlite) due to concurrent processes. With nix-daemon these
>  have gone away.
> 
>  One limitation that I identified recently is that on any of the other
>  machines that do not have nix-daemon running (but have read access to
>  the store) I can run commands from the store without problems but
>  cannot, for instance, use nix-shell.
>  So I thought of spawning a nix-daemon instance on those machines but
> I'm
>  not sure what is the best way to set it up without risking DB
> corruption
>  again.
> 
>  These are the requirements I'd like to meet:
>   * /shared/myuser/nix is shared across multiple machines with the same
>  architecture (x86_64-linux).
>   * All machines should be able to use nix-shell or any nix related
>  command, including requesting installation of packages.
>   * Only some machines are allowed to compile. Machines that are not
>  allowed should rely on something like remote-systems.conf to
> distribute
>  work.
> 
>  Is this kind of setup currently possible with nix? If so, can someone
>  provide some guidance?
> 
> 
>  Thanks,
>  Renato
> 
> 
> 
>  ___
>  nix-dev mailing list
>  nix-dev@lists.science.uu.nl
>  http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>>
> >>>
> >>>
> >>> ___
> >>> nix-dev mailing list
> >>> nix-dev@lists.science.uu.nl
> >>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>>
> >>
> >>
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
> >
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > 

Re: [Nix-dev] Multiple machines with nix store shared via NFS

[Azul]

how would you rollback a broken system if the /nix store was remote and the
box wouldnt boot or have no network ?

On 1 July 2016 at 21:34, Layus  wrote:

> I never sait it was a good idea, just soemthing worth investigating.
>
> Now, this has been discussed on the ML already
>
> See for example
> [Nix-dev] Sharing the Nix store between systems
> 
> *Rickard Nilsson *[Nix-dev] nix on compute cluster?
> 
>
> * Andreas Herrmann *And more generally Google search on site:
> http://lists.science.uu.nl/pipermail/nix-dev/ :-).
>
> -- Layus.
>
>
> On 01/07/16 21:55, Renato Alves wrote:
>
> I never tried it with sockets, only tcp ports.
>
> I need to move the socket to a different location. Currently it also
> sits on NFS. I can play with symlinks making it point to a local disk.
>
> The entire setup sounds terribly hackish though.
>
> If anyone has better alternatives I'd love to hear them.
>
> Thanks,
> R
>
> On 01/07/16 20:51, Layus wrote:
>
> Just one idea off the top of my head: Would it be possible to forward
> the unix socket of the daemon ?
>
> -- Layus ?
>
> On 01/07/16 15:10, Renato Alves wrote:
>
> Hi everyone,
>
>
> Recently I've been trying to setup nix on a multiple-machine shared
> environment. At the moment I have one machine setup with nix-daemon
> running from a store in a custom location (/shared/myuser/nix). This
> location is shared via NFS with several other machines. This is all
> still pretty new to me as I've only recently started using nix and
> haven't used nixos yet.
>
> Before setting up nix-daemon I ran into some problems with database
> corruption (sqlite) due to concurrent processes. With nix-daemon these
> have gone away.
>
> One limitation that I identified recently is that on any of the other
> machines that do not have nix-daemon running (but have read access to
> the store) I can run commands from the store without problems but
> cannot, for instance, use nix-shell.
> So I thought of spawning a nix-daemon instance on those machines but I'm
> not sure what is the best way to set it up without risking DB corruption
> again.
>
> These are the requirements I'd like to meet:
>  * /shared/myuser/nix is shared across multiple machines with the same
> architecture (x86_64-linux).
>  * All machines should be able to use nix-shell or any nix related
> command, including requesting installation of packages.
>  * Only some machines are allowed to compile. Machines that are not
> allowed should rely on something like remote-systems.conf to distribute
> work.
>
> Is this kind of setup currently possible with nix? If so, can someone
> provide some guidance?
>
>
> Thanks,
> Renato
>
>
>
> ___
> nix-dev mailing 
> listnix-...@lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
>
> ___
> nix-dev mailing 
> listnix-...@lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
> ___
> nix-dev mailing 
> listnix-...@lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Multiple machines with nix store shared via NFS

[Layus]

I never sait it was a good idea, just soemthing worth investigating.

Now, this has been discussed on the ML already

See for example
[Nix-dev] Sharing the Nix store between systems 
/Rickard 
Nilsson
/[Nix-dev] nix on compute cluster? 
//Andreas 
Herrmann


//And more generally Google search on 
site:http://lists.science.uu.nl/pipermail/nix-dev/ :-).


-- Layus.

On 01/07/16 21:55, Renato Alves wrote:

I never tried it with sockets, only tcp ports.

I need to move the socket to a different location. Currently it also
sits on NFS. I can play with symlinks making it point to a local disk.

The entire setup sounds terribly hackish though.

If anyone has better alternatives I'd love to hear them.

Thanks,
R

On 01/07/16 20:51, Layus wrote:

Just one idea off the top of my head: Would it be possible to forward
the unix socket of the daemon ?

-- Layus ?

On 01/07/16 15:10, Renato Alves wrote:

Hi everyone,


Recently I've been trying to setup nix on a multiple-machine shared
environment. At the moment I have one machine setup with nix-daemon
running from a store in a custom location (/shared/myuser/nix). This
location is shared via NFS with several other machines. This is all
still pretty new to me as I've only recently started using nix and
haven't used nixos yet.

Before setting up nix-daemon I ran into some problems with database
corruption (sqlite) due to concurrent processes. With nix-daemon these
have gone away.

One limitation that I identified recently is that on any of the other
machines that do not have nix-daemon running (but have read access to
the store) I can run commands from the store without problems but
cannot, for instance, use nix-shell.
So I thought of spawning a nix-daemon instance on those machines but I'm
not sure what is the best way to set it up without risking DB corruption
again.

These are the requirements I'd like to meet:
  * /shared/myuser/nix is shared across multiple machines with the same
architecture (x86_64-linux).
  * All machines should be able to use nix-shell or any nix related
command, including requesting installation of packages.
  * Only some machines are allowed to compile. Machines that are not
allowed should rely on something like remote-systems.conf to distribute
work.

Is this kind of setup currently possible with nix? If so, can someone
provide some guidance?


Thanks,
Renato



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev




___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev




___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Multiple machines with nix store shared via NFS

[Layus]

Just one idea off the top of my head: Would it be possible to forward 
the unix socket of the daemon ?


-- Layus ?

On 01/07/16 15:10, Renato Alves wrote:

Hi everyone,


Recently I've been trying to setup nix on a multiple-machine shared
environment. At the moment I have one machine setup with nix-daemon
running from a store in a custom location (/shared/myuser/nix). This
location is shared via NFS with several other machines. This is all
still pretty new to me as I've only recently started using nix and
haven't used nixos yet.

Before setting up nix-daemon I ran into some problems with database
corruption (sqlite) due to concurrent processes. With nix-daemon these
have gone away.

One limitation that I identified recently is that on any of the other
machines that do not have nix-daemon running (but have read access to
the store) I can run commands from the store without problems but
cannot, for instance, use nix-shell.
So I thought of spawning a nix-daemon instance on those machines but I'm
not sure what is the best way to set it up without risking DB corruption
again.

These are the requirements I'd like to meet:
  * /shared/myuser/nix is shared across multiple machines with the same
architecture (x86_64-linux).
  * All machines should be able to use nix-shell or any nix related
command, including requesting installation of packages.
  * Only some machines are allowed to compile. Machines that are not
allowed should rely on something like remote-systems.conf to distribute
work.

Is this kind of setup currently possible with nix? If so, can someone
provide some guidance?


Thanks,
Renato



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Multiple machines with nix store shared via NFS

[Renato Alves]

The number of machines is too large to have a nix store on each of them,
both in terms of maintenance as well as storage.
Some of these machines are clusters so NFS is necessary at least within
cluster nodes.

R



On 01/07/16 15:21, Domen Kožar wrote:
> In my experience, NFS is not worth the trouble. Are you absolutely sure
> that you need it?
> 
> On Fri, Jul 1, 2016 at 3:10 PM, Renato Alves  > wrote:
> 
> Hi everyone,
> 
> 
> Recently I've been trying to setup nix on a multiple-machine shared
> environment. At the moment I have one machine setup with nix-daemon
> running from a store in a custom location (/shared/myuser/nix). This
> location is shared via NFS with several other machines. This is all
> still pretty new to me as I've only recently started using nix and
> haven't used nixos yet.
> 
> Before setting up nix-daemon I ran into some problems with database
> corruption (sqlite) due to concurrent processes. With nix-daemon these
> have gone away.
> 
> One limitation that I identified recently is that on any of the other
> machines that do not have nix-daemon running (but have read access to
> the store) I can run commands from the store without problems but
> cannot, for instance, use nix-shell.
> So I thought of spawning a nix-daemon instance on those machines but I'm
> not sure what is the best way to set it up without risking DB corruption
> again.
> 
> These are the requirements I'd like to meet:
>  * /shared/myuser/nix is shared across multiple machines with the same
> architecture (x86_64-linux).
>  * All machines should be able to use nix-shell or any nix related
> command, including requesting installation of packages.
>  * Only some machines are allowed to compile. Machines that are not
> allowed should rely on something like remote-systems.conf to distribute
> work.
> 
> Is this kind of setup currently possible with nix? If so, can someone
> provide some guidance?
> 
> 
> Thanks,
> Renato
> 
> 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl 
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> 
> 



signature.asc
Description: OpenPGP digital signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev