Re: [lxc-users] nfs server in [unprivileged] container?
On Thu, Mar 30, 2017 at 12:47 PM, Marat Khalili wrote: > The story would be all different for userspace nfs server, but apparently > there's none. > > https://launchpad.net/~gluster/+archive/ubuntu/nfs-ganesha Disclamer: I haven't tested it. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] nfs server in [unprivileged] container?
I do run nfs in a privileged container, mostly because it is easier to manage it this way (separate IP-address and such -- reasons similar to yours actually). Since I use nfs-kernel-server, most (if not all) of the code is actually executed in kernel, not in container userspace. Also, I had to disable apparmor for this container (lxc.aa_profile = unconfined). Because of this, I'm not sure if trying unprivileged nfs container makes any sense. The story would be all different for userspace nfs server, but apparently there's none. -- With Best Regards, Marat Khalili ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD - Small Production Deployment - Storage
On Thu, Mar 30, 2017 at 12:01:14AM +0200, Gabriel Marais wrote: > On Wed, Mar 29, 2017 at 6:01 PM, Stéphane Graber > wrote: > > > On Wed, Mar 29, 2017 at 03:13:36PM +0200, Gabriel Marais wrote: > > > Hi Guys > > > > > > If this is the incorrect platform for this post, please point me in the > > > right direction. > > > > > > We are in the process of deploying a small production environment with > > the > > > following equipment:- > > > > > > 2 x Dell R430 servers each with 128GB Ram and 3 x 600GB SAS 10k drives > > > 1 x Dell PowerVault MD3400 with > > > 3 x 600GB 15k SAS Drives > > > 3 x 6TB 7.2k Nearline SAS drives > > > > > > The PowerVault is cabled directly to the Host Servers via Direct Attached > > > Storage, redundantly. > > > > > > > > > We would like to run a mixture of KVM and LXD containers on both Host > > > Servers. > > > > > > The big question is, how do we implement the PowerVault (and to a certain > > > extent the storage on the Host Servers themselves) to be most beneficial > > in > > > this mixed environment. > > > > > > I have a few ideas on what I could do, but since I don't have much > > > experience with shared storage, I am probably just picking straws and > > would > > > like to hear from others that probably has more experience than me. > > > > Hi, > > > > I'm not particularly familiar with the DELL PowerVault series, but it > > looks like the other answers you've received so far have entirely missed > > the "Direct Attached Storage" part of your description :) > > > > For others reading this thread, this setup will effectively show up on > > both servers as directly attached disks through /dev/mapper (because of > > multipath), there is no need to use any kind of networked storage on top > > of this. > > > > > > The answer to your question I suspect will depend greatly on whether > > you're dealing with a fixed number of VMs and containers, or if you > > intend to spawn and delete them frequently. And also on whether you need > > fast (no copy) migration of individual VMs and containers between the > > two hosts. > > > > We are not planning on having a fixed number of VMs and containers. It will > always grow as the requirement grow and as many as CPU and RAM will allow. > Migration of containers seem easy enough using the lxc migrate command, > although we have only done migrations with containers running on a normal > file system (e.g ext4) on a host. > > > > > > One approach is to have a physical partition per virtual machine. > > > > Meaning, to create physical partitions on the controller and assign it to > both hosts. So lets say we create a physical partitions on the controller, > 60GB to be used for a specific VM. That partition will show up as e.g. sdd > on the host. We can then install the VM on that partition. If I understand > you correctly? Correct, though you should use something more unique than sdd as those device names are sequential based on detection order and so can change after reboot. Since you're in a HA setup, you should have multiple paths to each drive. Then something like multipathd will detect the different paths and setup a virtual device in /dev/mapper for you to use. That device is typically named after the WWN of the drive which is a unique stable identifier that you can rely on. > > With this, you can then access the drive from either host (obviously > > never from both at the same time), which means that should you want to > > start the VM on the other host, you just need to stop the kvm process on > > one and start it again on the other, without any data ever being moved. > > > > I assume we would simply use XML (export / import) to create the VM on the > other host which is pointing to the partition sitting on the storage device? Yep > > For containers, it's a bit trickier as we don't support using a raw > > block device as the root of the container. So you'd need LXD to either > > use the host's local storage for the container root and then mount block > > devices into those containers at paths that hold the data you care > > about. Or you'd need to define a block device for each server in the > > PowerVault and have LXD use that for storage (avoiding using the local > > storage). > > > > I like the idea of creating a block device on the storage for each > container and have LXD use that block device for a specific container. I'm > not sure how and if we would be able to simply migrate a container from one > host to the other (assuming that we would have those block devices > available to both hosts)...? Right so as I mentioned, you can't have a LXD container use a single partition from your storage array as kvm lets you do. So instead your best bet is probably to setup one chunk of storage for use for the container root filesystems and point LXD to that as the default storage pool. For container data, you can then use a partition from your storage array, format it and pass it to the LXD container with something like
Re: [lxc-users] LXD - Small Production Deployment - Storage
On Wed, Mar 29, 2017 at 6:01 PM, Stéphane Graber wrote: > On Wed, Mar 29, 2017 at 03:13:36PM +0200, Gabriel Marais wrote: > > Hi Guys > > > > If this is the incorrect platform for this post, please point me in the > > right direction. > > > > We are in the process of deploying a small production environment with > the > > following equipment:- > > > > 2 x Dell R430 servers each with 128GB Ram and 3 x 600GB SAS 10k drives > > 1 x Dell PowerVault MD3400 with > > 3 x 600GB 15k SAS Drives > > 3 x 6TB 7.2k Nearline SAS drives > > > > The PowerVault is cabled directly to the Host Servers via Direct Attached > > Storage, redundantly. > > > > > > We would like to run a mixture of KVM and LXD containers on both Host > > Servers. > > > > The big question is, how do we implement the PowerVault (and to a certain > > extent the storage on the Host Servers themselves) to be most beneficial > in > > this mixed environment. > > > > I have a few ideas on what I could do, but since I don't have much > > experience with shared storage, I am probably just picking straws and > would > > like to hear from others that probably has more experience than me. > > Hi, > > I'm not particularly familiar with the DELL PowerVault series, but it > looks like the other answers you've received so far have entirely missed > the "Direct Attached Storage" part of your description :) > > For others reading this thread, this setup will effectively show up on > both servers as directly attached disks through /dev/mapper (because of > multipath), there is no need to use any kind of networked storage on top > of this. > > > The answer to your question I suspect will depend greatly on whether > you're dealing with a fixed number of VMs and containers, or if you > intend to spawn and delete them frequently. And also on whether you need > fast (no copy) migration of individual VMs and containers between the > two hosts. > We are not planning on having a fixed number of VMs and containers. It will always grow as the requirement grow and as many as CPU and RAM will allow. Migration of containers seem easy enough using the lxc migrate command, although we have only done migrations with containers running on a normal file system (e.g ext4) on a host. > > One approach is to have a physical partition per virtual machine. > Meaning, to create physical partitions on the controller and assign it to both hosts. So lets say we create a physical partitions on the controller, 60GB to be used for a specific VM. That partition will show up as e.g. sdd on the host. We can then install the VM on that partition. If I understand you correctly? > With this, you can then access the drive from either host (obviously > never from both at the same time), which means that should you want to > start the VM on the other host, you just need to stop the kvm process on > one and start it again on the other, without any data ever being moved. > I assume we would simply use XML (export / import) to create the VM on the other host which is pointing to the partition sitting on the storage device? > > For containers, it's a bit trickier as we don't support using a raw > block device as the root of the container. So you'd need LXD to either > use the host's local storage for the container root and then mount block > devices into those containers at paths that hold the data you care > about. Or you'd need to define a block device for each server in the > PowerVault and have LXD use that for storage (avoiding using the local > storage). > I like the idea of creating a block device on the storage for each container and have LXD use that block device for a specific container. I'm not sure how and if we would be able to simply migrate a container from one host to the other (assuming that we would have those block devices available to both hosts)...? > > The obvious advantage of the second option is that should one of the > server go away for whatever reason, you'd be able to mount that server's > LXD pool onto the other server and spawn a second LXD daemon on it, > taking over the role of the dead server. > >From what I've read, a particular host can only use one ZFS Pool. This creates a limitation since we won't be able to create two pools - 1 for the faster storage drives and 1 for the slower storage drives. My initial planning was to:- Option 1 Take the Fast Storage (3 x 600GB 15k SAS drives) and configure them on the controller as RAID5, split them into 2 and give each host a partition (sdx) of +/- 600GB Take the Slower Storage (3 x 6TB 7.2k drives) and configure them on the controller as RAID5, split them into 2 and give each host a partition (sdy) of +/- 6TB Setup LVM with two volume groups, e.g. Vol0 - Fast Storage (300GB) and Vol1 (3TB) - Slow Storage Create logical volumes as needed in terms of disk space per container and VMs and install the container and VMs onto those logical volumes. Option 2 The other option I w
[lxc-users] nfs server in [unprivileged] container?
Dear all, I need to set up a nfs server and was considering doing it in a container, in part because it helps keeping thins in lxc, which is true for 99% of the stuff I have, but also because of the need to integrate with ldap etc, and the desire to avoid having all of that on the host that runs other containers. However, even based off this thread on the zfs ml, this seems to be a bad idea: http://list.zfsonlinux.org/pipermail/zfs-discuss/2017-March/027822.html is anybody here doing something like that that could share come tips? is this indeed a no-no? clients would not be on other containers on the same server btw. One option I was considering was to expose an additional volume from the host to the nfs-server container and if any other container needed it they could access it directly. But there is still the problem of privileged container and potential issues managing zfs from inside, which I'd want to be able to use zfs share and create datasets for users. thank you for any input, Spike ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD - Small Production Deployment - Storage
On Wed, Mar 29, 2017 at 03:13:36PM +0200, Gabriel Marais wrote: > Hi Guys > > If this is the incorrect platform for this post, please point me in the > right direction. > > We are in the process of deploying a small production environment with the > following equipment:- > > 2 x Dell R430 servers each with 128GB Ram and 3 x 600GB SAS 10k drives > 1 x Dell PowerVault MD3400 with > 3 x 600GB 15k SAS Drives > 3 x 6TB 7.2k Nearline SAS drives > > The PowerVault is cabled directly to the Host Servers via Direct Attached > Storage, redundantly. > > > We would like to run a mixture of KVM and LXD containers on both Host > Servers. > > The big question is, how do we implement the PowerVault (and to a certain > extent the storage on the Host Servers themselves) to be most beneficial in > this mixed environment. > > I have a few ideas on what I could do, but since I don't have much > experience with shared storage, I am probably just picking straws and would > like to hear from others that probably has more experience than me. Hi, I'm not particularly familiar with the DELL PowerVault series, but it looks like the other answers you've received so far have entirely missed the "Direct Attached Storage" part of your description :) For others reading this thread, this setup will effectively show up on both servers as directly attached disks through /dev/mapper (because of multipath), there is no need to use any kind of networked storage on top of this. The answer to your question I suspect will depend greatly on whether you're dealing with a fixed number of VMs and containers, or if you intend to spawn and delete them frequently. And also on whether you need fast (no copy) migration of individual VMs and containers between the two hosts. One approach is to have a physical partition per virtual machine. With this, you can then access the drive from either host (obviously never from both at the same time), which means that should you want to start the VM on the other host, you just need to stop the kvm process on one and start it again on the other, without any data ever being moved. For containers, it's a bit trickier as we don't support using a raw block device as the root of the container. So you'd need LXD to either use the host's local storage for the container root and then mount block devices into those containers at paths that hold the data you care about. Or you'd need to define a block device for each server in the PowerVault and have LXD use that for storage (avoiding using the local storage). The obvious advantage of the second option is that should one of the server go away for whatever reason, you'd be able to mount that server's LXD pool onto the other server and spawn a second LXD daemon on it, taking over the role of the dead server. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: PGP signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD - Small Production Deployment - Storage
On 2017-03-29 22:47, Marat Khalili wrote: Just don't use NFS clients using NFS server on the same machine (same kernel), as this will break (hangs). Huh? Works for me between LXC containers. Only had to tune startup/shutdown sequence in systemd. In what exactly situation does it hang? /worried/ Kernel deadlocks. Note that you can have NFS client on machineA and NFS server on machineB - this is fine, it's two different kernels. NFS client on machineA, and KVM guest with NFS server also on machineA is fine, too - two different kernels. NFS client on machineA and NFS server on machineA, same kernel (as is the case with LXD) - will eventually deadlock. The whole machine will not freeze out of the sudden, but you will notice you have more and more processes in D state, which you're not able to kill. See for example: https://lwn.net/Articles/595652/ It could be something improved in newer kernels, but I'm not aware of that. If you need to export NFS-like mounts on the same server, you can use gluster mount, which is userspace - and which has poor performance, not really suitable for containers, but should be OK for i.e. large files accessed directly (no recursive find etc.). Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD - Small Production Deployment - Storage
Just don't use NFS clients using NFS server on the same machine (same kernel), as this will break (hangs). Huh? Works for me between LXC containers. Only had to tune startup/shutdown sequence in systemd. In what exactly situation does it hang? /worried/ -- With Best Regards, Marat Khalili ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD - Small Production Deployment - Storage
On 2017-03-29 22:13, Gabriel Marais wrote: Hi Guys If this is the incorrect platform for this post, please point me in the right direction. We are in the process of deploying a small production environment with the following equipment:- 2 x Dell R430 servers each with 128GB Ram and 3 x 600GB SAS 10k drives 1 x Dell PowerVault MD3400 with 3 x 600GB 15k SAS Drives 3 x 6TB 7.2k Nearline SAS drives The PowerVault is cabled directly to the Host Servers via Direct Attached Storage, redundantly. We would like to run a mixture of KVM and LXD containers on both Host Servers. The big question is, how do we implement the PowerVault (and to a certain extent the storage on the Host Servers themselves) to be most beneficial in this mixed environment. I have a few ideas on what I could do, but since I don't have much experience with shared storage, I am probably just picking straws and would like to hear from others that probably has more experience than me. NFS for LXD and iSCSI for KVM? Just don't use NFS clients using NFS server on the same machine (same kernel), as this will break (hangs). Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] LXD - Small Production Deployment - Storage
Hi Guys If this is the incorrect platform for this post, please point me in the right direction. We are in the process of deploying a small production environment with the following equipment:- 2 x Dell R430 servers each with 128GB Ram and 3 x 600GB SAS 10k drives 1 x Dell PowerVault MD3400 with 3 x 600GB 15k SAS Drives 3 x 6TB 7.2k Nearline SAS drives The PowerVault is cabled directly to the Host Servers via Direct Attached Storage, redundantly. We would like to run a mixture of KVM and LXD containers on both Host Servers. The big question is, how do we implement the PowerVault (and to a certain extent the storage on the Host Servers themselves) to be most beneficial in this mixed environment. I have a few ideas on what I could do, but since I don't have much experience with shared storage, I am probably just picking straws and would like to hear from others that probably has more experience than me. -- Regards Gabriel Marais Office: +27 861 466 546 x 7001 | Mobile: +27 83 663 Mail: gabriel.j.mar...@gmail.com Unit 11, Ground Floor, Berkley Office Park Cnr Bauhinia & Witch Hazel Str, Highveld, Centurion, South-Africa, 0157 PO Box 15846, Lyttelton, South Africa, 0140 ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users