Re: [gentoo-user] VMs - what technology would you advise?
On Wed, Dec 31, 2014 at 1:31 PM, Sid S wrote: > > Containers and such definitely sound interesting; I had been avoiding > Linux VMs for the longest time due to the overhead. The alternatives > sound rather light so I might reconsider. > There are a couple of ways to go with them. The heavy approach is something like Docker which basically wraps it all up in config management and such. The lighter way is to just create chroots and the launch them with something like nspawn (I'm sure there are non-systemd equivalents). Then you have two options inside the container. One is to just directly spawn the process of interest (ie have a init script that launches apache inside a container - not unlike running a chrooted daemon) - this is VERY lightweight though you do have the extra shared objects in memory since you're not using system libs. The other is to run a service manager inside the container (systemd definitely supports this, and I hear that openrc works now as well though you'd have to check the details on that and what versions work) - this is obviously going to be a bit heavier, but it lets you do things like run sshd inside the container, multiple daemons, cron, etc. If you're running under systemd you can also do tricks like having systemd manage the network sockets and launch non-priv'd daemons on demand (a la inetd) which get passed sockets but don't have access to any network interfaces otherwise (so, no outgoing connections). Either way your container can be anything compatible with your kernel. You could run a Gentoo host with a Debian container, and so on. The idea would be to pick the distro most suited to your problem. Maybe for one of your daemons you want to have a lot of control over dependencies so you run Gentoo. Maybe for another the vendor officially supports Debian and it gets rapid updates there, so you run Debian. The main thing you lose is some of the security of VMs, though if you just run your daemon in a container and you run it non-root then you're pretty darn secure (you'd need a very bad local priv escalation to get out). It certainly is more secure than just running your daemon on the host directly. -- Rich
Re: [gentoo-user] VMs - what technology would you advise?
On 31 December 2014 18:33:25 CET, Alan McKinnon wrote:
>On 31/12/2014 18:18, J. Roeleveld wrote:
>> Some of the software we deal with can take up to 30 minutes to fully
>shutdown
>> and re-initialize. (Gotta love those huge enterprise-level BI
>applications)
>
>
>Very OT: Mrs Alan wants to know:
>
>BusinessObjects XI r3?
>
>{apparently she's had bad experiences in the past, and won't talk about
>them :-) )
For Mrs. Alan,
No. BO is actually simple compared to Oracle EPM together with their SOA
infrastructure.
--
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] VMs - what technology would you advise?
On Wed, Dec 31, 2014 at 7:26 AM, Mick wrote: > Vbox seems to be coming last by quite some margin in the intel tests! I also > read this article and it looks that vbox is thankfully doing better on AMD; > but there are differences in the versions and kernels used between the two > articles: Yes, but it didn't seem relevant to your usecase, so I didn't lead with it. I was just hoping you might consider it at some point as an alternative to VirtualBox, as it is fairly complete at this point (though sans memory snapshotting, which is a useful feature I had not considered - I had been doing pretty well with disk snapshots). I originally researched virtualization with an eye to making it usable on a laptop/notebook. In this regard VT-x/VT-d with KVM give you usable battery lifespan and let you use less powerful hardware. Implication: You might not need the workstation to do your testing, depending on what testing you do. Something to consider for the future. I was elated to find I did not need to tie myself to a beefy machine to do what little Windows/.NET development I indulge in. Containers and such definitely sound interesting; I had been avoiding Linux VMs for the longest time due to the overhead. The alternatives sound rather light so I might reconsider.
Re: [gentoo-user] VMs - what technology would you advise?
On 31/12/2014 18:18, J. Roeleveld wrote:
> Some of the software we deal with can take up to 30 minutes to fully shutdown
> and re-initialize. (Gotta love those huge enterprise-level BI applications)
Very OT: Mrs Alan wants to know:
BusinessObjects XI r3?
{apparently she's had bad experiences in the past, and won't talk about
them :-) )
--
Alan McKinnon
alan.mckin...@gmail.com
Re: [gentoo-user] VMs - what technology would you advise?
On Wednesday, December 31, 2014 10:35:04 AM Rich Freeman wrote: > On Wed, Dec 31, 2014 at 10:00 AM, J. Roeleveld wrote: > > The thing lacking from KVM (and I believe also Containers) is that the > > memory contents are not included in snapshots. Making the snapshots > > basically result in an unclean-shutdown scenario. > > Which is ok-ish as a backup, but not when testing different steps where a > > quick and easy roll-back is often required. > > That is a very good point, and as far as I'm aware container memory > can't be snapshotted (unless you count suspend-to-disk of the entire > host). Which is what I was afraid of and is what is keeping me from using it. > Processes in containers are really just processes on the host, > and I don't think there is much support in linux for snapshotting a > process. The best I could find was BLCR, but that didn't really seem > too mainstream (maybe it is). Snapshotting of the disk is whatever > you can do at the filesystem level - a container typically just looks > like a chroot as far as the host is concerned - typically you stick it > on lvm or btrfs for snapshotting. As a chroot-on-steroids inside a VM, it sounds usable, but not as a replacement for VMs. > Now, a big advantage of containers is that startup/shutdown is REALLY > fast. It isn't uncommon for me to run something like "systemctl stop > container ; btrfs su snap container container-back ; systemctl start > container" or something to that effect - often it takes less than a > second to run. Containers are just processes in a separate namespace, > so starting/stopping them is as fast as starting/stopping a service > for the most part. Obviously if your process takes a while to > shutdown and you stop it in a graceful manner then you'll be waiting - > if your process takes a very long time to shutdown/startup then maybe > VM-level snapshotting makes more sense. Some of the software we deal with can take up to 30 minutes to fully shutdown and re-initialize. (Gotta love those huge enterprise-level BI applications) > Depending on what your VM is doing snapshotting and restoring at the > memory level may not be entirely graceful either - obviously any > external connections are not going to be in the same state when it > resumes. Most of the snapshots I take are during the installation and configuration steps. Not many external connections exist during those stages. And the few that do exist generally re-establish themselves when the snapshot is restored. (All nodes of a single instance will be snapshotted near- simultaneously.) -- Joost
Re: [gentoo-user] VMs - what technology would you advise?
On Wed, Dec 31, 2014 at 10:00 AM, J. Roeleveld wrote: > > The thing lacking from KVM (and I believe also Containers) is that the memory > contents are not included in snapshots. Making the snapshots basically result > in an unclean-shutdown scenario. > Which is ok-ish as a backup, but not when testing different steps where a > quick > and easy roll-back is often required. That is a very good point, and as far as I'm aware container memory can't be snapshotted (unless you count suspend-to-disk of the entire host). Processes in containers are really just processes on the host, and I don't think there is much support in linux for snapshotting a process. The best I could find was BLCR, but that didn't really seem too mainstream (maybe it is). Snapshotting of the disk is whatever you can do at the filesystem level - a container typically just looks like a chroot as far as the host is concerned - typically you stick it on lvm or btrfs for snapshotting. Now, a big advantage of containers is that startup/shutdown is REALLY fast. It isn't uncommon for me to run something like "systemctl stop container ; btrfs su snap container container-back ; systemctl start container" or something to that effect - often it takes less than a second to run. Containers are just processes in a separate namespace, so starting/stopping them is as fast as starting/stopping a service for the most part. Obviously if your process takes a while to shutdown and you stop it in a graceful manner then you'll be waiting - if your process takes a very long time to shutdown/startup then maybe VM-level snapshotting makes more sense. Depending on what your VM is doing snapshotting and restoring at the memory level may not be entirely graceful either - obviously any external connections are not going to be in the same state when it resumes. -- Rich
Re: [gentoo-user] VMs - what technology would you advise?
On Wednesday, December 31, 2014 09:42:11 AM Rich Freeman wrote: > On Wed, Dec 31, 2014 at 8:26 AM, Mick wrote: > > On Wednesday 31 Dec 2014 12:47:55 Sid S wrote: > > > > Vbox seems to be coming last by quite some margin in the intel tests! I > > also read this article and it looks that vbox is thankfully doing better > > on AMD; but there are differences in the versions and kernels used > > between the two > > articles: > I think you need to think about your use case. The requirements were > for a workstation testing environment. I think performance (as long > as somewhat reasonable) isn't going to be a big concern there vs ease > of setup, ability to snapshot, The thing lacking from KVM (and I believe also Containers) is that the memory contents are not included in snapshots. Making the snapshots basically result in an unclean-shutdown scenario. Which is ok-ish as a backup, but not when testing different steps where a quick and easy roll-back is often required. > convenience features like being able to > group guests, being able to get the right environment easily, etc. > You probably also want reasonable graphics performance if you're > testing clients inside VMs. If performance makes the difference > between being able to run the cluster you need to test on your > workstation or not, then that becomes a factor. Otherwise it is a > nice-to-have. > > If you're talking about running servers then performance becomes much > more important. However, if you're running linux guests you should > seriously consider containers, and if containers aren't the right > solution you should also be looking at stuff like VMWare (I don't know > how well the FOSS solutions do as far as enterprise-y features go). I compared the ease-of-use and performance between XenServer, VMWare and VirtualBox. VMWare generally is the slower of the three. Also, the weird errors occuring when VMs are migrated between nodes in a VMWare cluster makes me worry every time I hear it's being used for critical systems. > In any case, while not quite as simple as Virtualbox I've found that > virt-manager is very easy to use once you've gotten networking set up > (which isn't too hard to do under either openrc or networkd). I tend > to use the GUI for setting things up and for graphical guests, and I > used to create init.d scripts / units for the stuff that I > subsequently moved to containers. You can go back-and-forth between > the two (and to be fair you can do the same with virtualbox). One of > the advantages of KVM is that it doesn't require tainting your kernel, That is an advantage of KVM and Xen over Virtualbox and VMWare. > and you don't have to remember to rebuild the module anytime you > update your kernel. I've finally gotten to the point where I don't > have any external modules on one of my boxes and I'm very happy with > that (alas, my mythtv frontend needs nvidia-drivers - I don't think > the hardware acceleration is as good with the kernel drivers though to > be fair it has been a year or two since I last tried). I tend to use the nvidia-drivers where I need graphics. But those machines are not VMs. If graphical performance is a requirement, NVidia cards (apart from the expensive professional ones) are best avoided. They are actively crippled in a VM environment. -- Joost
Re: [gentoo-user] VMs - what technology would you advise?
On Wed, Dec 31, 2014 at 8:26 AM, Mick wrote: > On Wednesday 31 Dec 2014 12:47:55 Sid S wrote: > > Vbox seems to be coming last by quite some margin in the intel tests! I also > read this article and it looks that vbox is thankfully doing better on AMD; > but there are differences in the versions and kernels used between the two > articles: > I think you need to think about your use case. The requirements were for a workstation testing environment. I think performance (as long as somewhat reasonable) isn't going to be a big concern there vs ease of setup, ability to snapshot, convenience features like being able to group guests, being able to get the right environment easily, etc. You probably also want reasonable graphics performance if you're testing clients inside VMs. If performance makes the difference between being able to run the cluster you need to test on your workstation or not, then that becomes a factor. Otherwise it is a nice-to-have. If you're talking about running servers then performance becomes much more important. However, if you're running linux guests you should seriously consider containers, and if containers aren't the right solution you should also be looking at stuff like VMWare (I don't know how well the FOSS solutions do as far as enterprise-y features go). In any case, while not quite as simple as Virtualbox I've found that virt-manager is very easy to use once you've gotten networking set up (which isn't too hard to do under either openrc or networkd). I tend to use the GUI for setting things up and for graphical guests, and I used to create init.d scripts / units for the stuff that I subsequently moved to containers. You can go back-and-forth between the two (and to be fair you can do the same with virtualbox). One of the advantages of KVM is that it doesn't require tainting your kernel, and you don't have to remember to rebuild the module anytime you update your kernel. I've finally gotten to the point where I don't have any external modules on one of my boxes and I'm very happy with that (alas, my mythtv frontend needs nvidia-drivers - I don't think the hardware acceleration is as good with the kernel drivers though to be fair it has been a year or two since I last tried). -- Rich
Re: [gentoo-user] VMs - what technology would you advise?
On 31/12/14 21:26, Mick wrote: > On Wednesday 31 Dec 2014 12:47:55 Sid S wrote: >> Yes, in favor of KVM. >> >> http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualiza >> tion http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt > > Vbox seems to be coming last by quite some margin in the intel tests! I also > read this article and it looks that vbox is thankfully doing better on AMD; > but there are differences in the versions and kernels used between the two > articles: > > http://www.phoronix.com/scan.php?page=article&item=ubuntu_1204_virt&num=1 > According to Google, VB includes a lot of qemu code ... I was told way back that it was (once) based on qemu. BillK
Re: [gentoo-user] VMs - what technology would you advise?
On Wednesday 31 Dec 2014 12:47:55 Sid S wrote: > Yes, in favor of KVM. > > http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualiza > tion http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt Vbox seems to be coming last by quite some margin in the intel tests! I also read this article and it looks that vbox is thankfully doing better on AMD; but there are differences in the versions and kernels used between the two articles: http://www.phoronix.com/scan.php?page=article&item=ubuntu_1204_virt&num=1 -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] VMs - what technology would you advise?
Yes, in favor of KVM. http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualization http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt On Wed, Dec 31, 2014 at 5:44 AM, Mick wrote: > On Wednesday 31 Dec 2014 07:32:18 Sid S wrote: >> I would suggest QEMU/KVM takes the place of VirtualBox. I've not >> actually found anything it doesn't support, though VirtualBox is far >> more polished. >> >> Starting a VM will be as easy as running a shell script (or you can >> use virt-manager). > > Thanks Sid, other than the GUI and potential ease of use, is there a > difference in performance between Vbox and KVM? > > -- > Regards, > Mick
Re: [gentoo-user] VMs - what technology would you advise?
On Wednesday 31 Dec 2014 07:32:18 Sid S wrote: > I would suggest QEMU/KVM takes the place of VirtualBox. I've not > actually found anything it doesn't support, though VirtualBox is far > more polished. > > Starting a VM will be as easy as running a shell script (or you can > use virt-manager). Thanks Sid, other than the GUI and potential ease of use, is there a difference in performance between Vbox and KVM? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] VMs - what technology would you advise?
I would suggest QEMU/KVM takes the place of VirtualBox. I've not actually found anything it doesn't support, though VirtualBox is far more polished. Starting a VM will be as easy as running a shell script (or you can use virt-manager).
Re: [gentoo-user] VMs - what technology would you advise?
On Tuesday, December 30, 2014 07:52:44 PM Mick wrote: > For years now I have been running VirtualBox for testing purposes. > > It has served the users' needs well, but I have set up a new PC and am > wondering if I am missing out on new tricks. The VMs are launched may be a > couple of times a week for an hour or two, to test website designs on > different OS' and browsers. The Vbox does this flawlessly. Is there a > different technology and applications I should consider and how do these > compare with Vbox? For simple tests where you only briefly run the VM on a desktop/workstation/laptop, VirtualBox generally does the trick. If you actually need to run the VMs for longer periods of time, having a dedicated machine running the likes of Xen or VMWare is a better option. Of the 2, I would recommend Xen. If you don't feel like configuring the whole stack yourself, a pre-build image like "XCP" generally does the trick. This is a free version of the Citrix VM product. The Citrix VM manager works with XCP as well. -- Joost
Re: [gentoo-user] VMs - what technology would you advise?
On Tuesday 30 Dec 2014 22:01:00 Rich Freeman wrote: > On Tue, Dec 30, 2014 at 2:52 PM, Mick wrote: > > For years now I have been running VirtualBox for testing purposes. > > I used to run vbox, but ran into some issues along the way and > switched to KVM, with virt-manager as a front-end. It is a bit more > complicated to get bridged networking set up, but it doesn't require > any 3rd-party kernel modules to run. You might want to look into it. > It isn't as user-friendly as VirtualBox, but all the features are FOSS > (I forget if all the VirtualBox features are open-source - haven't > used it in a while). You can run VMs via the front-end, or as > daemons/etc. > > This wouldn't really fit your needs, but in general I'd advise anybody > doing virtualization of linux guests to consider running containers > instead. They are fairly mainstream technology now - the isolation > isn't as good as virtualization from a security standpoint, and I have > no idea if you can use one with a graphical console, but otherwise > they give you almost all the benefits of running a linux guest with > much better performance and far less overhead (no double-caching, > etc). I've been moving to containers for more of my daemons as it > generally reduces the hassle of updates (more updates to do, but when > you do an update only one service can break at a time). Containers > can even get their own network interfaces/IPs/etc - just like a VM. Thank you All, it seems that VM is probably still the simplest solution for my needs. I thought of looking into containers, but a few of the VMs are MSWindows. Also thank you Alec for pointing out selenium. I expect it would be useful for automating the testing of more complex websites and web apps. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] VMs - what technology would you advise?
On Tue, Dec 30, 2014 at 2:52 PM, Mick wrote: > For years now I have been running VirtualBox for testing purposes. > I used to run vbox, but ran into some issues along the way and switched to KVM, with virt-manager as a front-end. It is a bit more complicated to get bridged networking set up, but it doesn't require any 3rd-party kernel modules to run. You might want to look into it. It isn't as user-friendly as VirtualBox, but all the features are FOSS (I forget if all the VirtualBox features are open-source - haven't used it in a while). You can run VMs via the front-end, or as daemons/etc. This wouldn't really fit your needs, but in general I'd advise anybody doing virtualization of linux guests to consider running containers instead. They are fairly mainstream technology now - the isolation isn't as good as virtualization from a security standpoint, and I have no idea if you can use one with a graphical console, but otherwise they give you almost all the benefits of running a linux guest with much better performance and far less overhead (no double-caching, etc). I've been moving to containers for more of my daemons as it generally reduces the hassle of updates (more updates to do, but when you do an update only one service can break at a time). Containers can even get their own network interfaces/IPs/etc - just like a VM. -- Rich
Re: [gentoo-user] VMs - what technology would you advise?
On 30/12/2014 21:52, Mick wrote: > For years now I have been running VirtualBox for testing purposes. > > It has served the users' needs well, but I have set up a new PC and am > wondering if I am missing out on new tricks. The VMs are launched may be a > couple of times a week for an hour or two, to test website designs on > different OS' and browsers. The Vbox does this flawlessly. Is there a > different technology and applications I should consider and how do these > compare with Vbox? > You have a simple use-case there which any VM technology can fulfil. You have a working solution that does everything you need it to do and it's familiar to you. I'm not seeing any good reason to change it, unless you feel like playing with new toys, but that's never a good reason to change something in production. If you often create new VMs with different configuration and it's becoming a pain setting them all up anew each time, look into vagrant/docker/ansible for ways to automate that process. These apps support every VM platform out there -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] VMs - what technology would you advise?
On 12/30/2014 02:52 PM, Mick wrote: > For years now I have been running VirtualBox for testing purposes. > > It has served the users' needs well, but I have set up a new PC and am > wondering if I am missing out on new tricks. The VMs are launched may be a > couple of times a week for an hour or two, to test website designs on > different OS' and browsers. The Vbox does this flawlessly. Is there a > different technology and applications I should consider and how do these > compare with Vbox? > I have seen a decent amount of references to Selenium (http://www.seleniumhq.org/), a tool that automates the testing of the UI across multiple platforms/browsers. Never used it, don't know how well it works, etc, but I occasionally come across web stuff when I dabble with Java and Ruby. Alec
