Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Rich Freeman
On Wed, Dec 31, 2014 at 1:31 PM, Sid S  wrote:
>
> Containers and such definitely sound interesting; I had been avoiding
> Linux VMs for the longest time due to the overhead. The alternatives
> sound rather light so I might reconsider.
>

There are a couple of ways to go with them.  The heavy approach is
something like Docker which basically wraps it all up in config
management and such.  The lighter way is to just create chroots and
the launch them with something like nspawn (I'm sure there are
non-systemd equivalents).  Then you have two options inside the
container.  One is to just directly spawn the process of interest (ie
have a init script that launches apache inside a container - not
unlike running a chrooted daemon) - this is VERY lightweight though
you do have the extra shared objects in memory since you're not using
system libs.  The other is to run a service manager inside the
container (systemd definitely supports this, and I hear that openrc
works now as well though you'd have to check the details on that and
what versions work) - this is obviously going to be a bit heavier, but
it lets you do things like run sshd inside the container, multiple
daemons, cron, etc.  If you're running under systemd you can also do
tricks like having systemd manage the network sockets and launch
non-priv'd daemons on demand (a la inetd) which get passed sockets but
don't have access to any network interfaces otherwise (so, no outgoing
connections).

Either way your container can be anything compatible with your kernel.
You could run a Gentoo host with a Debian container, and so on.  The
idea would be to pick the distro most suited to your problem.  Maybe
for one of your daemons you want to have a lot of control over
dependencies so you run Gentoo.  Maybe for another the vendor
officially supports Debian and it gets rapid updates there, so you run
Debian.

The main thing you lose is some of the security of VMs, though if you
just run your daemon in a container and you run it non-root then
you're pretty darn secure (you'd need a very bad local priv escalation
to get out).  It certainly is more secure than just running your
daemon on the host directly.

-- 
Rich



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread J. Roeleveld
On 31 December 2014 18:33:25 CET, Alan McKinnon  wrote:
>On 31/12/2014 18:18, J. Roeleveld wrote:
>> Some of the software we deal with can take up to 30 minutes to fully
>shutdown 
>> and re-initialize. (Gotta love those huge enterprise-level BI
>applications)
>
>
>Very OT: Mrs Alan wants to know:
>
>BusinessObjects XI r3?
>
>{apparently she's had bad experiences in the past, and won't talk about
>them :-) )

For Mrs. Alan,

No. BO is actually simple compared to Oracle EPM together with their SOA 
infrastructure.

--
Joost
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Sid S
On Wed, Dec 31, 2014 at 7:26 AM, Mick  wrote:
> Vbox seems to be coming last by quite some margin in the intel tests!  I also
> read this article and it looks that vbox is thankfully doing better on AMD;
> but there are differences in the versions and kernels used between the two
> articles:

Yes, but it didn't seem relevant to your usecase, so I didn't lead
with it. I was just hoping you might consider it at some point as an
alternative to VirtualBox, as it is fairly complete at this point
(though sans memory snapshotting, which is a useful feature I had not
considered - I had been doing pretty well with disk snapshots).

I originally researched virtualization with an eye to making it usable
on a laptop/notebook. In this regard VT-x/VT-d with KVM give you
usable battery lifespan and let you use less powerful hardware.
Implication: You might not need the workstation to do your testing,
depending on what testing you do. Something to consider for the
future. I was elated to find I did not need to tie myself to a beefy
machine to do what little Windows/.NET development I indulge in.


Containers and such definitely sound interesting; I had been avoiding
Linux VMs for the longest time due to the overhead. The alternatives
sound rather light so I might reconsider.



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Alan McKinnon
On 31/12/2014 18:18, J. Roeleveld wrote:
> Some of the software we deal with can take up to 30 minutes to fully shutdown 
> and re-initialize. (Gotta love those huge enterprise-level BI applications)


Very OT: Mrs Alan wants to know:

BusinessObjects XI r3?

{apparently she's had bad experiences in the past, and won't talk about
them :-) )

-- 
Alan McKinnon
alan.mckin...@gmail.com




Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread J. Roeleveld
On Wednesday, December 31, 2014 10:35:04 AM Rich Freeman wrote:
> On Wed, Dec 31, 2014 at 10:00 AM, J. Roeleveld  wrote:
> > The thing lacking from KVM (and I believe also Containers) is that the
> > memory contents are not included in snapshots. Making the snapshots
> > basically result in an unclean-shutdown scenario.
> > Which is ok-ish as a backup, but not when testing different steps where a
> > quick and easy roll-back is often required.
> 
> That is a very good point, and as far as I'm aware container memory
> can't be snapshotted (unless you count suspend-to-disk of the entire
> host).

Which is what I was afraid of and is what is keeping me from using it.

> Processes in containers are really just processes on the host,
> and I don't think there is much support in linux for snapshotting a
> process.  The best I could find was BLCR, but that didn't really seem
> too mainstream (maybe it is).  Snapshotting of the disk is whatever
> you can do at the filesystem level - a container typically just looks
> like a chroot as far as the host is concerned - typically you stick it
> on lvm or btrfs for snapshotting.

As a chroot-on-steroids inside a VM, it sounds usable, but not as a 
replacement for VMs.

> Now, a big advantage of containers is that startup/shutdown is REALLY
> fast.  It isn't uncommon for me to run something like "systemctl stop
> container ; btrfs su snap container container-back ; systemctl start
> container" or something to that effect - often it takes less than a
> second to run.  Containers are just processes in a separate namespace,
> so starting/stopping them is as fast as starting/stopping a service
> for the most part.  Obviously if your process takes a while to
> shutdown and you stop it in a graceful manner then you'll be waiting -
> if your process takes a very long time to shutdown/startup then maybe
> VM-level snapshotting makes more sense.

Some of the software we deal with can take up to 30 minutes to fully shutdown 
and re-initialize. (Gotta love those huge enterprise-level BI applications)

> Depending on what your VM is doing snapshotting and restoring at the
> memory level may not be entirely graceful either - obviously any
> external connections are not going to be in the same state when it
> resumes.

Most of the snapshots I take are during the installation and configuration 
steps. Not many external connections exist during those stages.
And the few that do exist generally re-establish themselves when the snapshot 
is restored. (All nodes of a single instance will be snapshotted near-
simultaneously.)

--
Joost



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Rich Freeman
On Wed, Dec 31, 2014 at 10:00 AM, J. Roeleveld  wrote:
>
> The thing lacking from KVM (and I believe also Containers) is that the memory
> contents are not included in snapshots. Making the snapshots basically result
> in an unclean-shutdown scenario.
> Which is ok-ish as a backup, but not when testing different steps where a 
> quick
> and easy roll-back is often required.

That is a very good point, and as far as I'm aware container memory
can't be snapshotted (unless you count suspend-to-disk of the entire
host).  Processes in containers are really just processes on the host,
and I don't think there is much support in linux for snapshotting a
process.  The best I could find was BLCR, but that didn't really seem
too mainstream (maybe it is).  Snapshotting of the disk is whatever
you can do at the filesystem level - a container typically just looks
like a chroot as far as the host is concerned - typically you stick it
on lvm or btrfs for snapshotting.

Now, a big advantage of containers is that startup/shutdown is REALLY
fast.  It isn't uncommon for me to run something like "systemctl stop
container ; btrfs su snap container container-back ; systemctl start
container" or something to that effect - often it takes less than a
second to run.  Containers are just processes in a separate namespace,
so starting/stopping them is as fast as starting/stopping a service
for the most part.  Obviously if your process takes a while to
shutdown and you stop it in a graceful manner then you'll be waiting -
if your process takes a very long time to shutdown/startup then maybe
VM-level snapshotting makes more sense.

Depending on what your VM is doing snapshotting and restoring at the
memory level may not be entirely graceful either - obviously any
external connections are not going to be in the same state when it
resumes.

--
Rich



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread J. Roeleveld
On Wednesday, December 31, 2014 09:42:11 AM Rich Freeman wrote:
> On Wed, Dec 31, 2014 at 8:26 AM, Mick  wrote:
> > On Wednesday 31 Dec 2014 12:47:55 Sid S wrote:
> > 
> > Vbox seems to be coming last by quite some margin in the intel tests!  I
> > also read this article and it looks that vbox is thankfully doing better
> > on AMD; but there are differences in the versions and kernels used
> > between the two
> > articles:
> I think you need to think about your use case.  The requirements were
> for a workstation testing environment.  I think performance (as long
> as somewhat reasonable) isn't going to be a big concern there vs ease
> of setup, ability to snapshot,

The thing lacking from KVM (and I believe also Containers) is that the memory 
contents are not included in snapshots. Making the snapshots basically result 
in an unclean-shutdown scenario.
Which is ok-ish as a backup, but not when testing different steps where a quick 
and easy roll-back is often required.

> convenience features like being able to
> group guests, being able to get the right environment easily, etc.
> You probably also want reasonable graphics performance if you're
> testing clients inside VMs.  If performance makes the difference
> between being able to run the cluster you need to test on your
> workstation or not, then that becomes a factor.  Otherwise it is a
> nice-to-have.
> 
> If you're talking about running servers then performance becomes much
> more important.  However, if you're running linux guests you should
> seriously consider containers, and if containers aren't the right
> solution you should also be looking at stuff like VMWare (I don't know
> how well the FOSS solutions do as far as enterprise-y features go).

I compared the ease-of-use and performance between XenServer, VMWare and 
VirtualBox.
VMWare generally is the slower of the three.
Also, the weird errors occuring when VMs are migrated between nodes in a 
VMWare cluster makes me worry every time I hear it's being used for critical 
systems.

> In any case, while not quite as simple as Virtualbox I've found that
> virt-manager is very easy to use once you've gotten networking set up
> (which isn't too hard to do under either openrc or networkd).  I tend
> to use the GUI for setting things up and for graphical guests, and I
> used to create init.d scripts / units for the stuff that I
> subsequently moved to containers.  You can go back-and-forth between
> the two (and to be fair you can do the same with virtualbox).  One of
> the advantages of KVM is that it doesn't require tainting your kernel,

That is an advantage of KVM and Xen over Virtualbox and VMWare.

> and you don't have to remember to rebuild the module anytime you
> update your kernel.  I've finally gotten to the point where I don't
> have any external modules on one of my boxes and I'm very happy with
> that (alas, my mythtv frontend needs nvidia-drivers - I don't think
> the hardware acceleration is as good with the kernel drivers though to
> be fair it has been a year or two since I last tried).

I tend to use the nvidia-drivers where I need graphics. But those machines are 
not VMs.
If graphical performance is a requirement, NVidia cards (apart from the 
expensive professional ones) are best avoided. They are actively crippled in a 
VM environment.

--
Joost



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Rich Freeman
On Wed, Dec 31, 2014 at 8:26 AM, Mick  wrote:
> On Wednesday 31 Dec 2014 12:47:55 Sid S wrote:
>
> Vbox seems to be coming last by quite some margin in the intel tests!  I also
> read this article and it looks that vbox is thankfully doing better on AMD;
> but there are differences in the versions and kernels used between the two
> articles:
>

I think you need to think about your use case.  The requirements were
for a workstation testing environment.  I think performance (as long
as somewhat reasonable) isn't going to be a big concern there vs ease
of setup, ability to snapshot, convenience features like being able to
group guests, being able to get the right environment easily, etc.
You probably also want reasonable graphics performance if you're
testing clients inside VMs.  If performance makes the difference
between being able to run the cluster you need to test on your
workstation or not, then that becomes a factor.  Otherwise it is a
nice-to-have.

If you're talking about running servers then performance becomes much
more important.  However, if you're running linux guests you should
seriously consider containers, and if containers aren't the right
solution you should also be looking at stuff like VMWare (I don't know
how well the FOSS solutions do as far as enterprise-y features go).

In any case, while not quite as simple as Virtualbox I've found that
virt-manager is very easy to use once you've gotten networking set up
(which isn't too hard to do under either openrc or networkd).  I tend
to use the GUI for setting things up and for graphical guests, and I
used to create init.d scripts / units for the stuff that I
subsequently moved to containers.  You can go back-and-forth between
the two (and to be fair you can do the same with virtualbox).  One of
the advantages of KVM is that it doesn't require tainting your kernel,
and you don't have to remember to rebuild the module anytime you
update your kernel.  I've finally gotten to the point where I don't
have any external modules on one of my boxes and I'm very happy with
that (alas, my mythtv frontend needs nvidia-drivers - I don't think
the hardware acceleration is as good with the kernel drivers though to
be fair it has been a year or two since I last tried).

-- 
Rich



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Bill Kenworthy
On 31/12/14 21:26, Mick wrote:
> On Wednesday 31 Dec 2014 12:47:55 Sid S wrote:
>> Yes, in favor of KVM.
>>
>> http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualiza
>> tion http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt
> 
> Vbox seems to be coming last by quite some margin in the intel tests!  I also 
> read this article and it looks that vbox is thankfully doing better on AMD; 
> but there are differences in the versions and kernels used between the two 
> articles:
> 
>  http://www.phoronix.com/scan.php?page=article&item=ubuntu_1204_virt&num=1
> 

According to Google, VB includes a lot of qemu code ... I was told way
back that it was (once) based on qemu.

BillK




Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Mick
On Wednesday 31 Dec 2014 12:47:55 Sid S wrote:
> Yes, in favor of KVM.
> 
> http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualiza
> tion http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt

Vbox seems to be coming last by quite some margin in the intel tests!  I also 
read this article and it looks that vbox is thankfully doing better on AMD; 
but there are differences in the versions and kernels used between the two 
articles:

 http://www.phoronix.com/scan.php?page=article&item=ubuntu_1204_virt&num=1

-- 
Regards,
Mick


signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Sid S
Yes, in favor of KVM.

http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualization
http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmboxt

On Wed, Dec 31, 2014 at 5:44 AM, Mick  wrote:
> On Wednesday 31 Dec 2014 07:32:18 Sid S wrote:
>> I would suggest QEMU/KVM takes the place of VirtualBox. I've not
>> actually found anything it doesn't support, though VirtualBox is far
>> more polished.
>>
>> Starting a VM will be as easy as running a shell script (or you can
>> use virt-manager).
>
> Thanks Sid, other than the GUI and potential ease of use, is there a
> difference in performance between Vbox and KVM?
>
> --
> Regards,
> Mick



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-31 Thread Mick
On Wednesday 31 Dec 2014 07:32:18 Sid S wrote:
> I would suggest QEMU/KVM takes the place of VirtualBox. I've not
> actually found anything it doesn't support, though VirtualBox is far
> more polished.
> 
> Starting a VM will be as easy as running a shell script (or you can
> use virt-manager).

Thanks Sid, other than the GUI and potential ease of use, is there a 
difference in performance between Vbox and KVM?

-- 
Regards,
Mick


signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread Sid S
I would suggest QEMU/KVM takes the place of VirtualBox. I've not
actually found anything it doesn't support, though VirtualBox is far
more polished.

Starting a VM will be as easy as running a shell script (or you can
use virt-manager).



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread J. Roeleveld
On Tuesday, December 30, 2014 07:52:44 PM Mick wrote:
> For years now I have been running VirtualBox for testing purposes.
> 
> It has served the users' needs well, but I have set up a new PC and am
> wondering if I am missing out on new tricks.  The VMs are launched may be a
> couple of times a week for an hour or two, to test website designs on
> different OS' and browsers.  The Vbox does this flawlessly.  Is there a
> different technology and applications I should consider and how do these
> compare with Vbox?

For simple tests where you only briefly run the VM on a 
desktop/workstation/laptop, VirtualBox generally does the trick.

If you actually need to run the VMs for longer periods of time, having a 
dedicated machine running the likes of Xen or VMWare is a better option.

Of the 2, I would recommend Xen.

If you don't feel like configuring the whole stack yourself, a pre-build image 
like "XCP" generally does the trick. This is a free version of the Citrix VM 
product. The Citrix VM manager works with XCP as well.

--
Joost



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread Mick
On Tuesday 30 Dec 2014 22:01:00 Rich Freeman wrote:
> On Tue, Dec 30, 2014 at 2:52 PM, Mick  wrote:
> > For years now I have been running VirtualBox for testing purposes.
> 
> I used to run vbox, but ran into some issues along the way and
> switched to KVM, with virt-manager as a front-end.  It is a bit more
> complicated to get bridged networking set up, but it doesn't require
> any 3rd-party kernel modules to run.  You might want to look into it.
> It isn't as user-friendly as VirtualBox, but all the features are FOSS
> (I forget if all the VirtualBox features are open-source - haven't
> used it in a while).  You can run VMs via the front-end, or as
> daemons/etc.
> 
> This wouldn't really fit your needs, but in general I'd advise anybody
> doing virtualization of linux guests to consider running containers
> instead.  They are fairly mainstream technology now - the isolation
> isn't as good as virtualization from a security standpoint, and I have
> no idea if you can use one with a graphical console, but otherwise
> they give you almost all the benefits of running a linux guest with
> much better performance and far less overhead (no double-caching,
> etc).  I've been moving to containers for more of my daemons as it
> generally reduces the hassle of updates (more updates to do, but when
> you do an update only one service can break at a time).  Containers
> can even get their own network interfaces/IPs/etc - just like a VM.


Thank you All, it seems that VM is probably still the simplest solution for my 
needs.  I thought of looking into containers, but a few of the VMs are 
MSWindows.

Also thank you Alec for pointing out selenium.  I expect it would be useful 
for automating the testing of more complex websites and web apps.

-- 
Regards,
Mick


signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread Rich Freeman
On Tue, Dec 30, 2014 at 2:52 PM, Mick  wrote:
> For years now I have been running VirtualBox for testing purposes.
>

I used to run vbox, but ran into some issues along the way and
switched to KVM, with virt-manager as a front-end.  It is a bit more
complicated to get bridged networking set up, but it doesn't require
any 3rd-party kernel modules to run.  You might want to look into it.
It isn't as user-friendly as VirtualBox, but all the features are FOSS
(I forget if all the VirtualBox features are open-source - haven't
used it in a while).  You can run VMs via the front-end, or as
daemons/etc.

This wouldn't really fit your needs, but in general I'd advise anybody
doing virtualization of linux guests to consider running containers
instead.  They are fairly mainstream technology now - the isolation
isn't as good as virtualization from a security standpoint, and I have
no idea if you can use one with a graphical console, but otherwise
they give you almost all the benefits of running a linux guest with
much better performance and far less overhead (no double-caching,
etc).  I've been moving to containers for more of my daemons as it
generally reduces the hassle of updates (more updates to do, but when
you do an update only one service can break at a time).  Containers
can even get their own network interfaces/IPs/etc - just like a VM.

-- 
Rich



Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread Alan McKinnon
On 30/12/2014 21:52, Mick wrote:
> For years now I have been running VirtualBox for testing purposes.
> 
> It has served the users' needs well, but I have set up a new PC and am 
> wondering if I am missing out on new tricks.  The VMs are launched may be a 
> couple of times a week for an hour or two, to test website designs on 
> different OS' and browsers.  The Vbox does this flawlessly.  Is there a 
> different technology and applications I should consider and how do these 
> compare with Vbox?
> 

You have a simple use-case there which any VM technology can fulfil.
You have a working solution that does everything you need it to do and
it's familiar to you.

I'm not seeing any good reason to change it, unless you feel like
playing with new toys, but that's never a good reason to change
something in production.

If you often create new VMs with different configuration and it's
becoming a pain setting them all up anew each time, look into
vagrant/docker/ansible for ways to automate that process. These apps
support every VM platform out there


-- 
Alan McKinnon
alan.mckin...@gmail.com




Re: [gentoo-user] VMs - what technology would you advise?

2014-12-30 Thread Alec Ten Harmsel

On 12/30/2014 02:52 PM, Mick wrote:
> For years now I have been running VirtualBox for testing purposes.
>
> It has served the users' needs well, but I have set up a new PC and am 
> wondering if I am missing out on new tricks.  The VMs are launched may be a 
> couple of times a week for an hour or two, to test website designs on 
> different OS' and browsers.  The Vbox does this flawlessly.  Is there a 
> different technology and applications I should consider and how do these 
> compare with Vbox?
>

I have seen a decent amount of references to Selenium
(http://www.seleniumhq.org/), a tool that automates the testing of the
UI across multiple platforms/browsers. Never used it, don't know how
well it works, etc, but I occasionally come across web stuff when I
dabble with Java and Ruby.

Alec