Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Daniel P. Berrange]

On Tue, Feb 21, 2017 at 10:08:54AM -0500, David Medberry wrote:
> Doesn't the virtio solution assume/require a libvirt or more exactly a
> QEMU/KVM based hypervisor?
> 
> What about the N-1 other hypervisors?

vsock is an equivalent of UNIX domain sockets, for host<->guest communication
that was invented by VMWare.  KVM adopted the same socket protocol and layers
it over virtio. So from an application developer POV, whether you use VMWare
or KVM, vsock works the same way.  From a technology POV it wouldbe possible
to use vsock over virtio with Xen, since Xen supports the virtio bus, but I
doubt XenAPI will support it any time soon. Likewise I doubt hyperv will
support it, but they may well provide an equivalent alternatie, since this
is a commonly needed concept.

> I think the idea of a "hot remove, hot add" of the configdrive has some
> merit (but remember it is not always ISO-9660 but could be VFAT as well to
> aid in some migrations.)

NB current config drive is not well suited to hot remove. If using CDROM
with ISO-9660 you the guest to unlock the CDROM tray before the host is
able to eject & change media in it.  If using virtio disk with vfat, you
need the guest to honour ACPI request to unplug the PCI device. If the
disk is mounted, this unplug will not complete.

So realistically you'd need to change to use USB disks, which the host is
able to rip out from under the guest unconditionally, even if it is mountd
by the guest. Albeit with the caveat that the guest now needs to clean
up dangling mount points.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[David Medberry]

Doesn't the virtio solution assume/require a libvirt or more exactly a
QEMU/KVM based hypervisor?

What about the N-1 other hypervisors?

I think the idea of a "hot remove, hot add" of the configdrive has some
merit (but remember it is not always ISO-9660 but could be VFAT as well to
aid in some migrations.)

On the plus side, we do actually run the md service.

On Mon, Feb 20, 2017 at 1:22 PM, Artom Lifshitz  wrote:

> We've been having a discussion [1] in openstack-dev about how to best
> expose dynamic metadata that changes over a server's lifetime to the
> server. The specific use case is device role tagging with hotplugged
> devices, where a network interface or volume is attached with a role
> tag, and the guest would like to know what that role tag is right
> away.
>
> The metadata API currently fulfills this function, but my
> understanding is that it's not hugely popular amongst operators and is
> therefore not universally deployed.
>
> Dan Berrange came up with an idea [2] to add virtio-vsock support to
> Nova. To quote his explanation, " think of this as UNIX domain sockets
> between the host and guest. [...] It'd likely address at least some
> people's security concerns wrt metadata service. It would also fix the
> ability to use the metadata service in IPv6-only environments, as we
> would not be using IP at all."
>
> So to those operators who are not deploying the metadata service -
> what are your reasons for doing so, and would those concerns be
> addressed by Dan's idea?
>
> Cheers!
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-
> February/112490.html
> [2] http://lists.openstack.org/pipermail/openstack-dev/2017-
> February/112602.html
>
> --
> Artom Lifshitz
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Daniel P. Berrange]

On Tue, Feb 21, 2017 at 01:09:43PM +, Jeremy Stanley wrote:
> On 2017-02-21 06:24:20 -0500 (-0500), Clint Byrum wrote:
> [...]
> > Why not just make it a virtual USB drive that ejects and
> > re-attaches on changes?
> [...]
> 
> Is there a graceful way to trigger that from the host so that the
> guest knows to unmount it prior to ejection? Or is that not actually
> necessary?

No, there's no way to coordinate USB removal with the guest - you would
just be ripping the drive out from under the guest and expecting the
guest to clean up its mounts if it had it mounted. Not too pleasant,
but also not an entirely unusual scenario for an OS to deal with. After
all, users of physical hardware sometimes pull out USB sticks without
going through any unmount / safe remove process.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Jeremy Stanley]

On 2017-02-21 06:24:20 -0500 (-0500), Clint Byrum wrote:
[...]
> Why not just make it a virtual USB drive that ejects and
> re-attaches on changes?
[...]

Is there a graceful way to trigger that from the host so that the
guest knows to unmount it prior to ejection? Or is that not actually
necessary?
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Clint Byrum]

Excerpts from Daniel P. Berrange's message of 2017-02-21 10:40:02 +:
> On Mon, Feb 20, 2017 at 02:36:15PM -0500, Clint Byrum wrote:
> > What exactly is the security concern of the metadata service? Perhaps
> > those concerns can be addressed directly?
> > 
> > I ask because anything that requires special software on the guest is
> > a non-starter IMO. virtio is a Linux thing, so what does this do for
> > users of Windows?  FreeBSD? etc.
> 
> Red Hat is investing in creating virtio vsock drivers for Windows
> but I don't have an ETA for that yet. There's no work in *BSD in
> this area that I know of, but BSD does have support for virtio
> in general, so if virtio-vsock becomes used in any important
> places I would not be suprised if some BSD developers implemented
> vsock too.
> 

> In any case, I don't think it neccessarily needs to be supported
> in every single possible scenario. The config drive provides the
> same data in a highly portable manner, albeit with the caveat
> about it being read-only. The use of metadata service (whether
> TCP or vsock based) is useful for cases needing the info from
> config drive to be dynamically updated - eg the role device
> tagging metadata. Only a very small subset of guests running on
> openstack actually use that data today. So it would not be the
> end of the world if some guests don't support vsock in the short
> to medium term - if the facility proves to be critically important
> to a wider range of guests that'll motivate developers of those
> OS to support it.
> 

Cool, so there's a chance it gets to near ubiquitous usability.

However, I wonder, there's no need for performance here. Why not just
make it a virtual USB drive that ejects and re-attaches on changes? That
way you don't need Windows/BSD drivers.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Daniel P. Berrange]

On Mon, Feb 20, 2017 at 02:36:15PM -0500, Clint Byrum wrote:
> What exactly is the security concern of the metadata service? Perhaps
> those concerns can be addressed directly?
> 
> I ask because anything that requires special software on the guest is
> a non-starter IMO. virtio is a Linux thing, so what does this do for
> users of Windows?  FreeBSD? etc.

Red Hat is investing in creating virtio vsock drivers for Windows
but I don't have an ETA for that yet. There's no work in *BSD in
this area that I know of, but BSD does have support for virtio
in general, so if virtio-vsock becomes used in any important
places I would not be suprised if some BSD developers implemented
vsock too.

In any case, I don't think it neccessarily needs to be supported
in every single possible scenario. The config drive provides the
same data in a highly portable manner, albeit with the caveat
about it being read-only. The use of metadata service (whether
TCP or vsock based) is useful for cases needing the info from
config drive to be dynamically updated - eg the role device
tagging metadata. Only a very small subset of guests running on
openstack actually use that data today. So it would not be the
end of the world if some guests don't support vsock in the short
to medium term - if the facility proves to be critically important
to a wider range of guests that'll motivate developers of those
OS to support it.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Daniel P. Berrange]

On Mon, Feb 20, 2017 at 08:08:00PM +, Jeremy Stanley wrote:
> On 2017-02-20 14:36:15 -0500 (-0500), Clint Byrum wrote:
> > What exactly is the security concern of the metadata service? Perhaps
> > those concerns can be addressed directly?
> [...]
> 
> A few I'm aware of:
> 
> 1. It's something that runs in the control plane but needs to be
> reachable from untrusted server instances (which may themselves even
> want to be on completely non-routed networks).

That is the key problem that virtio-vsock solves, by separating
traffic out from the network stack there's no way for a guest
to use vsock to access anything except services on the local
compute host listening on vsock

> 2. If you put a Web proxy between your server instances and the
> metadata service and also make it reachable without going through
> that proxy then instances may be able to spoof one another
> (OSSN-0074).

FYI, with virtio-vsock it is impossible for the guest to spoof
the sending address of another guest. So the process on the host
can use the socket peer address to reliably identify which guest
it is communicating with. With the IP based metadata service
you need to setup firewall rules on the host to drop traffic
with spoofed source mac/ip address.

> 3. Lots of things, for example facter, like to beat on it heavily
> which makes for a fun DDoS and so is a bit of a scaling challenge in
> large deployments.

FYI, with virtio-vsock, you would need to either run the metdata
service on every compute host, or have some kind of vhost<->tcp
proxy on every compute host that forwards requests to the real
metadata service off-host.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Clint Byrum]

Excerpts from Jeremy Stanley's message of 2017-02-20 20:08:00 +:
> On 2017-02-20 14:36:15 -0500 (-0500), Clint Byrum wrote:
> > What exactly is the security concern of the metadata service? Perhaps
> > those concerns can be addressed directly?
> [...]
> 
> A few I'm aware of:
> 

Thanks!

> 1. It's something that runs in the control plane but needs to be
> reachable from untrusted server instances (which may themselves even
> want to be on completely non-routed networks).
> 

As is DHCP

> 2. If you put a Web proxy between your server instances and the
> metadata service and also make it reachable without going through
> that proxy then instances may be able to spoof one another
> (OSSN-0074).
> 

That's assuming the link-local approach used by the EC2 style service.

If you have DHCP hand out a metadata URL with a nonce in it, that's no
longer an issue.

> 3. Lots of things, for example facter, like to beat on it heavily
> which makes for a fun DDoS and so is a bit of a scaling challenge in
> large deployments.
> 

These are fully mitigated by caching.

> There are probably plenty more I don't know since I'm not steeped in
> operating OpenStack deployments.

Thanks. I don't mean to combat the suggestions, but rather just see
what it is exactly that makes us dislike the metadata service.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Jeremy Stanley]

On 2017-02-20 14:36:15 -0500 (-0500), Clint Byrum wrote:
> What exactly is the security concern of the metadata service? Perhaps
> those concerns can be addressed directly?
[...]

A few I'm aware of:

1. It's something that runs in the control plane but needs to be
reachable from untrusted server instances (which may themselves even
want to be on completely non-routed networks).

2. If you put a Web proxy between your server instances and the
metadata service and also make it reachable without going through
that proxy then instances may be able to spoof one another
(OSSN-0074).

3. Lots of things, for example facter, like to beat on it heavily
which makes for a fun DDoS and so is a bit of a scaling challenge in
large deployments.

There are probably plenty more I don't know since I'm not steeped in
operating OpenStack deployments.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Metadata service over virtio-vsock

[Clint Byrum]

What exactly is the security concern of the metadata service? Perhaps
those concerns can be addressed directly?

I ask because anything that requires special software on the guest is
a non-starter IMO. virtio is a Linux thing, so what does this do for
users of Windows?  FreeBSD? etc.

Excerpts from Artom Lifshitz's message of 2017-02-20 13:22:36 -0500:
> We've been having a discussion [1] in openstack-dev about how to best
> expose dynamic metadata that changes over a server's lifetime to the
> server. The specific use case is device role tagging with hotplugged
> devices, where a network interface or volume is attached with a role
> tag, and the guest would like to know what that role tag is right
> away.
> 
> The metadata API currently fulfills this function, but my
> understanding is that it's not hugely popular amongst operators and is
> therefore not universally deployed.
> 
> Dan Berrange came up with an idea [2] to add virtio-vsock support to
> Nova. To quote his explanation, " think of this as UNIX domain sockets
> between the host and guest. [...] It'd likely address at least some
> people's security concerns wrt metadata service. It would also fix the
> ability to use the metadata service in IPv6-only environments, as we
> would not be using IP at all."
> 
> So to those operators who are not deploying the metadata service -
> what are your reasons for doing so, and would those concerns be
> addressed by Dan's idea?
> 
> Cheers!
> 
> [1] 
> http://lists.openstack.org/pipermail/openstack-dev/2017-February/112490.html
> [2] 
> http://lists.openstack.org/pipermail/openstack-dev/2017-February/112602.html
> 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators