On Tue, Aug 25, 2020 at 05:53:32PM +0200, Jiri Denemark wrote:
> On Tue, Aug 25, 2020 at 07:47:14 +0200, Martin Kletzander wrote:
> > This allows:
> >
> > a) migration without access to network
> >
> > b) complete control of the migration stream
> >
> > c) easy migration between containerised libvirt daemons on the same host
> >
> > Resolves: https://bugzilla.redhat.com/1638889
> >
> > Signed-off-by: Martin Kletzander <mklet...@redhat.com>
> > ---
> > docs/manpages/virsh.rst | 17 ++++-
> > docs/migration.html.in | 33 ++++++++++
> > src/qemu/qemu_migration.c | 128 +++++++++++++++++++++++++++-----------
> > 3 files changed, 139 insertions(+), 39 deletions(-)
> >
> > diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst
> > index cbb3c18deb30..82f7c9f77488 100644
> > --- a/docs/manpages/virsh.rst
> > +++ b/docs/manpages/virsh.rst
> > @@ -3237,12 +3237,12 @@ has different semantics:
> >
> > In a special circumstance where you require a complete control of the
> > connection
> > and/or libvirt does not have network access to the remote side you can use
> > a
> > -unix transport in the URI and specify a socket path in the query, for
> > example
> > +UNIX transport in the URI and specify a socket path in the query, for
> > example
> > with the qemu driver you could use this:
> >
> > .. code-block::
> >
> > - qemu+unix://?socket=/path/to/socket
> > + qemu+unix:///system?socket=/path/to/socket
> >
> > When *migrateuri* is not specified, libvirt will automatically determine
> > the
> > hypervisor specific URI. Some hypervisors, including QEMU, have an
> > optional
>
> This hunk looks like it should go into the previous patch 7/9 (peer2peer
> migration: allow connecting to local sockets).
>
> > @@ -3270,6 +3270,14 @@ There are a few scenarios where specifying
> > *migrateuri* may help:
> > might be specified to choose a specific port number outside the default
> > range in
> > order to comply with local firewall policies.
> >
> > +* The *desturi* uses UNIX transport method. In this advanced case libvirt
> > + should not guess a *migrateuri* and it should be specified using
> > + UNIX socket path URI:
> > +
> > +.. code-block::
> > +
> > + unix://?socket=/path/to/socket
> > +
> > See `https://libvirt.org/migration.html#uris
> > <https://libvirt.org/migration.html#uris>`_ for more details on
> > migration URIs.
> >
> > @@ -3296,7 +3304,10 @@ specific parameters separated by '&'. Currently
> > recognized parameters are
> > Optional *listen-address* sets the listen address that hypervisor on the
> > destination side should bind to for incoming migration. Both IPv4 and IPv6
> > addresses are accepted as well as hostnames (the resolving is done on
> > -destination). Some hypervisors do not support this feature and will return
> > an
> > +destination). In niche scenarios you can also use UNIX socket to make the
> > +hypervisor connection over UNIX socket in which case you must make sure the
> > +source can connect to the destination using the socket path provided by
> > you.
> > +Some hypervisors do not support specifying the listen address and will
> > return an
> > error if this parameter is used.
> >
> > Optional *disks-port* sets the port that hypervisor on destination side
> > should
> > diff --git a/docs/migration.html.in b/docs/migration.html.in
> > index e95ee9de6f1b..9c8417674b22 100644
> > --- a/docs/migration.html.in
> > +++ b/docs/migration.html.in
> > @@ -201,6 +201,9 @@
> > numbers. In the latter case the management application may wish
> > to choose a specific port number outside the default range in order
> > to comply with local firewall policies.</li>
> > + <li>The second URI uses UNIX transport method. In this advanced case
> > + libvirt should not guess a *migrateuri* and it should be specified
> > using
> > + UNIX socket path URI:
> > <code>unix://?socket=/path/to/socket</code>.</li>
> > </ol>
> >
> > <h2><a id="config">Configuration file handling</a></h2>
> > @@ -628,5 +631,35 @@ virsh migrate --p2p --tunnelled web1
> > qemu+ssh://desthost/system qemu+ssh://10.0.
> > Supported by QEMU driver
> > </p>
> >
> > +
> > + <h3><a id="scenariounixsocket">Migration using only UNIX
> > sockets</a></h3>
> > +
> > + <p>
> > + In a niche scenarion where libvirt daemon does not have access to the
>
> s/scenarion/scenario/
>
> or
>
> s/a // and s/scenarion/scenarios/
>
> > + network (e.g. running in a restricted container on a host that has
> > + accessible network), when a management application wants to have
> > complete
> > + control over the transfer or when migrating between two containers
> > on the
> > + same host all the communication can be done using UNIX sockets. This
> > + includes connecting to non-standard socket path for the destination
> > + daemon, using UNIX sockets for hypervisor's communication or for the
> > NBD
> > + data transfer. All of that can be used with both peer2peer and
> > direct
> > + migration options.
> > + </p>
> > +
> > + <p>
> > + Example using <code>/tmp/migdir</code> as a directory representing
> > the
> > + same path visible from both libvirt daemons. That can be achieved by
> > + bind-mounting the same directory to different containers running
> > separate
> > + daemons or forwarding connections to these sockets manually
> > + (using <code>socat</code>, <code>netcat</code> or a custom piece of
> > + software):
> > + <pre>
> > +virsh migrate web1 [--p2p] --copy-storage-all
> > 'qemu+unix:///system?socket=/tmp/migdir/test-sock-driver'
> > 'unix://?socket=/tmp/migdir/test-sock-qemu' [--listen-address
> > /tmp/migdir/test-sock-qemu] --disks-socket /tmp/migdir/test-sock-nbd
> > + </pre>
> > +
> > + <p>
> > + Supported by QEMU driver
> > + </p>
> > +
> > </body>
> > </html>
> > diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
> > index 3f4690f8fb72..1158d152869c 100644
> > --- a/src/qemu/qemu_migration.c
> > +++ b/src/qemu/qemu_migration.c
> ...
> > @@ -3931,16 +3969,34 @@ qemuMigrationSrcPerformNative(virQEMUDriverPtr
> > driver,
> > }
> > }
> >
> > - /* RDMA and multi-fd migration requires QEMU to connect to the
> > destination
> > - * itself.
> > - */
> > - if (STREQ(uribits->scheme, "rdma") || (flags & VIR_MIGRATE_PARALLEL))
> > - spec.destType = MIGRATION_DEST_HOST;
> > - else
> > - spec.destType = MIGRATION_DEST_CONNECT_HOST;
> > - spec.dest.host.protocol = uribits->scheme;
> > - spec.dest.host.name = uribits->server;
> > - spec.dest.host.port = uribits->port;
> > + if (STREQ(uribits->scheme, "unix")) {
> > + if (flags & VIR_MIGRATE_TLS) {
> > + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
> > + _("Migration over UNIX socket with TLS is not
> > supported"));
> > + return -1;
> > + }
> > + if (flags & VIR_MIGRATE_PARALLEL) {
> > + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
> > + _("Parallel migration over UNIX socket is not
> > supported"));
> > + return -1;
> > + }
>
> Is there a reason for not supporting TLS and multi-fd with unix sockets?
> >From libvirt's POV this should be fairly trivial (can be a follow-up
> patch, though): introduce MIGRATION_DEST_SOCKET in addition to
> MIGRATION_DEST_CONNECT_SOCKET and use it to instruct QEMU to connect
> using unix: URI (unless this support was removed from QEMU).
multi-fd is certainly desirable to support and I don't see any reason
to block that.
TLS is more problematic, at least if you are using x509 credentials then
you need to tell QEMU what hostname to use for validation. This would
require us to accept a hostname parameter in the URI giving the UNIX
socket.
This reminds me, however, that we should add support for PSK credentials
in libvirt. PSK is an alternative to x509 where you just provide QEMU
the private keypair directly. To use PSK you need to have a trusted
channel to both src and dst over which libvirt can send the key.
...fortunately we should have that already as libvirtd RPC always uses
a secure connection between libvirtds for sending migration cookies,
etc. Libvirtd just needs to generate the key which is trivial.
If we add support for PSK, then a user can use TLS encrypted migration
out of the box with zero config required, if they're using SSH tunnel
for libvirtd.
>
> > +
> > + spec.destType = MIGRATION_DEST_CONNECT_SOCKET;
> > + spec.dest.socket.path = virURIGetParam(uribits, "socket");
> > + } else {
> > + /* RDMA and multi-fd migration requires QEMU to connect to the
> > destination
> > + * itself.
> > + */
> > + if (STREQ(uribits->scheme, "rdma") || (flags &
> > VIR_MIGRATE_PARALLEL))
> > + spec.destType = MIGRATION_DEST_HOST;
> > + else
> > + spec.destType = MIGRATION_DEST_CONNECT_HOST;
> > +
> > + spec.dest.host.protocol = uribits->scheme;
> > + spec.dest.host.name = uribits->server;
> > + spec.dest.host.port = uribits->port;
> > + }
> > +
> > spec.fwdType = MIGRATION_FWD_DIRECT;
> >
> > ret = qemuMigrationSrcRun(driver, vm, persist_xml, cookiein,
> > cookieinlen, cookieout,
>
> Reviewed-by: Jiri Denemark <jdene...@redhat.com>
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
