Support transferring of TAP state (including open fd). Add new property "local-migration-supported", which defines whether local-migration is actually supported for this TAP device. Starting from 11.2 QEMU Machine Types it's enabled by default.
Note, that local-migration (including migrating opened FDs through migration channel, which must be UNIX socket), is enabled by global "local" migration parameters. But individual devices may have additional options to enable/disable it per device. The tricky thing, is that we need to know, should we do call open/connect in TAP initialization code. That means, that on incoming migration we want to know status of "local" migration parameter at time of creating TAP device. So, currently, for incoming local migration only TAP device created through QMP is supported, as you need to set migration parameter first (through QMP as well). So the full picture is: On source, to start outgoing "local" migration you need: - migration parameter "local" set to true - "local-migration-supported" TAP option set to true (the default, starting from 11.2 QEMU Machine Types) If at least one of these options is not set, TAP backend doesn't participate in migration. On target, things are more difficult: Same, you need both "local" and "local-migration-supported" be set. And same, if one of them is not set, TAP backend is initialized as usual, and doesn't accept any incoming state. Additionally, if you are going to set "local", it must be set before creating the TAP device. If TAP device created with "local" unset, it initializes as usual. If you enable "local" after it and start incoming migration, it will fail in .pre_load handler of TAP backend. Moreover, there are interface restrictions: if you create TAP device when QEMU is in INCOMING state, and both "local" and "local-migration-supported" set, most of TAP options are not allowed, and script/downscript are required to be explicitly unset (set to "" or "no"). Signed-off-by: Vladimir Sementsov-Ogievskiy <[email protected]> --- In v17 thread we considered making new qapi parameters "unstable". Still, it was because of doubtful "incoming-fds" parameter. In this v18 it's completely removed, and we may live only with "local" migration parameter + "local-migration-supported" TAP option. This way the series doesn't conflict with suggested by Peter [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters So, I think, worth making one more iteration as "stable candidate". If still some doubts, not problem for me to resend with "unstable" features. hw/core/machine.c | 7 ++ net/tap.c | 162 ++++++++++++++++++++++++++++++++++++++++++++-- qapi/net.json | 23 ++++++- 3 files changed, 185 insertions(+), 7 deletions(-) diff --git a/hw/core/machine.c b/hw/core/machine.c index 15886a56b19..2a1abb29041 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -36,10 +36,16 @@ #include "hw/virtio/virtio-pci.h" #include "hw/virtio/virtio-net.h" #include "hw/virtio/virtio-iommu.h" +#include "net/tap.h" #include "hw/acpi/generic_event_device.h" #include "qemu/audio.h" #include "hw/arm/smmuv3.h" +/* + * TODO: When hw_compat_11_1 appears, local-migration-supported=false + * for TYPE_TAP_NETDEV should be move to it from hw_compat_11_0. + */ + GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, @@ -49,6 +55,7 @@ GlobalProperty hw_compat_11_0[] = { { TYPE_ARM_SMMUV3, "ssidsize", "0" }, { TYPE_ARM_SMMUV3, "oas", "44" }, { "migration", "switchover-ack-legacy", "on" }, + { TYPE_TAP_NETDEV, "local-migration-supported", "false" }, }; const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0); diff --git a/net/tap.c b/net/tap.c index 605e5867b01..adcac70b7ea 100644 --- a/net/tap.c +++ b/net/tap.c @@ -38,12 +38,17 @@ #include "monitor/monitor.h" #include "system/runstate.h" #include "system/system.h" +#include "migration/misc.h" #include "qapi/error.h" #include "qemu/cutils.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" #include "qemu/sockets.h" #include "hw/virtio/vhost.h" +#include "hw/core/vmstate-if.h" +#include "migration/vmstate.h" +#include "qom/object.h" +#include "qom/compat-properties.h" #include "net/tap.h" #include "net/util.h" @@ -71,6 +76,8 @@ static const int kernel_feature_bits[] = { OBJECT_DECLARE_SIMPLE_TYPE(TAPState, TAP_NETDEV) +static const VMStateDescription vmstate_tap; + struct TAPState { Object parent_obj; @@ -95,6 +102,7 @@ struct TAPState { int queue_index; bool read_poll_detached; VMChangeStateEntry *vmstate; + bool local_migration_supported; }; static void launch_script(const char *setup_script, const char *ifname, @@ -405,6 +413,8 @@ static void tap_cleanup(NetClientState *nc) tap_write_poll(s, false); close(s->fd); s->fd = -1; + + vmstate_unregister(VMSTATE_IF(s), &vmstate_tap, s); } static void tap_poll(NetClientState *nc, bool enable) @@ -441,6 +451,73 @@ static VHostNetState *tap_get_vhost_net(NetClientState *nc) return s->vhost_net; } +static bool tap_is_wait_incoming(NetClientState *nc) +{ + TAPState *s = container_of(nc, TAPState, nc); + assert(nc->info->type == NET_CLIENT_DRIVER_TAP); + return s->fd == -1; +} + +static int tap_pre_load(void *opaque) +{ + TAPState *s = opaque; + + if (s->fd != -1) { + error_report( + "TAP is already initialized and cannot receive incoming fd"); + return -EINVAL; + } + + return 0; +} + +static bool tap_setup_vhost(TAPState *s, Error **errp); + +static int tap_post_load(void *opaque, int version_id) +{ + TAPState *s = opaque; + Error *local_err = NULL; + + tap_read_poll(s, true); + + if (s->fd < 0) { + return -1; + } + + if (!tap_setup_vhost(s, &local_err)) { + error_prepend(&local_err, + "Failed to setup vhost during TAP post-load: "); + error_report_err(local_err); + return -1; + } + + return 0; +} + +static bool tap_needed(void *opaque) +{ + TAPState *s = opaque; + + return s->local_migration_supported && migrate_local(); +} + +static const VMStateDescription vmstate_tap = { + .name = "net-tap", + .priority = MIG_PRI_BACKEND, + .pre_load = tap_pre_load, + .post_load = tap_post_load, + .needed = tap_needed, + .fields = (const VMStateField[]) { + VMSTATE_FD(fd, TAPState), + VMSTATE_BOOL(using_vnet_hdr, TAPState), + VMSTATE_BOOL(has_ufo, TAPState), + VMSTATE_BOOL(has_uso, TAPState), + VMSTATE_BOOL(has_tunnel, TAPState), + VMSTATE_BOOL(enabled, TAPState), + VMSTATE_UINT32(host_vnet_hdr_len, TAPState), + VMSTATE_END_OF_LIST() + } +}; static char *tap_vmstate_if_get_id(VMStateIf *obj) { @@ -449,17 +526,42 @@ static char *tap_vmstate_if_get_id(VMStateIf *obj) return res; } +static bool tap_get_local_migration_supported_prop(Object *obj, Error **errp) +{ + TAPState *s = TAP_NETDEV(obj); + return s->local_migration_supported; +} + +static void tap_set_local_migration_supported_prop(Object *obj, bool value, + Error **errp) +{ + TAPState *s = TAP_NETDEV(obj); + s->local_migration_supported = value; +} + +static void tap_instance_init(Object *obj) +{ + TAPState *s = TAP_NETDEV(obj); + s->local_migration_supported = true; +} + static void tap_class_init(ObjectClass *klass, const void *data) { VMStateIfClass *vc = VMSTATE_IF_CLASS(klass); vc->get_id = tap_vmstate_if_get_id; + + object_class_property_add_bool(klass, "local-migration-supported", + tap_get_local_migration_supported_prop, + tap_set_local_migration_supported_prop); } static const TypeInfo tap_netdev_info = { .name = TYPE_TAP_NETDEV, .parent = TYPE_OBJECT, .instance_size = sizeof(TAPState), + .instance_init = tap_instance_init, + .instance_post_init = object_apply_compat_props, .class_init = tap_class_init, .interfaces = (const InterfaceInfo[]) { { TYPE_VMSTATE_IF }, @@ -492,13 +594,16 @@ static NetClientInfo net_tap_info = { .set_vnet_le = tap_set_vnet_le, .set_vnet_be = tap_set_vnet_be, .set_steering_ebpf = tap_set_steering_ebpf, + .is_wait_incoming = tap_is_wait_incoming, .get_vhost_net = tap_get_vhost_net, }; static TAPState *new_tap(NetClientState *peer, const char *model, const char *name, - int queue_index) + int queue_index, + bool has_local_migration_supported, + bool local_migration_supported) { TAPState *s = TAP_NETDEV(object_new(TYPE_TAP_NETDEV)); @@ -507,6 +612,12 @@ static TAPState *new_tap(NetClientState *peer, s->queue_index = queue_index; + if (has_local_migration_supported) { + s->local_migration_supported = local_migration_supported; + } + + vmstate_register(VMSTATE_IF(s), VMSTATE_INSTANCE_ID_ANY, &vmstate_tap, s); + return s; } @@ -515,10 +626,14 @@ static TAPState *net_tap_fd_init(NetClientState *peer, const char *name, int fd, int vnet_hdr, - int queue_index) + int queue_index, + bool has_local_migration_supported, + bool local_migration_supported) { NetOffloads ol = {}; - TAPState *s = new_tap(peer, model, name, queue_index); + TAPState *s = new_tap(peer, model, name, queue_index, + has_local_migration_supported, + local_migration_supported); s->fd = fd; s->host_vnet_hdr_len = vnet_hdr ? sizeof(struct virtio_net_hdr) : 0; @@ -755,7 +870,7 @@ int net_init_bridge(const Netdev *netdev, const char *name, close(fd); return -1; } - s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr, 0); + s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr, 0, true, false); qemu_set_info_str(&s->nc, "helper=%s,br=%s", helper, br); @@ -835,7 +950,9 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer, Error **errp) { TAPState *s = net_tap_fd_init(peer, tap->helper ? "bridge" : "tap", - name, fd, vnet_hdr, queue_index); + name, fd, vnet_hdr, queue_index, + tap->has_local_migration_supported, + tap->local_migration_supported); bool sndbuf_required = tap->has_sndbuf; int sndbuf = (tap->has_sndbuf && tap->sndbuf) ? MIN(tap->sndbuf, INT_MAX) : INT_MAX; @@ -983,6 +1100,7 @@ int net_init_tap(const Netdev *netdev, const char *name, /* for the no-fd, no-helper case */ char ifname[128]; int *fds = NULL, *vhost_fds = NULL; + bool incoming_fds; assert(netdev->type == NET_CLIENT_DRIVER_TAP); tap = &netdev->u.tap; @@ -1005,6 +1123,23 @@ int net_init_tap(const Netdev *netdev, const char *name, return -1; } + incoming_fds = tap->local_migration_supported && migrate_local() && + runstate_check(RUN_STATE_INMIGRATE); + + if (incoming_fds && + (tap->fd || tap->fds || tap->helper || tap->br || tap->ifname || + tap->has_sndbuf || tap->has_vnet_hdr || + !tap_is_explicit_no_script(tap->script) || + !tap_is_explicit_no_script(tap->downscript))) { + error_setg(errp, "Local incoming migration of TAP device (-incoming, " + "migration parameter @local is set, " + "TAP parameter @local-migration-supported is set) " + "is incompatible with " + "fd=, fds=, helper=, br=, ifname=, sndbuf= and vnet_hdr=, " + "and requires explicit empty script= and downscript="); + return -1; + } + queues = tap_parse_fds_and_queues(tap, &fds, errp); if (queues < 0) { return -1; @@ -1023,7 +1158,22 @@ int net_init_tap(const Netdev *netdev, const char *name, goto fail; } - if (fds) { + if (incoming_fds) { + for (i = 0; i < queues; i++) { + TAPState *s = new_tap(peer, "tap", name, i, + tap->has_local_migration_supported, + tap->local_migration_supported); + qemu_set_info_str(&s->nc, "incoming"); + + s->fd = -1; + if (vhost_fds) { + s->vhostfd = vhost_fds[i]; + s->vhost_busyloop_timeout = tap->has_poll_us ? tap->poll_us : 0; + } else { + s->vhostfd = -1; + } + } + } else if (fds) { for (i = 0; i < queues; i++) { if (i == 0) { vnet_hdr = tap_probe_vnet_hdr(fds[i], errp); diff --git a/qapi/net.json b/qapi/net.json index ada0329ef9d..2070979f20e 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -435,6 +435,26 @@ # @poll-us: maximum number of microseconds that could be spent on busy # polling for tap (since 2.7) # +# @local-migration-supported: enable local migration for this TAP +# backend. When set, local migration is enabled/disabled by +# migration parameter @local for this TAP backend. When unset, +# migration parameter @local is ignored for this TAP backend. +# To be able to do incoming local migration of TAP backend, +# migration parameter @local must be set _before_ creating the +# TAP backend. Otherwise, TAP backend is initialized as usual, +# opening/creating TAP devices in kernel. In this case further +# local incoming migration (with migration parameter @local set +# after creating TAP backend with @local-migration-supporeted +# parameter set) will simply fail. +# Moreover, when QEMU is in incoming migration state, migration +# parameter @local is set and @local-migration-supported is set, +# following options are not supported and must not be set: +# @ds, @fds, @helper, @br, @ifname, @sndbuf, @vnet_hdr. +# Additionally in this case @script and @downscipt must be +# explicitly disabled (empty strings or "no"). +# (Since 11.2. Defaults to true for QEMU Machine Types >= 11.2, +# and to false for QEMU Machine Types < 11.2) +# # Since: 1.2 ## { 'struct': 'NetdevTapOptions', @@ -453,7 +473,8 @@ '*vhostfds': 'str', '*vhostforce': 'bool', '*queues': 'uint32', - '*poll-us': 'uint32'} } + '*poll-us': 'uint32', + '*local-migration-supported': 'bool' } } ## # @NetdevSocketOptions: -- 2.43.0
