Re: [PATCH v5 0/7] vhost-user-blk: fix the migration issue and enhance qtests
On Tue, Sep 29, 2020 at 12:48:38PM +0300, Dima Stepanov wrote: > On Tue, Sep 29, 2020 at 03:13:09AM -0400, Michael S. Tsirkin wrote: > > On Sun, Sep 27, 2020 at 09:48:28AM +0300, Dima Stepanov wrote: > > > On Thu, Sep 24, 2020 at 07:26:14AM -0400, Michael S. Tsirkin wrote: > > > > On Fri, Sep 11, 2020 at 11:39:42AM +0300, Dima Stepanov wrote: > > > > > v4 -> v5: > > > > > - vhost: check queue state in the vhost_dev_set_log routine > > > > > tests/qtest/vhost-user-test: prepare the tests for adding new > > > > > dev class > > > > > tests/qtest/vhost-user-test: add support for the > > > > > vhost-user-blk device > > > > > tests/qtest/vhost-user-test: add migrate_reconnect test > > > > > Reviewed-by: Raphael Norwitz > > > > > - Update qtest, by merging vhost-user-blk "if" case with the > > > > > virtio-blk case. > > > > > > > > I dropped patches 3-7 since they were stalling on some systems. > > > > Pls work with Peter Maydell (cc'd) to figure it out. > > > Thanks! > > > > > > Peter, can you share any details for the stalling errors with me? > > > > I can say for sure that even on x86/linux the affected tests take > > much longer to run with these applied. > > I'd suggest making sure there are no timeouts involved in the good case > Could you help me to reproduce it? Because on my system i see only 10+ seconds > increase for the qos-test set to pass (both x86_64 and i386). So on the > current master i'm running it like: > - ./configure --target-list="x86_64-softmmu i386-softmmu" > - with no patch set: > time QTEST_QEMU_BINARY=./build/x86_64-softmmu/qemu-system-x86_64 > ./build/tests/qtest/qos-test > real0m6.394s > user0m3.643s > sys 0m3.477s > - without patch 7 (where i include vhost-user-net tests also): > real0m9.955s > user0m4.133s > sys 0m4.397s > - full patch set: > real0m17.263s > user0m4.530s > sys 0m4.802s > For i386 target i see pretty the same numbers: > time QTEST_QEMU_BINARY=./build/i386-softmmu/qemu-system-i386 > ./build/tests/qtest/qos-test > real0m17.386s > user0m4.503s > sys 0m4.911s > So it looks like that i'm missing some step to reproduce an issue. > > And if i run the exact test it takes ~2-3s to pass: > $ time QTEST_QEMU_BINARY=./build/x86_64-softmmu/qemu-system-x86_64 > ./build/tests/qtest/qos-test -p > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/vhost-user-blk-pci/vhost-user-blk/vhost-user-blk-tests/reconnect > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/vhost-user-blk-pci/vhost-user-blk/vhost-user-blk-tests/reconnect: > OK > real0m2.253s > user0m0.118s > sys 0m0.104s > And same numbers for i386. > Sorry, but still fail to reproduce the stall issue on both i386 and x86_64 targets. Any help is highly appreciated. Thanks, Dima. > > > > > > > > > > > > > > > v3 -> v4: > > > > > - vhost: recheck dev state in the vhost_migration_log routine > > > > > Reviewed-by: Raphael Norwitz > > > > > - vhost: check queue state in the vhost_dev_set_log routine > > > > > Use "continue" instead of "break" to handle non-initialized > > > > > virtqueue case. > > > > > > > > > > v2 -> v3: > > > > > - update commit message for the > > > > > "vhost: recheck dev state in the vhost_migration_log routine" > > > > > commit > > > > > - rename "started" field of the VhostUserBlk structure to > > > > > "started_vu", so there will be no confustion with the VHOST > > > > > started > > > > > field > > > > > - update vhost-user-test.c to always initialize nq local variable > > > > > (spotted by patchew) > > > > > > > > > > v1 -> v2: > > > > > - add comments to connected/started fields in the header file > > > > > - move the "s->started" logic from the vhost_user_blk_disconnect > > > > > routine to the vhost_user_blk_stop routine > > > > > > > > > > Reference e-mail threads: > > > > > - > > > > > https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html > > > > > - > > &
Re: [PATCH v5 0/7] vhost-user-blk: fix the migration issue and enhance qtests
On Tue, Sep 29, 2020 at 03:13:09AM -0400, Michael S. Tsirkin wrote: > On Sun, Sep 27, 2020 at 09:48:28AM +0300, Dima Stepanov wrote: > > On Thu, Sep 24, 2020 at 07:26:14AM -0400, Michael S. Tsirkin wrote: > > > On Fri, Sep 11, 2020 at 11:39:42AM +0300, Dima Stepanov wrote: > > > > v4 -> v5: > > > > - vhost: check queue state in the vhost_dev_set_log routine > > > > tests/qtest/vhost-user-test: prepare the tests for adding new > > > > dev class > > > > tests/qtest/vhost-user-test: add support for the > > > > vhost-user-blk device > > > > tests/qtest/vhost-user-test: add migrate_reconnect test > > > > Reviewed-by: Raphael Norwitz > > > > - Update qtest, by merging vhost-user-blk "if" case with the > > > > virtio-blk case. > > > > > > I dropped patches 3-7 since they were stalling on some systems. > > > Pls work with Peter Maydell (cc'd) to figure it out. > > Thanks! > > > > Peter, can you share any details for the stalling errors with me? > > I can say for sure that even on x86/linux the affected tests take > much longer to run with these applied. > I'd suggest making sure there are no timeouts involved in the good case Could you help me to reproduce it? Because on my system i see only 10+ seconds increase for the qos-test set to pass (both x86_64 and i386). So on the current master i'm running it like: - ./configure --target-list="x86_64-softmmu i386-softmmu" - with no patch set: time QTEST_QEMU_BINARY=./build/x86_64-softmmu/qemu-system-x86_64 ./build/tests/qtest/qos-test real0m6.394s user0m3.643s sys 0m3.477s - without patch 7 (where i include vhost-user-net tests also): real0m9.955s user0m4.133s sys 0m4.397s - full patch set: real0m17.263s user0m4.530s sys 0m4.802s For i386 target i see pretty the same numbers: time QTEST_QEMU_BINARY=./build/i386-softmmu/qemu-system-i386 ./build/tests/qtest/qos-test real0m17.386s user0m4.503s sys 0m4.911s So it looks like that i'm missing some step to reproduce an issue. And if i run the exact test it takes ~2-3s to pass: $ time QTEST_QEMU_BINARY=./build/x86_64-softmmu/qemu-system-x86_64 ./build/tests/qtest/qos-test -p /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/vhost-user-blk-pci/vhost-user-blk/vhost-user-blk-tests/reconnect /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/vhost-user-blk-pci/vhost-user-blk/vhost-user-blk-tests/reconnect: OK real0m2.253s user0m0.118s sys 0m0.104s And same numbers for i386. > > > > > > > > > > > v3 -> v4: > > > > - vhost: recheck dev state in the vhost_migration_log routine > > > > Reviewed-by: Raphael Norwitz > > > > - vhost: check queue state in the vhost_dev_set_log routine > > > > Use "continue" instead of "break" to handle non-initialized > > > > virtqueue case. > > > > > > > > v2 -> v3: > > > > - update commit message for the > > > > "vhost: recheck dev state in the vhost_migration_log routine" commit > > > > - rename "started" field of the VhostUserBlk structure to > > > > "started_vu", so there will be no confustion with the VHOST started > > > > field > > > > - update vhost-user-test.c to always initialize nq local variable > > > > (spotted by patchew) > > > > > > > > v1 -> v2: > > > > - add comments to connected/started fields in the header file > > > > - move the "s->started" logic from the vhost_user_blk_disconnect > > > > routine to the vhost_user_blk_stop routine > > > > > > > > Reference e-mail threads: > > > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html > > > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html > > > > > > > > If vhost-user daemon is used as a backend for the vhost device, then we > > > > should consider a possibility of disconnect at any moment. There was a > > > > general > > > > question here: should we consider it as an error or okay state for the > > > > vhost-user > > > > devices during migration process? > > > > I think the disconnect event for the vhost-user devices should not > > > > break the > > > > migration process, because: > > > > - the device will be in the stopped state, so it will not be cha
Re: [PATCH v5 0/7] vhost-user-blk: fix the migration issue and enhance qtests
On Thu, Sep 24, 2020 at 07:26:14AM -0400, Michael S. Tsirkin wrote: > On Fri, Sep 11, 2020 at 11:39:42AM +0300, Dima Stepanov wrote: > > v4 -> v5: > > - vhost: check queue state in the vhost_dev_set_log routine > > tests/qtest/vhost-user-test: prepare the tests for adding new > > dev class > > tests/qtest/vhost-user-test: add support for the > > vhost-user-blk device > > tests/qtest/vhost-user-test: add migrate_reconnect test > > Reviewed-by: Raphael Norwitz > > - Update qtest, by merging vhost-user-blk "if" case with the > > virtio-blk case. > > I dropped patches 3-7 since they were stalling on some systems. > Pls work with Peter Maydell (cc'd) to figure it out. Thanks! Peter, can you share any details for the stalling errors with me? > > > > v3 -> v4: > > - vhost: recheck dev state in the vhost_migration_log routine > > Reviewed-by: Raphael Norwitz > > - vhost: check queue state in the vhost_dev_set_log routine > > Use "continue" instead of "break" to handle non-initialized > > virtqueue case. > > > > v2 -> v3: > > - update commit message for the > > "vhost: recheck dev state in the vhost_migration_log routine" commit > > - rename "started" field of the VhostUserBlk structure to > > "started_vu", so there will be no confustion with the VHOST started > > field > > - update vhost-user-test.c to always initialize nq local variable > > (spotted by patchew) > > > > v1 -> v2: > > - add comments to connected/started fields in the header file > > - move the "s->started" logic from the vhost_user_blk_disconnect > > routine to the vhost_user_blk_stop routine > > > > Reference e-mail threads: > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html > > > > If vhost-user daemon is used as a backend for the vhost device, then we > > should consider a possibility of disconnect at any moment. There was a > > general > > question here: should we consider it as an error or okay state for the > > vhost-user > > devices during migration process? > > I think the disconnect event for the vhost-user devices should not break the > > migration process, because: > > - the device will be in the stopped state, so it will not be changed > > during migration > > - if reconnect will be made the migration log will be reinitialized as > > part of reconnect/init process: > > #0 vhost_log_global_start (listener=0x563989cf7be0) > > at hw/virtio/vhost.c:920 > > #1 0x56398603d8bc in listener_add_address_space > > (listener=0x563989cf7be0, > > as=0x563986ea4340 ) > > at softmmu/memory.c:2664 > > #2 0x56398603dd30 in memory_listener_register > > (listener=0x563989cf7be0, > > as=0x563986ea4340 ) > > at softmmu/memory.c:2740 > > #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, > > opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, > > busyloop_timeout=0) > > at hw/virtio/vhost.c:1385 > > #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) > > at hw/block/vhost-user-blk.c:315 > > #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, > > event=CHR_EVENT_OPENED) > > at hw/block/vhost-user-blk.c:379 > > The first patch in the patchset fixes this issue by setting vhost device to > > the > > stopped state in the disconnect handler and check it the > > vhost_migration_log() > > routine before returning from the function. > > qtest framework was updated to test vhost-user-blk functionality. The > > vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to > > reproduce > > the original issue found. > > > > Dima Stepanov (7): > > vhost: recheck dev state in the vhost_migration_log routine > > vhost: check queue state in the vhost_dev_set_log routine > > tests/qtest/vhost-user-test: prepare the tests for adding new dev > > class > > tests/qtest/libqos/virtio-blk: add support for vhost-user-blk > > tests/qtest/vhost-user-test: add support for the vhost-user-blk device > > tests/qtest/vhost-user-test: add migrate_reconnect test > > tests/qtest/vhost-user-test: enable the reconnect tests > > > > hw/block/vhost-user-blk.c | 19 ++- > > hw/virtio/vhost.c | 39 - > > include/hw/virtio/vhost-user-blk.h | 10 ++ > > tests/qtest/libqos/virtio-blk.c| 14 +- > > tests/qtest/vhost-user-test.c | 290 > > +++-- > > 5 files changed, 322 insertions(+), 50 deletions(-) > > > > -- > > 2.7.4 >
Re: [PATCH v5 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
On Mon, Sep 21, 2020 at 07:04:20PM -0400, Raphael Norwitz wrote:
> MST already sent a PR with all the patches :)
Thank you! )
>
> On Wed, Sep 16, 2020 at 6:13 PM Dima Stepanov wrote:
> >
> > On Mon, Sep 14, 2020 at 09:23:42PM -0400, Raphael Norwitz wrote:
> > > On Fri, Sep 11, 2020 at 4:43 AM Dima Stepanov
> > > wrote:
> > > >
> > > > Add support for the vhost-user-blk-pci device. This node can be used by
> > > > the vhost-user-blk tests. Tests for the vhost-user-blk device are added
> > > > in the following patches.
> > > >
> > > > Signed-off-by: Dima Stepanov
> > >
> > > Reviewed-by: Raphael Norwitz
> > Hi,
> >
> > Looks like that all the patch set is reviewed except 7/7. If it is an
> > issue, we can cut it from the set and merge other six commits.
> >
> > Raphael,
> >
> > Will you take it for merge?
> >
> > Thanks, Dima.
> >
> > >
> > > > ---
> > > > tests/qtest/libqos/virtio-blk.c | 14 +-
> > > > 1 file changed, 13 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/tests/qtest/libqos/virtio-blk.c
> > > > b/tests/qtest/libqos/virtio-blk.c
> > > > index 5da0259..c0fd9d2 100644
> > > > --- a/tests/qtest/libqos/virtio-blk.c
> > > > +++ b/tests/qtest/libqos/virtio-blk.c
> > > > @@ -30,7 +30,8 @@
> > > > static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
> > > > const char *interface)
> > > > {
> > > > -if (!g_strcmp0(interface, "virtio-blk")) {
> > > > +if (!g_strcmp0(interface, "virtio-blk") ||
> > > > +!g_strcmp0(interface, "vhost-user-blk")) {
> > > > return v_blk;
> > > > }
> > > > if (!g_strcmp0(interface, "virtio")) {
> > > > @@ -120,6 +121,17 @@ static void virtio_blk_register_nodes(void)
> > > > qos_node_produces("virtio-blk-pci", "virtio-blk");
> > > >
> > > > g_free(arg);
> > > > +
> > > > +/* vhost-user-blk-pci */
> > > > +arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
> > > > +PCI_SLOT, PCI_FN);
> > > > +opts.extra_device_opts = arg;
> > > > +add_qpci_address(, );
> > > > +qos_node_create_driver("vhost-user-blk-pci",
> > > > virtio_blk_pci_create);
> > > > +qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
> > > > +qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
> > > > +
> > > > +g_free(arg);
> > > > }
> > > >
> > > > libqos_init(virtio_blk_register_nodes);
> > > > --
> > > > 2.7.4
> > > >
> > > >
Re: [PATCH v5 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
On Mon, Sep 14, 2020 at 09:23:42PM -0400, Raphael Norwitz wrote:
> On Fri, Sep 11, 2020 at 4:43 AM Dima Stepanov wrote:
> >
> > Add support for the vhost-user-blk-pci device. This node can be used by
> > the vhost-user-blk tests. Tests for the vhost-user-blk device are added
> > in the following patches.
> >
> > Signed-off-by: Dima Stepanov
>
> Reviewed-by: Raphael Norwitz
Hi,
Looks like that all the patch set is reviewed except 7/7. If it is an
issue, we can cut it from the set and merge other six commits.
Raphael,
Will you take it for merge?
Thanks, Dima.
>
> > ---
> > tests/qtest/libqos/virtio-blk.c | 14 +-
> > 1 file changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/qtest/libqos/virtio-blk.c
> > b/tests/qtest/libqos/virtio-blk.c
> > index 5da0259..c0fd9d2 100644
> > --- a/tests/qtest/libqos/virtio-blk.c
> > +++ b/tests/qtest/libqos/virtio-blk.c
> > @@ -30,7 +30,8 @@
> > static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
> > const char *interface)
> > {
> > -if (!g_strcmp0(interface, "virtio-blk")) {
> > +if (!g_strcmp0(interface, "virtio-blk") ||
> > +!g_strcmp0(interface, "vhost-user-blk")) {
> > return v_blk;
> > }
> > if (!g_strcmp0(interface, "virtio")) {
> > @@ -120,6 +121,17 @@ static void virtio_blk_register_nodes(void)
> > qos_node_produces("virtio-blk-pci", "virtio-blk");
> >
> > g_free(arg);
> > +
> > +/* vhost-user-blk-pci */
> > +arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
> > +PCI_SLOT, PCI_FN);
> > +opts.extra_device_opts = arg;
> > +add_qpci_address(, );
> > +qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
> > +qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
> > +qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
> > +
> > +g_free(arg);
> > }
> >
> > libqos_init(virtio_blk_register_nodes);
> > --
> > 2.7.4
> >
> >
[PATCH v5 7/7] tests/qtest/vhost-user-test: enable the reconnect tests
For now a QTEST_VHOST_USER_FIXME environment variable is used to
separate reconnect tests for the vhost-user-net device. Looks like the
reconnect functionality is pretty stable, so this separation is
deprecated.
Remove it and enable these tests for the default run.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +++--
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 4b715d3..4b96312 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -1140,20 +1140,17 @@ static void register_vhost_user_test(void)
"virtio-net",
test_migrate, );
-/* keeps failing on build-system since Aug 15 2017 */
-if (getenv("QTEST_VHOST_USER_FIXME")) {
-opts.before = vhost_user_test_setup_reconnect;
-qos_add_test("vhost-user/reconnect", "virtio-net",
- test_reconnect, );
-
-opts.before = vhost_user_test_setup_connect_fail;
-qos_add_test("vhost-user/connect-fail", "virtio-net",
- test_vhost_user_started, );
-
-opts.before = vhost_user_test_setup_flags_mismatch;
-qos_add_test("vhost-user/flags-mismatch", "virtio-net",
- test_vhost_user_started, );
-}
+opts.before = vhost_user_test_setup_reconnect;
+qos_add_test("vhost-user/reconnect", "virtio-net",
+ test_reconnect, );
+
+opts.before = vhost_user_test_setup_connect_fail;
+qos_add_test("vhost-user/connect-fail", "virtio-net",
+ test_vhost_user_started, );
+
+opts.before = vhost_user_test_setup_flags_mismatch;
+qos_add_test("vhost-user/flags-mismatch", "virtio-net",
+ test_vhost_user_started, );
opts.before = vhost_user_test_setup_multiqueue;
opts.edge.extra_device_opts = "mq=on";
--
2.7.4
[PATCH v5 3/7] tests/qtest/vhost-user-test: prepare the tests for adding new dev class
For now only vhost-user-net device is supported by the test. Other
vhost-user devices are not tested. As a first step make source code
refactoring so new devices can reuse the same test routines. To make
this provide a new vhost_user_ops structure with the methods to
initialize device, its command line or make a proper vhost-user
responses.
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
tests/qtest/vhost-user-test.c | 105 ++
1 file changed, 76 insertions(+), 29 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9ee0f1e..3df5322 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -135,6 +135,10 @@ enum {
TEST_FLAGS_END,
};
+enum {
+VHOST_USER_NET,
+};
+
typedef struct TestServer {
gchar *socket_path;
gchar *mig_path;
@@ -154,10 +158,25 @@ typedef struct TestServer {
bool test_fail;
int test_flags;
int queues;
+struct vhost_user_ops *vu_ops;
} TestServer;
+struct vhost_user_ops {
+/* Device types. */
+int type;
+void (*append_opts)(TestServer *s, GString *cmd_line,
+const char *chr_opts);
+
+/* VHOST-USER commands. */
+void (*set_features)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
+void (*get_protocol_features)(TestServer *s,
+CharBackend *chr, VhostUserMsg *msg);
+};
+
static const char *init_hugepagefs(void);
-static TestServer *test_server_new(const gchar *name);
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops);
static void test_server_free(TestServer *server);
static void test_server_listen(TestServer *server);
@@ -167,7 +186,7 @@ enum test_memfd {
TEST_MEMFD_NO,
};
-static void append_vhost_opts(TestServer *s, GString *cmd_line,
+static void append_vhost_net_opts(TestServer *s, GString *cmd_line,
const char *chr_opts)
{
g_string_append_printf(cmd_line, QEMU_CMD_CHR QEMU_CMD_NETDEV,
@@ -332,25 +351,15 @@ static void chr_read(void *opaque, const uint8_t *buf,
int size)
break;
case VHOST_USER_SET_FEATURES:
-g_assert_cmpint(msg.payload.u64 & (0x1ULL <<
VHOST_USER_F_PROTOCOL_FEATURES),
-!=, 0ULL);
-if (s->test_flags == TEST_FLAGS_DISCONNECT) {
-qemu_chr_fe_disconnect(chr);
-s->test_flags = TEST_FLAGS_BAD;
+if (s->vu_ops->set_features) {
+s->vu_ops->set_features(s, chr, );
}
break;
case VHOST_USER_GET_PROTOCOL_FEATURES:
-/* send back features to qemu */
-msg.flags |= VHOST_USER_REPLY_MASK;
-msg.size = sizeof(m.payload.u64);
-msg.payload.u64 = 1 << VHOST_USER_PROTOCOL_F_LOG_SHMFD;
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_CROSS_ENDIAN;
-if (s->queues > 1) {
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_MQ;
+if (s->vu_ops->get_protocol_features) {
+s->vu_ops->get_protocol_features(s, chr, );
}
-p = (uint8_t *)
-qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
case VHOST_USER_GET_VRING_BASE:
@@ -467,7 +476,8 @@ static const char *init_hugepagefs(void)
#endif
}
-static TestServer *test_server_new(const gchar *name)
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops)
{
TestServer *server = g_new0(TestServer, 1);
char template[] = "/tmp/vhost-test-XX";
@@ -495,6 +505,7 @@ static TestServer *test_server_new(const gchar *name)
server->log_fd = -1;
server->queues = 1;
+server->vu_ops = ops;
return server;
}
@@ -669,11 +680,11 @@ static void vhost_user_test_cleanup(void *s)
static void *vhost_user_test_setup(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_AUTO);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -682,11 +693,11 @@ static void *vhost_user_test_setup(GString *cmd_line,
void *arg)
static void *vhost_user_test_setup_memfd(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_YES);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_t
[PATCH v5 6/7] tests/qtest/vhost-user-test: add migrate_reconnect test
Add new migrate_reconnect test for the vhost-user-blk device. Perform a
disconnect after sending response for the VHOST_USER_SET_LOG_BASE
command.
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
tests/qtest/vhost-user-test.c | 25 +
1 file changed, 25 insertions(+)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index a8af613..4b715d3 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -146,6 +146,7 @@ static VhostUserMsg m __attribute__ ((unused));
enum {
TEST_FLAGS_OK,
TEST_FLAGS_DISCONNECT,
+TEST_FLAGS_MIGRATE_DISCONNECT,
TEST_FLAGS_BAD,
TEST_FLAGS_END,
};
@@ -436,6 +437,15 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE);
g_cond_broadcast(>data_cond);
+/*
+ * Perform disconnect after sending a response. In this
+ * case the next write command on the QEMU side (for now
+ * it is SET_FEATURES will return -1, because of disconnect.
+ */
+if (s->test_flags == TEST_FLAGS_MIGRATE_DISCONNECT) {
+qemu_chr_fe_disconnect(chr);
+s->test_flags = TEST_FLAGS_BAD;
+}
break;
case VHOST_USER_SET_VRING_BASE:
@@ -737,6 +747,17 @@ static void *vhost_user_test_setup_memfd(GString
*cmd_line, void *arg)
return server;
}
+static void *vhost_user_test_setup_migrate_reconnect(GString *cmd_line,
+void *arg)
+{
+TestServer *server;
+
+server = vhost_user_test_setup_memfd(cmd_line, arg);
+server->test_flags = TEST_FLAGS_MIGRATE_DISCONNECT;
+
+return server;
+}
+
static void test_read_guest_mem(void *obj, void *arg, QGuestAllocator *alloc)
{
TestServer *server = arg;
@@ -1150,5 +1171,9 @@ static void register_vhost_user_test(void)
opts.before = vhost_user_test_setup_memfd;
qos_add_test("migrate", "vhost-user-blk",
test_migrate, );
+
+opts.before = vhost_user_test_setup_migrate_reconnect;
+qos_add_test("migrate_reconnect", "vhost-user-blk",
+ test_migrate, );
}
libqos_init(register_vhost_user_test);
--
2.7.4
[PATCH v5 0/7] vhost-user-blk: fix the migration issue and enhance qtests
v4 -> v5: - vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test Reviewed-by: Raphael Norwitz - Update qtest, by merging vhost-user-blk "if" case with the virtio-blk case. v3 -> v4: - vhost: recheck dev state in the vhost_migration_log routine Reviewed-by: Raphael Norwitz - vhost: check queue state in the vhost_dev_set_log routine Use "continue" instead of "break" to handle non-initialized virtqueue case. v2 -> v3: - update commit message for the "vhost: recheck dev state in the vhost_migration_log routine" commit - rename "started" field of the VhostUserBlk structure to "started_vu", so there will be no confustion with the VHOST started field - update vhost-user-test.c to always initialize nq local variable (spotted by patchew) v1 -> v2: - add comments to connected/started fields in the header file - move the "s->started" logic from the vhost_user_blk_disconnect routine to the vhost_user_blk_stop routine Reference e-mail threads: - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html If vhost-user daemon is used as a backend for the vhost device, then we should consider a possibility of disconnect at any moment. There was a general question here: should we consider it as an error or okay state for the vhost-user devices during migration process? I think the disconnect event for the vhost-user devices should not break the migration process, because: - the device will be in the stopped state, so it will not be changed during migration - if reconnect will be made the migration log will be reinitialized as part of reconnect/init process: #0 vhost_log_global_start (listener=0x563989cf7be0) at hw/virtio/vhost.c:920 #1 0x56398603d8bc in listener_add_address_space (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2664 #2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2740 #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, busyloop_timeout=0) at hw/virtio/vhost.c:1385 #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) at hw/block/vhost-user-blk.c:315 #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, event=CHR_EVENT_OPENED) at hw/block/vhost-user-blk.c:379 The first patch in the patchset fixes this issue by setting vhost device to the stopped state in the disconnect handler and check it the vhost_migration_log() routine before returning from the function. qtest framework was updated to test vhost-user-blk functionality. The vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to reproduce the original issue found. Dima Stepanov (7): vhost: recheck dev state in the vhost_migration_log routine vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/libqos/virtio-blk: add support for vhost-user-blk tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test tests/qtest/vhost-user-test: enable the reconnect tests hw/block/vhost-user-blk.c | 19 ++- hw/virtio/vhost.c | 39 - include/hw/virtio/vhost-user-blk.h | 10 ++ tests/qtest/libqos/virtio-blk.c| 14 +- tests/qtest/vhost-user-test.c | 290 +++-- 5 files changed, 322 insertions(+), 50 deletions(-) -- 2.7.4
[PATCH v5 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
Add support for the vhost-user-blk-pci device. This node can be used by
the vhost-user-blk tests. Tests for the vhost-user-blk device are added
in the following patches.
Signed-off-by: Dima Stepanov
---
tests/qtest/libqos/virtio-blk.c | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/tests/qtest/libqos/virtio-blk.c b/tests/qtest/libqos/virtio-blk.c
index 5da0259..c0fd9d2 100644
--- a/tests/qtest/libqos/virtio-blk.c
+++ b/tests/qtest/libqos/virtio-blk.c
@@ -30,7 +30,8 @@
static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
const char *interface)
{
-if (!g_strcmp0(interface, "virtio-blk")) {
+if (!g_strcmp0(interface, "virtio-blk") ||
+!g_strcmp0(interface, "vhost-user-blk")) {
return v_blk;
}
if (!g_strcmp0(interface, "virtio")) {
@@ -120,6 +121,17 @@ static void virtio_blk_register_nodes(void)
qos_node_produces("virtio-blk-pci", "virtio-blk");
g_free(arg);
+
+/* vhost-user-blk-pci */
+arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
+
+g_free(arg);
}
libqos_init(virtio_blk_register_nodes);
--
2.7.4
[PATCH v5 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
Add vhost_user_ops structure for the vhost-user-blk device class. Add
the test_reconnect and test_migrate tests for this device.
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
tests/qtest/vhost-user-test.c | 139 +-
1 file changed, 137 insertions(+), 2 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 3df5322..a8af613 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -24,6 +24,7 @@
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
#include "libqos/virtio-pci.h"
+#include "libqos/virtio-blk.h"
#include "libqos/malloc-pc.h"
#include "hw/virtio/virtio-net.h"
@@ -31,6 +32,7 @@
#include "standard-headers/linux/vhost_types.h"
#include "standard-headers/linux/virtio_ids.h"
#include "standard-headers/linux/virtio_net.h"
+#include "standard-headers/linux/virtio_blk.h"
#ifdef CONFIG_LINUX
#include
@@ -43,6 +45,7 @@
" -numa node,memdev=mem"
#define QEMU_CMD_CHR" -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=hs0,chardev=%s,vhostforce"
+#define QEMU_CMD_BLKCHR " -chardev socket,id=chdev0,path=%s%s"
#define HUGETLBFS_MAGIC 0x958458f6
@@ -55,6 +58,7 @@
#define VHOST_USER_PROTOCOL_F_MQ 0
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+#define VHOST_USER_PROTOCOL_F_CONFIG 9
#define VHOST_LOG_PAGE 0x1000
@@ -78,6 +82,8 @@ typedef enum VhostUserRequest {
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
+VHOST_USER_GET_CONFIG = 24,
+VHOST_USER_SET_CONFIG = 25,
VHOST_USER_MAX
} VhostUserRequest;
@@ -99,6 +105,14 @@ typedef struct VhostUserLog {
uint64_t mmap_offset;
} VhostUserLog;
+#define VHOST_USER_MAX_CONFIG_SIZE 256
+typedef struct VhostUserConfig {
+uint32_t offset;
+uint32_t size;
+uint32_t flags;
+uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+} VhostUserConfig;
+
typedef struct VhostUserMsg {
VhostUserRequest request;
@@ -114,6 +128,7 @@ typedef struct VhostUserMsg {
struct vhost_vring_addr addr;
VhostUserMemory memory;
VhostUserLog log;
+VhostUserConfig config;
} payload;
} QEMU_PACKED VhostUserMsg;
@@ -137,6 +152,7 @@ enum {
enum {
VHOST_USER_NET,
+VHOST_USER_BLK,
};
typedef struct TestServer {
@@ -166,12 +182,15 @@ struct vhost_user_ops {
int type;
void (*append_opts)(TestServer *s, GString *cmd_line,
const char *chr_opts);
+void (*driver_init)(void *obj, QGuestAllocator *alloc);
/* VHOST-USER commands. */
void (*set_features)(TestServer *s, CharBackend *chr,
VhostUserMsg *msg);
void (*get_protocol_features)(TestServer *s,
CharBackend *chr, VhostUserMsg *msg);
+void (*get_config)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
};
static const char *init_hugepagefs(void);
@@ -194,6 +213,14 @@ static void append_vhost_net_opts(TestServer *s, GString
*cmd_line,
chr_opts, s->chr_name);
}
+static void append_vhost_blk_opts(TestServer *s, GString *cmd_line,
+ const char *chr_opts)
+{
+g_string_append_printf(cmd_line, QEMU_CMD_BLKCHR,
+ s->socket_path,
+ chr_opts);
+}
+
static void append_mem_opts(TestServer *server, GString *cmd_line,
int size, enum test_memfd memfd)
{
@@ -425,6 +452,12 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
+case VHOST_USER_GET_CONFIG:
+if (s->vu_ops->get_config) {
+s->vu_ops->get_config(s, chr, );
+}
+break;
+
default:
break;
}
@@ -727,6 +760,9 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
guint8 *log;
guint64 size;
+if (s->vu_ops->driver_init) {
+s->vu_ops->driver_init(obj, alloc);
+}
if (!wait_for_fds(s)) {
return;
}
@@ -796,6 +832,24 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
g_string_free(dest_cmdline, true);
}
+static void vu_blk_driver_init(void *obj, QGuestAllocator *alloc)
+{
+QVirtioBlk *blk_if;
+QVirtioDevice *dev;
+QVirtQueue *vq;
+uint64_t features;
+
+blk_if = obj;
+dev = blk_if->vdev;
+features = qvirtio_get_features(dev);
+qvirtio_set_features(dev, features);
+
+vq = qvirtqueue_setup(dev, alloc, 0);
+g_assert(vq);
+
+qvirtio_set_driver_ok(dev);
+}
+
static void wait_for_rings_started(Test
[PATCH v5 1/7] vhost: recheck dev state in the vhost_migration_log routine
vhost-user devices can get a disconnect in the middle of the VHOST-USER
handshake on the migration start. If disconnect event happened right
before sending next VHOST-USER command, then the vhost_dev_set_log()
call in the vhost_migration_log() function will return error. This error
will lead to the assert() and close the QEMU migration source process.
For the vhost-user devices the disconnect event should not break the
migration process, because:
- the device will be in the stopped state, so it will not be changed
during migration
- if reconnect will be made the migration log will be reinitialized as
part of reconnect/init process:
#0 vhost_log_global_start (listener=0x563989cf7be0)
at hw/virtio/vhost.c:920
#1 0x56398603d8bc in listener_add_address_space
(listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2664
#2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2740
#3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=0)
at hw/virtio/vhost.c:1385
#4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
at hw/block/vhost-user-blk.c:315
#5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
event=CHR_EVENT_OPENED)
at hw/block/vhost-user-blk.c:379
Update the vhost-user-blk device with the internal started_vu field which
will be used for initialization (vhost_user_blk_start) and clean up
(vhost_user_blk_stop). This additional flag in the VhostUserBlk structure
will be used to track whether the device really needs to be stopped and
cleaned up on a vhost-user level.
The disconnect event will set the overall VHOST device (not vhost-user) to
the stopped state, so it can be used by the general vhost_migration_log
routine.
Such approach could be propogated to the other vhost-user devices, but
better idea is just to make the same connect/disconnect code for all the
vhost-user devices.
This migration issue was slightly discussed earlier:
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
hw/block/vhost-user-blk.c | 19 ---
hw/virtio/vhost.c | 27 ---
include/hw/virtio/vhost-user-blk.h | 10 ++
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 39aec42..a076b1e 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
error_report("Error starting vhost: %d", -ret);
goto err_guest_notifiers;
}
+s->started_vu = true;
/* guest_notifier_mask/pending not used yet, so just unmask
* everything here. virtio-pci will do the right thing by
@@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!s->started_vu) {
+return;
+}
+s->started_vu = false;
+
if (!k->set_guest_notifiers) {
return;
}
@@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
@@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
NULL, NULL, false);
aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
}
+
+/*
+ * Move vhost device to the stopped state. The vhost-user device
+ * will be clean up and disconnected in BH. This can be useful in
+ * the vhost migration code. If disconnect was caught there is an
+ * option for the general vhost code to get the dev state without
+ * knowing its type (in this case vhost-user).
+ */
+s->dev.started = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1a1384e..ffef7ab 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -861,21 +861,42 @@ static int vhost_migration_log(MemoryListener *listener,
bool enable)
dev->log_enabled = enable;
return 0;
}
+
+r = 0;
if (!enable) {
r = vhost_dev_set_log(dev, false);
if (r < 0) {
-return r;
+goto check_dev_state;
}
vhost_log_put(dev, false);
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
[PATCH v5 2/7] vhost: check queue state in the vhost_dev_set_log routine
If the vhost-user-blk daemon provides only one virtqueue, but device was
added with several queues, then QEMU will send more VHOST-USER command
than expected by daemon side. The vhost_virtqueue_start() routine
handles such case by checking the return value from the
virtio_queue_get_desc_addr() function call. Add the same check to the
vhost_dev_set_log() routine.
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
hw/virtio/vhost.c | 12
1 file changed, 12 insertions(+)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ffef7ab..a08b7d8 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+hwaddr addr;
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
}
for (i = 0; i < dev->nvqs; ++i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
+addr = virtio_queue_get_desc_addr(dev->vdev, idx);
+if (!addr) {
+/*
+ * The queue might not be ready for start. If this
+ * is the case there is no reason to continue the process.
+ * The similar logic is used by the vhost_virtqueue_start()
+ * routine.
+ */
+continue;
+}
r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
enable_log);
if (r < 0) {
--
2.7.4
Re: [PATCH v4 1/7] vhost: recheck dev state in the vhost_migration_log routine
On Tue, Sep 08, 2020 at 03:25:20PM +0100, Stefan Hajnoczi wrote:
> On Fri, Sep 04, 2020 at 12:31:13PM +0300, Dima Stepanov wrote:
> > vhost-user devices can get a disconnect in the middle of the VHOST-USER
> > handshake on the migration start. If disconnect event happened right
> > before sending next VHOST-USER command, then the vhost_dev_set_log()
> > call in the vhost_migration_log() function will return error. This error
> > will lead to the assert() and close the QEMU migration source process.
> > For the vhost-user devices the disconnect event should not break the
> > migration process, because:
> > - the device will be in the stopped state, so it will not be changed
> > during migration
> > - if reconnect will be made the migration log will be reinitialized as
> > part of reconnect/init process:
> > #0 vhost_log_global_start (listener=0x563989cf7be0)
> > at hw/virtio/vhost.c:920
> > #1 0x56398603d8bc in listener_add_address_space
> > (listener=0x563989cf7be0,
> > as=0x563986ea4340 )
> > at softmmu/memory.c:2664
> > #2 0x56398603dd30 in memory_listener_register
> > (listener=0x563989cf7be0,
> > as=0x563986ea4340 )
> > at softmmu/memory.c:2740
> > #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
> > opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
> > busyloop_timeout=0)
> > at hw/virtio/vhost.c:1385
> > #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
> > at hw/block/vhost-user-blk.c:315
> > #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
> > event=CHR_EVENT_OPENED)
> > at hw/block/vhost-user-blk.c:379
> > Update the vhost-user-blk device with the internal started_vu field which
> > will be used for initialization (vhost_user_blk_start) and clean up
> > (vhost_user_blk_stop). This additional flag in the VhostUserBlk structure
> > will be used to track whether the device really needs to be stopped and
> > cleaned up on a vhost-user level.
> > The disconnect event will set the overall VHOST device (not vhost-user) to
> > the stopped state, so it can be used by the general vhost_migration_log
> > routine.
> > Such approach could be propogated to the other vhost-user devices, but
> > better idea is just to make the same connect/disconnect code for all the
> > vhost-user devices.
> >
> > This migration issue was slightly discussed earlier:
> > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
> > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
> >
> > Signed-off-by: Dima Stepanov
> > Reviewed-by: Raphael Norwitz
> > ---
> > hw/block/vhost-user-blk.c | 19 ---
> > hw/virtio/vhost.c | 27 ---
> > include/hw/virtio/vhost-user-blk.h | 10 ++
> > 3 files changed, 50 insertions(+), 6 deletions(-)
> >
> > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > index 39aec42..a076b1e 100644
> > --- a/hw/block/vhost-user-blk.c
> > +++ b/hw/block/vhost-user-blk.c
> > @@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
> > error_report("Error starting vhost: %d", -ret);
> > goto err_guest_notifiers;
> > }
> > +s->started_vu = true;
> >
> > /* guest_notifier_mask/pending not used yet, so just unmask
> > * everything here. virtio-pci will do the right thing by
> > @@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
> > VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
> > int ret;
> >
> > +if (!s->started_vu) {
> > +return;
> > +}
> > +s->started_vu = false;
> > +
> > if (!k->set_guest_notifiers) {
> > return;
> > }
> > @@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > }
> > s->connected = false;
> >
> > -if (s->dev.started) {
> > -vhost_user_blk_stop(vdev);
> > -}
> > +vhost_user_blk_stop(vdev);
> >
> > vhost_dev_cleanup(>dev);
> > }
> > @@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
> > QEMUChrEvent event)
> > NULL, NULL, false);
> > aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh,
> > opaque);
> > }
> > +
> > +/
Re: [PATCH v4 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
On Tue, Sep 08, 2020 at 11:03:56PM -0400, Raphael Norwitz wrote:
> On Fri, Sep 4, 2020 at 5:35 AM Dima Stepanov wrote:
> >
> > Add vhost_user_ops structure for the vhost-user-blk device class. Add
> > the test_reconnect and test_migrate tests for this device.
> >
> > Signed-off-by: Dima Stepanov
>
> Reviewed-by: Raphael Norwitz
>
> Just one small suggestion.
>
> > ---
> > tests/qtest/vhost-user-test.c | 139
> > +-
> > 1 file changed, 137 insertions(+), 2 deletions(-)
>
> > @@ -857,12 +911,21 @@ static void test_reconnect(void *obj, void *arg,
> > QGuestAllocator *alloc)
> > {
> > TestServer *s = arg;
> > GSource *src;
> > +int nq;
> >
> > +if (s->vu_ops->driver_init) {
> > +s->vu_ops->driver_init(obj, alloc);
> > +}
> > if (!wait_for_fds(s)) {
> > return;
> > }
> >
>
> Maybe we could break this logic out into a helper? I imagine there may
> be other cases where we might want to get a number of rings for a
> given device type.
Yes, i've also thought about it and my point is that it isn't clear
right now how it will be used. So i decided to use it in the pretty
straigtforward way. As soon as we add some more vhost-user devices for
testing we can update this code properly.
>
>
> > -wait_for_rings_started(s, 2);
> > +nq = 1;
> > +if (s->vu_ops->type == VHOST_USER_NET) {
> > +/* tx and rx queues */
> > +nq = 2;
> > +}
> > +wait_for_rings_started(s, nq);
> >
Re: [PATCH v4 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
On Tue, Sep 08, 2020 at 11:01:57PM -0400, Raphael Norwitz wrote:
> On Fri, Sep 4, 2020 at 5:34 AM Dima Stepanov wrote:
> >
> > Add support for the vhost-user-blk-pci device. This node can be used by
> > the vhost-user-blk tests. Tests for the vhost-user-blk device are added
> > in the following patches.
> >
> > Signed-off-by: Dima Stepanov
> > ---
> > tests/qtest/libqos/virtio-blk.c | 14 ++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/tests/qtest/libqos/virtio-blk.c
> > b/tests/qtest/libqos/virtio-blk.c
> > index 5da0259..959c5dc 100644
> > --- a/tests/qtest/libqos/virtio-blk.c
> > +++ b/tests/qtest/libqos/virtio-blk.c
> > @@ -36,6 +36,9 @@ static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
> > if (!g_strcmp0(interface, "virtio")) {
> > return v_blk->vdev;
> > }
> > +if (!g_strcmp0(interface, "vhost-user-blk")) {
>
> Small point but why not merge this conditional with the
> !g_strcmp0(interface, "virtio-blk") check above? They both return
> v_blk.
Good point, will made an update. Thanks!
>
> Otherwise looks good.
>
> > +return v_blk;
> > +}
> >
> > fprintf(stderr, "%s not present in virtio-blk-device\n", interface);
> > g_assert_not_reached();
> > @@ -120,6 +123,17 @@ static void virtio_blk_register_nodes(void)
> > qos_node_produces("virtio-blk-pci", "virtio-blk");
> >
> > g_free(arg);
> > +
> > +/* vhost-user-blk-pci */
> > +arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
> > +PCI_SLOT, PCI_FN);
> > +opts.extra_device_opts = arg;
> > +add_qpci_address(, );
> > +qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
> > +qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
> > +qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
> > +
> > +g_free(arg);
> > }
> >
> > libqos_init(virtio_blk_register_nodes);
> > --
> > 2.7.4
> >
> >
[PATCH v4 6/7] tests/qtest/vhost-user-test: add migrate_reconnect test
Add new migrate_reconnect test for the vhost-user-blk device. Perform a
disconnect after sending response for the VHOST_USER_SET_LOG_BASE
command.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +
1 file changed, 25 insertions(+)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index a8af613..4b715d3 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -146,6 +146,7 @@ static VhostUserMsg m __attribute__ ((unused));
enum {
TEST_FLAGS_OK,
TEST_FLAGS_DISCONNECT,
+TEST_FLAGS_MIGRATE_DISCONNECT,
TEST_FLAGS_BAD,
TEST_FLAGS_END,
};
@@ -436,6 +437,15 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE);
g_cond_broadcast(>data_cond);
+/*
+ * Perform disconnect after sending a response. In this
+ * case the next write command on the QEMU side (for now
+ * it is SET_FEATURES will return -1, because of disconnect.
+ */
+if (s->test_flags == TEST_FLAGS_MIGRATE_DISCONNECT) {
+qemu_chr_fe_disconnect(chr);
+s->test_flags = TEST_FLAGS_BAD;
+}
break;
case VHOST_USER_SET_VRING_BASE:
@@ -737,6 +747,17 @@ static void *vhost_user_test_setup_memfd(GString
*cmd_line, void *arg)
return server;
}
+static void *vhost_user_test_setup_migrate_reconnect(GString *cmd_line,
+void *arg)
+{
+TestServer *server;
+
+server = vhost_user_test_setup_memfd(cmd_line, arg);
+server->test_flags = TEST_FLAGS_MIGRATE_DISCONNECT;
+
+return server;
+}
+
static void test_read_guest_mem(void *obj, void *arg, QGuestAllocator *alloc)
{
TestServer *server = arg;
@@ -1150,5 +1171,9 @@ static void register_vhost_user_test(void)
opts.before = vhost_user_test_setup_memfd;
qos_add_test("migrate", "vhost-user-blk",
test_migrate, );
+
+opts.before = vhost_user_test_setup_migrate_reconnect;
+qos_add_test("migrate_reconnect", "vhost-user-blk",
+ test_migrate, );
}
libqos_init(register_vhost_user_test);
--
2.7.4
[PATCH v4 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
Add support for the vhost-user-blk-pci device. This node can be used by
the vhost-user-blk tests. Tests for the vhost-user-blk device are added
in the following patches.
Signed-off-by: Dima Stepanov
---
tests/qtest/libqos/virtio-blk.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/tests/qtest/libqos/virtio-blk.c b/tests/qtest/libqos/virtio-blk.c
index 5da0259..959c5dc 100644
--- a/tests/qtest/libqos/virtio-blk.c
+++ b/tests/qtest/libqos/virtio-blk.c
@@ -36,6 +36,9 @@ static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
if (!g_strcmp0(interface, "virtio")) {
return v_blk->vdev;
}
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
fprintf(stderr, "%s not present in virtio-blk-device\n", interface);
g_assert_not_reached();
@@ -120,6 +123,17 @@ static void virtio_blk_register_nodes(void)
qos_node_produces("virtio-blk-pci", "virtio-blk");
g_free(arg);
+
+/* vhost-user-blk-pci */
+arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
+
+g_free(arg);
}
libqos_init(virtio_blk_register_nodes);
--
2.7.4
[PATCH v4 0/7] vhost-user-blk: fix the migration issue and enhance qtests
v3 -> v4: - vhost: recheck dev state in the vhost_migration_log routine Reviewed-by: Raphael Norwitz - vhost: check queue state in the vhost_dev_set_log routine Use "continue" instead of "break" to handle non-initialized virtqueue case. v2 -> v3: - update commit message for the "vhost: recheck dev state in the vhost_migration_log routine" commit - rename "started" field of the VhostUserBlk structure to "started_vu", so there will be no confustion with the VHOST started field - update vhost-user-test.c to always initialize nq local variable (spotted by patchew) v1 -> v2: - add comments to connected/started fields in the header file - move the "s->started" logic from the vhost_user_blk_disconnect routine to the vhost_user_blk_stop routine Reference e-mail threads: - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html If vhost-user daemon is used as a backend for the vhost device, then we should consider a possibility of disconnect at any moment. There was a general question here: should we consider it as an error or okay state for the vhost-user devices during migration process? I think the disconnect event for the vhost-user devices should not break the migration process, because: - the device will be in the stopped state, so it will not be changed during migration - if reconnect will be made the migration log will be reinitialized as part of reconnect/init process: #0 vhost_log_global_start (listener=0x563989cf7be0) at hw/virtio/vhost.c:920 #1 0x56398603d8bc in listener_add_address_space (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2664 #2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2740 #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, busyloop_timeout=0) at hw/virtio/vhost.c:1385 #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) at hw/block/vhost-user-blk.c:315 #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, event=CHR_EVENT_OPENED) at hw/block/vhost-user-blk.c:379 The first patch in the patchset fixes this issue by setting vhost device to the stopped state in the disconnect handler and check it the vhost_migration_log() routine before returning from the function. qtest framework was updated to test vhost-user-blk functionality. The vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to reproduce the original issue found. Dima Stepanov (7): vhost: recheck dev state in the vhost_migration_log routine vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/libqos/virtio-blk: add support for vhost-user-blk tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test tests/qtest/vhost-user-test: enable the reconnect tests hw/block/vhost-user-blk.c | 19 ++- hw/virtio/vhost.c | 39 - include/hw/virtio/vhost-user-blk.h | 10 ++ tests/qtest/libqos/virtio-blk.c| 14 ++ tests/qtest/vhost-user-test.c | 290 +++-- 5 files changed, 323 insertions(+), 49 deletions(-) -- 2.7.4
[PATCH v4 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
Add vhost_user_ops structure for the vhost-user-blk device class. Add
the test_reconnect and test_migrate tests for this device.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 139 +-
1 file changed, 137 insertions(+), 2 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 3df5322..a8af613 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -24,6 +24,7 @@
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
#include "libqos/virtio-pci.h"
+#include "libqos/virtio-blk.h"
#include "libqos/malloc-pc.h"
#include "hw/virtio/virtio-net.h"
@@ -31,6 +32,7 @@
#include "standard-headers/linux/vhost_types.h"
#include "standard-headers/linux/virtio_ids.h"
#include "standard-headers/linux/virtio_net.h"
+#include "standard-headers/linux/virtio_blk.h"
#ifdef CONFIG_LINUX
#include
@@ -43,6 +45,7 @@
" -numa node,memdev=mem"
#define QEMU_CMD_CHR" -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=hs0,chardev=%s,vhostforce"
+#define QEMU_CMD_BLKCHR " -chardev socket,id=chdev0,path=%s%s"
#define HUGETLBFS_MAGIC 0x958458f6
@@ -55,6 +58,7 @@
#define VHOST_USER_PROTOCOL_F_MQ 0
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+#define VHOST_USER_PROTOCOL_F_CONFIG 9
#define VHOST_LOG_PAGE 0x1000
@@ -78,6 +82,8 @@ typedef enum VhostUserRequest {
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
+VHOST_USER_GET_CONFIG = 24,
+VHOST_USER_SET_CONFIG = 25,
VHOST_USER_MAX
} VhostUserRequest;
@@ -99,6 +105,14 @@ typedef struct VhostUserLog {
uint64_t mmap_offset;
} VhostUserLog;
+#define VHOST_USER_MAX_CONFIG_SIZE 256
+typedef struct VhostUserConfig {
+uint32_t offset;
+uint32_t size;
+uint32_t flags;
+uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+} VhostUserConfig;
+
typedef struct VhostUserMsg {
VhostUserRequest request;
@@ -114,6 +128,7 @@ typedef struct VhostUserMsg {
struct vhost_vring_addr addr;
VhostUserMemory memory;
VhostUserLog log;
+VhostUserConfig config;
} payload;
} QEMU_PACKED VhostUserMsg;
@@ -137,6 +152,7 @@ enum {
enum {
VHOST_USER_NET,
+VHOST_USER_BLK,
};
typedef struct TestServer {
@@ -166,12 +182,15 @@ struct vhost_user_ops {
int type;
void (*append_opts)(TestServer *s, GString *cmd_line,
const char *chr_opts);
+void (*driver_init)(void *obj, QGuestAllocator *alloc);
/* VHOST-USER commands. */
void (*set_features)(TestServer *s, CharBackend *chr,
VhostUserMsg *msg);
void (*get_protocol_features)(TestServer *s,
CharBackend *chr, VhostUserMsg *msg);
+void (*get_config)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
};
static const char *init_hugepagefs(void);
@@ -194,6 +213,14 @@ static void append_vhost_net_opts(TestServer *s, GString
*cmd_line,
chr_opts, s->chr_name);
}
+static void append_vhost_blk_opts(TestServer *s, GString *cmd_line,
+ const char *chr_opts)
+{
+g_string_append_printf(cmd_line, QEMU_CMD_BLKCHR,
+ s->socket_path,
+ chr_opts);
+}
+
static void append_mem_opts(TestServer *server, GString *cmd_line,
int size, enum test_memfd memfd)
{
@@ -425,6 +452,12 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
+case VHOST_USER_GET_CONFIG:
+if (s->vu_ops->get_config) {
+s->vu_ops->get_config(s, chr, );
+}
+break;
+
default:
break;
}
@@ -727,6 +760,9 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
guint8 *log;
guint64 size;
+if (s->vu_ops->driver_init) {
+s->vu_ops->driver_init(obj, alloc);
+}
if (!wait_for_fds(s)) {
return;
}
@@ -796,6 +832,24 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
g_string_free(dest_cmdline, true);
}
+static void vu_blk_driver_init(void *obj, QGuestAllocator *alloc)
+{
+QVirtioBlk *blk_if;
+QVirtioDevice *dev;
+QVirtQueue *vq;
+uint64_t features;
+
+blk_if = obj;
+dev = blk_if->vdev;
+features = qvirtio_get_features(dev);
+qvirtio_set_features(dev, features);
+
+vq = qvirtqueue_setup(dev, alloc, 0);
+g_assert(vq);
+
+qvirtio_set_driver_ok(dev);
+}
+
static void wait_for_rings_started(TestServer *s, size_t count)
{
[PATCH v4 1/7] vhost: recheck dev state in the vhost_migration_log routine
vhost-user devices can get a disconnect in the middle of the VHOST-USER
handshake on the migration start. If disconnect event happened right
before sending next VHOST-USER command, then the vhost_dev_set_log()
call in the vhost_migration_log() function will return error. This error
will lead to the assert() and close the QEMU migration source process.
For the vhost-user devices the disconnect event should not break the
migration process, because:
- the device will be in the stopped state, so it will not be changed
during migration
- if reconnect will be made the migration log will be reinitialized as
part of reconnect/init process:
#0 vhost_log_global_start (listener=0x563989cf7be0)
at hw/virtio/vhost.c:920
#1 0x56398603d8bc in listener_add_address_space
(listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2664
#2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2740
#3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=0)
at hw/virtio/vhost.c:1385
#4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
at hw/block/vhost-user-blk.c:315
#5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
event=CHR_EVENT_OPENED)
at hw/block/vhost-user-blk.c:379
Update the vhost-user-blk device with the internal started_vu field which
will be used for initialization (vhost_user_blk_start) and clean up
(vhost_user_blk_stop). This additional flag in the VhostUserBlk structure
will be used to track whether the device really needs to be stopped and
cleaned up on a vhost-user level.
The disconnect event will set the overall VHOST device (not vhost-user) to
the stopped state, so it can be used by the general vhost_migration_log
routine.
Such approach could be propogated to the other vhost-user devices, but
better idea is just to make the same connect/disconnect code for all the
vhost-user devices.
This migration issue was slightly discussed earlier:
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
Signed-off-by: Dima Stepanov
Reviewed-by: Raphael Norwitz
---
hw/block/vhost-user-blk.c | 19 ---
hw/virtio/vhost.c | 27 ---
include/hw/virtio/vhost-user-blk.h | 10 ++
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 39aec42..a076b1e 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
error_report("Error starting vhost: %d", -ret);
goto err_guest_notifiers;
}
+s->started_vu = true;
/* guest_notifier_mask/pending not used yet, so just unmask
* everything here. virtio-pci will do the right thing by
@@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!s->started_vu) {
+return;
+}
+s->started_vu = false;
+
if (!k->set_guest_notifiers) {
return;
}
@@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
@@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
NULL, NULL, false);
aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
}
+
+/*
+ * Move vhost device to the stopped state. The vhost-user device
+ * will be clean up and disconnected in BH. This can be useful in
+ * the vhost migration code. If disconnect was caught there is an
+ * option for the general vhost code to get the dev state without
+ * knowing its type (in this case vhost-user).
+ */
+s->dev.started = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1a1384e..ffef7ab 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -861,21 +861,42 @@ static int vhost_migration_log(MemoryListener *listener,
bool enable)
dev->log_enabled = enable;
return 0;
}
+
+r = 0;
if (!enable) {
r = vhost_dev_set_log(dev, false);
if (r < 0) {
-return r;
+goto check_dev_state;
}
vhost_log_put(dev, false);
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
[PATCH v4 7/7] tests/qtest/vhost-user-test: enable the reconnect tests
For now a QTEST_VHOST_USER_FIXME environment variable is used to
separate reconnect tests for the vhost-user-net device. Looks like the
reconnect functionality is pretty stable, so this separation is
deprecated.
Remove it and enable these tests for the default run.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +++--
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 4b715d3..4b96312 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -1140,20 +1140,17 @@ static void register_vhost_user_test(void)
"virtio-net",
test_migrate, );
-/* keeps failing on build-system since Aug 15 2017 */
-if (getenv("QTEST_VHOST_USER_FIXME")) {
-opts.before = vhost_user_test_setup_reconnect;
-qos_add_test("vhost-user/reconnect", "virtio-net",
- test_reconnect, );
-
-opts.before = vhost_user_test_setup_connect_fail;
-qos_add_test("vhost-user/connect-fail", "virtio-net",
- test_vhost_user_started, );
-
-opts.before = vhost_user_test_setup_flags_mismatch;
-qos_add_test("vhost-user/flags-mismatch", "virtio-net",
- test_vhost_user_started, );
-}
+opts.before = vhost_user_test_setup_reconnect;
+qos_add_test("vhost-user/reconnect", "virtio-net",
+ test_reconnect, );
+
+opts.before = vhost_user_test_setup_connect_fail;
+qos_add_test("vhost-user/connect-fail", "virtio-net",
+ test_vhost_user_started, );
+
+opts.before = vhost_user_test_setup_flags_mismatch;
+qos_add_test("vhost-user/flags-mismatch", "virtio-net",
+ test_vhost_user_started, );
opts.before = vhost_user_test_setup_multiqueue;
opts.edge.extra_device_opts = "mq=on";
--
2.7.4
[PATCH v4 2/7] vhost: check queue state in the vhost_dev_set_log routine
If the vhost-user-blk daemon provides only one virtqueue, but device was
added with several queues, then QEMU will send more VHOST-USER command
than expected by daemon side. The vhost_virtqueue_start() routine
handles such case by checking the return value from the
virtio_queue_get_desc_addr() function call. Add the same check to the
vhost_dev_set_log() routine.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 12
1 file changed, 12 insertions(+)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ffef7ab..a08b7d8 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+hwaddr addr;
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
}
for (i = 0; i < dev->nvqs; ++i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
+addr = virtio_queue_get_desc_addr(dev->vdev, idx);
+if (!addr) {
+/*
+ * The queue might not be ready for start. If this
+ * is the case there is no reason to continue the process.
+ * The similar logic is used by the vhost_virtqueue_start()
+ * routine.
+ */
+continue;
+}
r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
enable_log);
if (r < 0) {
--
2.7.4
[PATCH v4 3/7] tests/qtest/vhost-user-test: prepare the tests for adding new dev class
For now only vhost-user-net device is supported by the test. Other
vhost-user devices are not tested. As a first step make source code
refactoring so new devices can reuse the same test routines. To make
this provide a new vhost_user_ops structure with the methods to
initialize device, its command line or make a proper vhost-user
responses.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 105 ++
1 file changed, 76 insertions(+), 29 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9ee0f1e..3df5322 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -135,6 +135,10 @@ enum {
TEST_FLAGS_END,
};
+enum {
+VHOST_USER_NET,
+};
+
typedef struct TestServer {
gchar *socket_path;
gchar *mig_path;
@@ -154,10 +158,25 @@ typedef struct TestServer {
bool test_fail;
int test_flags;
int queues;
+struct vhost_user_ops *vu_ops;
} TestServer;
+struct vhost_user_ops {
+/* Device types. */
+int type;
+void (*append_opts)(TestServer *s, GString *cmd_line,
+const char *chr_opts);
+
+/* VHOST-USER commands. */
+void (*set_features)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
+void (*get_protocol_features)(TestServer *s,
+CharBackend *chr, VhostUserMsg *msg);
+};
+
static const char *init_hugepagefs(void);
-static TestServer *test_server_new(const gchar *name);
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops);
static void test_server_free(TestServer *server);
static void test_server_listen(TestServer *server);
@@ -167,7 +186,7 @@ enum test_memfd {
TEST_MEMFD_NO,
};
-static void append_vhost_opts(TestServer *s, GString *cmd_line,
+static void append_vhost_net_opts(TestServer *s, GString *cmd_line,
const char *chr_opts)
{
g_string_append_printf(cmd_line, QEMU_CMD_CHR QEMU_CMD_NETDEV,
@@ -332,25 +351,15 @@ static void chr_read(void *opaque, const uint8_t *buf,
int size)
break;
case VHOST_USER_SET_FEATURES:
-g_assert_cmpint(msg.payload.u64 & (0x1ULL <<
VHOST_USER_F_PROTOCOL_FEATURES),
-!=, 0ULL);
-if (s->test_flags == TEST_FLAGS_DISCONNECT) {
-qemu_chr_fe_disconnect(chr);
-s->test_flags = TEST_FLAGS_BAD;
+if (s->vu_ops->set_features) {
+s->vu_ops->set_features(s, chr, );
}
break;
case VHOST_USER_GET_PROTOCOL_FEATURES:
-/* send back features to qemu */
-msg.flags |= VHOST_USER_REPLY_MASK;
-msg.size = sizeof(m.payload.u64);
-msg.payload.u64 = 1 << VHOST_USER_PROTOCOL_F_LOG_SHMFD;
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_CROSS_ENDIAN;
-if (s->queues > 1) {
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_MQ;
+if (s->vu_ops->get_protocol_features) {
+s->vu_ops->get_protocol_features(s, chr, );
}
-p = (uint8_t *)
-qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
case VHOST_USER_GET_VRING_BASE:
@@ -467,7 +476,8 @@ static const char *init_hugepagefs(void)
#endif
}
-static TestServer *test_server_new(const gchar *name)
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops)
{
TestServer *server = g_new0(TestServer, 1);
char template[] = "/tmp/vhost-test-XX";
@@ -495,6 +505,7 @@ static TestServer *test_server_new(const gchar *name)
server->log_fd = -1;
server->queues = 1;
+server->vu_ops = ops;
return server;
}
@@ -669,11 +680,11 @@ static void vhost_user_test_cleanup(void *s)
static void *vhost_user_test_setup(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_AUTO);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -682,11 +693,11 @@ static void *vhost_user_test_setup(GString *cmd_line,
void *arg)
static void *vhost_user_test_setup_memfd(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_YES);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -720,7 +73
Re: [PATCH v2 2/7] vhost: check queue state in the vhost_dev_set_log routine
On Mon, Aug 31, 2020 at 11:22:14PM -0400, Raphael Norwitz wrote:
> On Mon, Aug 31, 2020 at 4:37 AM Dima Stepanov wrote:
> >
> > On Thu, Aug 27, 2020 at 09:46:03PM -0400, Raphael Norwitz wrote:
> > > On Mon, Aug 24, 2020 at 4:41 AM Dima Stepanov
> > > wrote:
> > > >
> > > > If the vhost-user-blk daemon provides only one virtqueue, but device was
> > > > added with several queues, then QEMU will send more VHOST-USER command
> > > > than expected by daemon side. The vhost_virtqueue_start() routine
> > > > handles such case by checking the return value from the
> > > > virtio_queue_get_desc_addr() function call. Add the same check to the
> > > > vhost_dev_set_log() routine.
> > > >
> > > > Signed-off-by: Dima Stepanov
> > > > ---
> > > > hw/virtio/vhost.c | 12
> > > > 1 file changed, 12 insertions(+)
> > > >
> > > > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > > > index ffef7ab..a33ffd4 100644
> > > > --- a/hw/virtio/vhost.c
> > > > +++ b/hw/virtio/vhost.c
> > > > @@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct
> > > > vhost_dev *dev,
> > > > static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> > > > {
> > > > int r, i, idx;
> > > > +hwaddr addr;
> > > > +
> > > > r = vhost_dev_set_features(dev, enable_log);
> > > > if (r < 0) {
> > > > goto err_features;
> > > > }
> > > > for (i = 0; i < dev->nvqs; ++i) {
> > > > idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index +
> > > > i);
> > > > +addr = virtio_queue_get_desc_addr(dev->vdev, idx);
> > > > +if (!addr) {
> > > > +/*
> > > > + * The queue might not be ready for start. If this
> > > > + * is the case there is no reason to continue the process.
> > > > + * The similar logic is used by the vhost_virtqueue_start()
> > > > + * routine.
> > > > + */
> > >
> > > Shouldn’t we goto err_vq here to reset the logging state of any vqs
> > > which have already been set?
> > As i understand it, no we shouldn't reset the state of other queues. In
> > general it is pretty valid case. Let's assume that the backend
> > vhost-user device supports only two queues. But for instance, the QEMU
> > command line is using value 4 to define number of virtqueues of such
> > device. In this case only 2 queues will be initializaed.
>
> I see - makes more sense now.
>
> >
> > I've tried to reflect it in the comment section, that the
> > vhost_virtqueue_start() routine has been alread made the same:
> > "if a queue isn't ready for start, just return 0 without any error"
> > So i made the same here.
> >
>
> In your example is a reason why, if queue 3 is uninitialized, queue 4
> must also be uninitialized? I realize queue 4 being initialized while
> queue 3 is not is a strange case, but it may still make the code more
> robust to use a "continue" here instead of a "break". This also seems
> more like the logic in vhost_virtqueue_start()/vhost_dev_start().
Good point! Should really use "continue" instead of a "break" to keep
the logic. Will update it in v4. Hope to get some review feedback for
the qtest framework update aswell ).
>
> > I've found this issue, while testing migration with the default
> > vhost-user-blk daemon. It fails with assert or sigsegv (don't remember),
> > because it receives NULL for the queues it doesn't have. In general
> > the daemon should not fall, because of unexpected VHOST_USER
> > communication, but also there is no reason for QEMU to send additional
> > packets.
> >
> > >
> > > > +break;
> > > > +}
> > > > r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
> > > > enable_log);
> > > > if (r < 0) {
> > > > --
> > > > 2.7.4
> > > >
> > > >
[PATCH v3 6/7] tests/qtest/vhost-user-test: add migrate_reconnect test
Add new migrate_reconnect test for the vhost-user-blk device. Perform a
disconnect after sending response for the VHOST_USER_SET_LOG_BASE
command.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +
1 file changed, 25 insertions(+)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index a8af613..4b715d3 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -146,6 +146,7 @@ static VhostUserMsg m __attribute__ ((unused));
enum {
TEST_FLAGS_OK,
TEST_FLAGS_DISCONNECT,
+TEST_FLAGS_MIGRATE_DISCONNECT,
TEST_FLAGS_BAD,
TEST_FLAGS_END,
};
@@ -436,6 +437,15 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE);
g_cond_broadcast(>data_cond);
+/*
+ * Perform disconnect after sending a response. In this
+ * case the next write command on the QEMU side (for now
+ * it is SET_FEATURES will return -1, because of disconnect.
+ */
+if (s->test_flags == TEST_FLAGS_MIGRATE_DISCONNECT) {
+qemu_chr_fe_disconnect(chr);
+s->test_flags = TEST_FLAGS_BAD;
+}
break;
case VHOST_USER_SET_VRING_BASE:
@@ -737,6 +747,17 @@ static void *vhost_user_test_setup_memfd(GString
*cmd_line, void *arg)
return server;
}
+static void *vhost_user_test_setup_migrate_reconnect(GString *cmd_line,
+void *arg)
+{
+TestServer *server;
+
+server = vhost_user_test_setup_memfd(cmd_line, arg);
+server->test_flags = TEST_FLAGS_MIGRATE_DISCONNECT;
+
+return server;
+}
+
static void test_read_guest_mem(void *obj, void *arg, QGuestAllocator *alloc)
{
TestServer *server = arg;
@@ -1150,5 +1171,9 @@ static void register_vhost_user_test(void)
opts.before = vhost_user_test_setup_memfd;
qos_add_test("migrate", "vhost-user-blk",
test_migrate, );
+
+opts.before = vhost_user_test_setup_migrate_reconnect;
+qos_add_test("migrate_reconnect", "vhost-user-blk",
+ test_migrate, );
}
libqos_init(register_vhost_user_test);
--
2.7.4
[PATCH v3 2/7] vhost: check queue state in the vhost_dev_set_log routine
If the vhost-user-blk daemon provides only one virtqueue, but device was
added with several queues, then QEMU will send more VHOST-USER command
than expected by daemon side. The vhost_virtqueue_start() routine
handles such case by checking the return value from the
virtio_queue_get_desc_addr() function call. Add the same check to the
vhost_dev_set_log() routine.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 12
1 file changed, 12 insertions(+)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ffef7ab..a33ffd4 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+hwaddr addr;
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
}
for (i = 0; i < dev->nvqs; ++i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
+addr = virtio_queue_get_desc_addr(dev->vdev, idx);
+if (!addr) {
+/*
+ * The queue might not be ready for start. If this
+ * is the case there is no reason to continue the process.
+ * The similar logic is used by the vhost_virtqueue_start()
+ * routine.
+ */
+break;
+}
r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
enable_log);
if (r < 0) {
--
2.7.4
[PATCH v3 3/7] tests/qtest/vhost-user-test: prepare the tests for adding new dev class
For now only vhost-user-net device is supported by the test. Other
vhost-user devices are not tested. As a first step make source code
refactoring so new devices can reuse the same test routines. To make
this provide a new vhost_user_ops structure with the methods to
initialize device, its command line or make a proper vhost-user
responses.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 105 ++
1 file changed, 76 insertions(+), 29 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9ee0f1e..3df5322 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -135,6 +135,10 @@ enum {
TEST_FLAGS_END,
};
+enum {
+VHOST_USER_NET,
+};
+
typedef struct TestServer {
gchar *socket_path;
gchar *mig_path;
@@ -154,10 +158,25 @@ typedef struct TestServer {
bool test_fail;
int test_flags;
int queues;
+struct vhost_user_ops *vu_ops;
} TestServer;
+struct vhost_user_ops {
+/* Device types. */
+int type;
+void (*append_opts)(TestServer *s, GString *cmd_line,
+const char *chr_opts);
+
+/* VHOST-USER commands. */
+void (*set_features)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
+void (*get_protocol_features)(TestServer *s,
+CharBackend *chr, VhostUserMsg *msg);
+};
+
static const char *init_hugepagefs(void);
-static TestServer *test_server_new(const gchar *name);
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops);
static void test_server_free(TestServer *server);
static void test_server_listen(TestServer *server);
@@ -167,7 +186,7 @@ enum test_memfd {
TEST_MEMFD_NO,
};
-static void append_vhost_opts(TestServer *s, GString *cmd_line,
+static void append_vhost_net_opts(TestServer *s, GString *cmd_line,
const char *chr_opts)
{
g_string_append_printf(cmd_line, QEMU_CMD_CHR QEMU_CMD_NETDEV,
@@ -332,25 +351,15 @@ static void chr_read(void *opaque, const uint8_t *buf,
int size)
break;
case VHOST_USER_SET_FEATURES:
-g_assert_cmpint(msg.payload.u64 & (0x1ULL <<
VHOST_USER_F_PROTOCOL_FEATURES),
-!=, 0ULL);
-if (s->test_flags == TEST_FLAGS_DISCONNECT) {
-qemu_chr_fe_disconnect(chr);
-s->test_flags = TEST_FLAGS_BAD;
+if (s->vu_ops->set_features) {
+s->vu_ops->set_features(s, chr, );
}
break;
case VHOST_USER_GET_PROTOCOL_FEATURES:
-/* send back features to qemu */
-msg.flags |= VHOST_USER_REPLY_MASK;
-msg.size = sizeof(m.payload.u64);
-msg.payload.u64 = 1 << VHOST_USER_PROTOCOL_F_LOG_SHMFD;
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_CROSS_ENDIAN;
-if (s->queues > 1) {
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_MQ;
+if (s->vu_ops->get_protocol_features) {
+s->vu_ops->get_protocol_features(s, chr, );
}
-p = (uint8_t *)
-qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
case VHOST_USER_GET_VRING_BASE:
@@ -467,7 +476,8 @@ static const char *init_hugepagefs(void)
#endif
}
-static TestServer *test_server_new(const gchar *name)
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops)
{
TestServer *server = g_new0(TestServer, 1);
char template[] = "/tmp/vhost-test-XX";
@@ -495,6 +505,7 @@ static TestServer *test_server_new(const gchar *name)
server->log_fd = -1;
server->queues = 1;
+server->vu_ops = ops;
return server;
}
@@ -669,11 +680,11 @@ static void vhost_user_test_cleanup(void *s)
static void *vhost_user_test_setup(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_AUTO);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -682,11 +693,11 @@ static void *vhost_user_test_setup(GString *cmd_line,
void *arg)
static void *vhost_user_test_setup_memfd(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_YES);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -720,7 +73
[PATCH v3 0/7] vhost-user-blk: fix the migration issue and enhance qtests
v2 -> v3: - update commit message for the "vhost: recheck dev state in the vhost_migration_log routine" commit - rename "started" field of the VhostUserBlk structure to "started_vu", so there will be no confustion with the VHOST started field - update vhost-user-test.c to always initialize nq local variable (spotted by patchew) v1 -> v2: - add comments to connected/started fields in the header file - move the "s->started" logic from the vhost_user_blk_disconnect routine to the vhost_user_blk_stop routine Reference e-mail threads: - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html If vhost-user daemon is used as a backend for the vhost device, then we should consider a possibility of disconnect at any moment. There was a general question here: should we consider it as an error or okay state for the vhost-user devices during migration process? I think the disconnect event for the vhost-user devices should not break the migration process, because: - the device will be in the stopped state, so it will not be changed during migration - if reconnect will be made the migration log will be reinitialized as part of reconnect/init process: #0 vhost_log_global_start (listener=0x563989cf7be0) at hw/virtio/vhost.c:920 #1 0x56398603d8bc in listener_add_address_space (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2664 #2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2740 #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, busyloop_timeout=0) at hw/virtio/vhost.c:1385 #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) at hw/block/vhost-user-blk.c:315 #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, event=CHR_EVENT_OPENED) at hw/block/vhost-user-blk.c:379 The first patch in the patchset fixes this issue by setting vhost device to the stopped state in the disconnect handler and check it the vhost_migration_log() routine before returning from the function. qtest framework was updated to test vhost-user-blk functionality. The vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to reproduce the original issue found. Dima Stepanov (7): vhost: recheck dev state in the vhost_migration_log routine vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/libqos/virtio-blk: add support for vhost-user-blk tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test tests/qtest/vhost-user-test: enable the reconnect tests hw/block/vhost-user-blk.c | 19 ++- hw/virtio/vhost.c | 39 - include/hw/virtio/vhost-user-blk.h | 10 ++ tests/qtest/libqos/virtio-blk.c| 14 ++ tests/qtest/vhost-user-test.c | 290 +++-- 5 files changed, 323 insertions(+), 49 deletions(-) -- 2.7.4
[PATCH v3 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
Add support for the vhost-user-blk-pci device. This node can be used by
the vhost-user-blk tests. Tests for the vhost-user-blk device are added
in the following patches.
Signed-off-by: Dima Stepanov
---
tests/qtest/libqos/virtio-blk.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/tests/qtest/libqos/virtio-blk.c b/tests/qtest/libqos/virtio-blk.c
index 5da0259..959c5dc 100644
--- a/tests/qtest/libqos/virtio-blk.c
+++ b/tests/qtest/libqos/virtio-blk.c
@@ -36,6 +36,9 @@ static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
if (!g_strcmp0(interface, "virtio")) {
return v_blk->vdev;
}
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
fprintf(stderr, "%s not present in virtio-blk-device\n", interface);
g_assert_not_reached();
@@ -120,6 +123,17 @@ static void virtio_blk_register_nodes(void)
qos_node_produces("virtio-blk-pci", "virtio-blk");
g_free(arg);
+
+/* vhost-user-blk-pci */
+arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
+
+g_free(arg);
}
libqos_init(virtio_blk_register_nodes);
--
2.7.4
[PATCH v3 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
Add vhost_user_ops structure for the vhost-user-blk device class. Add
the test_reconnect and test_migrate tests for this device.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 139 +-
1 file changed, 137 insertions(+), 2 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 3df5322..a8af613 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -24,6 +24,7 @@
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
#include "libqos/virtio-pci.h"
+#include "libqos/virtio-blk.h"
#include "libqos/malloc-pc.h"
#include "hw/virtio/virtio-net.h"
@@ -31,6 +32,7 @@
#include "standard-headers/linux/vhost_types.h"
#include "standard-headers/linux/virtio_ids.h"
#include "standard-headers/linux/virtio_net.h"
+#include "standard-headers/linux/virtio_blk.h"
#ifdef CONFIG_LINUX
#include
@@ -43,6 +45,7 @@
" -numa node,memdev=mem"
#define QEMU_CMD_CHR" -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=hs0,chardev=%s,vhostforce"
+#define QEMU_CMD_BLKCHR " -chardev socket,id=chdev0,path=%s%s"
#define HUGETLBFS_MAGIC 0x958458f6
@@ -55,6 +58,7 @@
#define VHOST_USER_PROTOCOL_F_MQ 0
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+#define VHOST_USER_PROTOCOL_F_CONFIG 9
#define VHOST_LOG_PAGE 0x1000
@@ -78,6 +82,8 @@ typedef enum VhostUserRequest {
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
+VHOST_USER_GET_CONFIG = 24,
+VHOST_USER_SET_CONFIG = 25,
VHOST_USER_MAX
} VhostUserRequest;
@@ -99,6 +105,14 @@ typedef struct VhostUserLog {
uint64_t mmap_offset;
} VhostUserLog;
+#define VHOST_USER_MAX_CONFIG_SIZE 256
+typedef struct VhostUserConfig {
+uint32_t offset;
+uint32_t size;
+uint32_t flags;
+uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+} VhostUserConfig;
+
typedef struct VhostUserMsg {
VhostUserRequest request;
@@ -114,6 +128,7 @@ typedef struct VhostUserMsg {
struct vhost_vring_addr addr;
VhostUserMemory memory;
VhostUserLog log;
+VhostUserConfig config;
} payload;
} QEMU_PACKED VhostUserMsg;
@@ -137,6 +152,7 @@ enum {
enum {
VHOST_USER_NET,
+VHOST_USER_BLK,
};
typedef struct TestServer {
@@ -166,12 +182,15 @@ struct vhost_user_ops {
int type;
void (*append_opts)(TestServer *s, GString *cmd_line,
const char *chr_opts);
+void (*driver_init)(void *obj, QGuestAllocator *alloc);
/* VHOST-USER commands. */
void (*set_features)(TestServer *s, CharBackend *chr,
VhostUserMsg *msg);
void (*get_protocol_features)(TestServer *s,
CharBackend *chr, VhostUserMsg *msg);
+void (*get_config)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
};
static const char *init_hugepagefs(void);
@@ -194,6 +213,14 @@ static void append_vhost_net_opts(TestServer *s, GString
*cmd_line,
chr_opts, s->chr_name);
}
+static void append_vhost_blk_opts(TestServer *s, GString *cmd_line,
+ const char *chr_opts)
+{
+g_string_append_printf(cmd_line, QEMU_CMD_BLKCHR,
+ s->socket_path,
+ chr_opts);
+}
+
static void append_mem_opts(TestServer *server, GString *cmd_line,
int size, enum test_memfd memfd)
{
@@ -425,6 +452,12 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
+case VHOST_USER_GET_CONFIG:
+if (s->vu_ops->get_config) {
+s->vu_ops->get_config(s, chr, );
+}
+break;
+
default:
break;
}
@@ -727,6 +760,9 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
guint8 *log;
guint64 size;
+if (s->vu_ops->driver_init) {
+s->vu_ops->driver_init(obj, alloc);
+}
if (!wait_for_fds(s)) {
return;
}
@@ -796,6 +832,24 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
g_string_free(dest_cmdline, true);
}
+static void vu_blk_driver_init(void *obj, QGuestAllocator *alloc)
+{
+QVirtioBlk *blk_if;
+QVirtioDevice *dev;
+QVirtQueue *vq;
+uint64_t features;
+
+blk_if = obj;
+dev = blk_if->vdev;
+features = qvirtio_get_features(dev);
+qvirtio_set_features(dev, features);
+
+vq = qvirtqueue_setup(dev, alloc, 0);
+g_assert(vq);
+
+qvirtio_set_driver_ok(dev);
+}
+
static void wait_for_rings_started(TestServer *s, size_t count)
{
[PATCH v3 7/7] tests/qtest/vhost-user-test: enable the reconnect tests
For now a QTEST_VHOST_USER_FIXME environment variable is used to
separate reconnect tests for the vhost-user-net device. Looks like the
reconnect functionality is pretty stable, so this separation is
deprecated.
Remove it and enable these tests for the default run.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +++--
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 4b715d3..4b96312 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -1140,20 +1140,17 @@ static void register_vhost_user_test(void)
"virtio-net",
test_migrate, );
-/* keeps failing on build-system since Aug 15 2017 */
-if (getenv("QTEST_VHOST_USER_FIXME")) {
-opts.before = vhost_user_test_setup_reconnect;
-qos_add_test("vhost-user/reconnect", "virtio-net",
- test_reconnect, );
-
-opts.before = vhost_user_test_setup_connect_fail;
-qos_add_test("vhost-user/connect-fail", "virtio-net",
- test_vhost_user_started, );
-
-opts.before = vhost_user_test_setup_flags_mismatch;
-qos_add_test("vhost-user/flags-mismatch", "virtio-net",
- test_vhost_user_started, );
-}
+opts.before = vhost_user_test_setup_reconnect;
+qos_add_test("vhost-user/reconnect", "virtio-net",
+ test_reconnect, );
+
+opts.before = vhost_user_test_setup_connect_fail;
+qos_add_test("vhost-user/connect-fail", "virtio-net",
+ test_vhost_user_started, );
+
+opts.before = vhost_user_test_setup_flags_mismatch;
+qos_add_test("vhost-user/flags-mismatch", "virtio-net",
+ test_vhost_user_started, );
opts.before = vhost_user_test_setup_multiqueue;
opts.edge.extra_device_opts = "mq=on";
--
2.7.4
[PATCH v3 1/7] vhost: recheck dev state in the vhost_migration_log routine
vhost-user devices can get a disconnect in the middle of the VHOST-USER
handshake on the migration start. If disconnect event happened right
before sending next VHOST-USER command, then the vhost_dev_set_log()
call in the vhost_migration_log() function will return error. This error
will lead to the assert() and close the QEMU migration source process.
For the vhost-user devices the disconnect event should not break the
migration process, because:
- the device will be in the stopped state, so it will not be changed
during migration
- if reconnect will be made the migration log will be reinitialized as
part of reconnect/init process:
#0 vhost_log_global_start (listener=0x563989cf7be0)
at hw/virtio/vhost.c:920
#1 0x56398603d8bc in listener_add_address_space
(listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2664
#2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2740
#3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=0)
at hw/virtio/vhost.c:1385
#4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
at hw/block/vhost-user-blk.c:315
#5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
event=CHR_EVENT_OPENED)
at hw/block/vhost-user-blk.c:379
Update the vhost-user-blk device with the internal started_vu field which
will be used for initialization (vhost_user_blk_start) and clean up
(vhost_user_blk_stop). This additional flag in the VhostUserBlk structure
will be used to track whether the device really needs to be stopped and
cleaned up on a vhost-user level.
The disconnect event will set the overall VHOST device (not vhost-user) to
the stopped state, so it can be used by the general vhost_migration_log
routine.
Such approach could be propogated to the other vhost-user devices, but
better idea is just to make the same connect/disconnect code for all the
vhost-user devices.
This migration issue was slightly discussed earlier:
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 19 ---
hw/virtio/vhost.c | 27 ---
include/hw/virtio/vhost-user-blk.h | 10 ++
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 39aec42..a076b1e 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
error_report("Error starting vhost: %d", -ret);
goto err_guest_notifiers;
}
+s->started_vu = true;
/* guest_notifier_mask/pending not used yet, so just unmask
* everything here. virtio-pci will do the right thing by
@@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!s->started_vu) {
+return;
+}
+s->started_vu = false;
+
if (!k->set_guest_notifiers) {
return;
}
@@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
@@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
NULL, NULL, false);
aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
}
+
+/*
+ * Move vhost device to the stopped state. The vhost-user device
+ * will be clean up and disconnected in BH. This can be useful in
+ * the vhost migration code. If disconnect was caught there is an
+ * option for the general vhost code to get the dev state without
+ * knowing its type (in this case vhost-user).
+ */
+s->dev.started = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1a1384e..ffef7ab 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -861,21 +861,42 @@ static int vhost_migration_log(MemoryListener *listener,
bool enable)
dev->log_enabled = enable;
return 0;
}
+
+r = 0;
if (!enable) {
r = vhost_dev_set_log(dev, false);
if (r < 0) {
-return r;
+goto check_dev_state;
}
vhost_log_put(dev, false);
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
if (r < 0) {
-
Re: [PATCH v2 2/7] vhost: check queue state in the vhost_dev_set_log routine
On Thu, Aug 27, 2020 at 09:46:03PM -0400, Raphael Norwitz wrote:
> On Mon, Aug 24, 2020 at 4:41 AM Dima Stepanov wrote:
> >
> > If the vhost-user-blk daemon provides only one virtqueue, but device was
> > added with several queues, then QEMU will send more VHOST-USER command
> > than expected by daemon side. The vhost_virtqueue_start() routine
> > handles such case by checking the return value from the
> > virtio_queue_get_desc_addr() function call. Add the same check to the
> > vhost_dev_set_log() routine.
> >
> > Signed-off-by: Dima Stepanov
> > ---
> > hw/virtio/vhost.c | 12
> > 1 file changed, 12 insertions(+)
> >
> > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > index ffef7ab..a33ffd4 100644
> > --- a/hw/virtio/vhost.c
> > +++ b/hw/virtio/vhost.c
> > @@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct vhost_dev
> > *dev,
> > static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> > {
> > int r, i, idx;
> > +hwaddr addr;
> > +
> > r = vhost_dev_set_features(dev, enable_log);
> > if (r < 0) {
> > goto err_features;
> > }
> > for (i = 0; i < dev->nvqs; ++i) {
> > idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
> > +addr = virtio_queue_get_desc_addr(dev->vdev, idx);
> > +if (!addr) {
> > +/*
> > + * The queue might not be ready for start. If this
> > + * is the case there is no reason to continue the process.
> > + * The similar logic is used by the vhost_virtqueue_start()
> > + * routine.
> > + */
>
> Shouldn’t we goto err_vq here to reset the logging state of any vqs
> which have already been set?
As i understand it, no we shouldn't reset the state of other queues. In
general it is pretty valid case. Let's assume that the backend
vhost-user device supports only two queues. But for instance, the QEMU
command line is using value 4 to define number of virtqueues of such
device. In this case only 2 queues will be initializaed.
I've tried to reflect it in the comment section, that the
vhost_virtqueue_start() routine has been alread made the same:
"if a queue isn't ready for start, just return 0 without any error"
So i made the same here.
I've found this issue, while testing migration with the default
vhost-user-blk daemon. It fails with assert or sigsegv (don't remember),
because it receives NULL for the queues it doesn't have. In general
the daemon should not fall, because of unexpected VHOST_USER
communication, but also there is no reason for QEMU to send additional
packets.
>
> > +break;
> > +}
> > r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
> > enable_log);
> > if (r < 0) {
> > --
> > 2.7.4
> >
> >
Re: [PATCH v2 1/7] vhost: recheck dev state in the vhost_migration_log routine
On Thu, Aug 27, 2020 at 09:44:22PM -0400, Raphael Norwitz wrote:
> Generally I’m happy with this. Some comments:
>
> On Mon, Aug 24, 2020 at 4:40 AM Dima Stepanov wrote:
> >
> > vhost-user devices can get a disconnect in the middle of the VHOST-USER
> > handshake on the migration start. If disconnect event happened right
> > before sending next VHOST-USER command, then the vhost_dev_set_log()
> > call in the vhost_migration_log() function will return error. This error
> > will lead to the assert() and close the QEMU migration source process.
> > For the vhost-user devices the disconnect event should not break the
> > migration process, because:
> > - the device will be in the stopped state, so it will not be changed
> > during migration
> > - if reconnect will be made the migration log will be reinitialized as
> > part of reconnect/init process:
> > #0 vhost_log_global_start (listener=0x563989cf7be0)
> > at hw/virtio/vhost.c:920
> > #1 0x56398603d8bc in listener_add_address_space
> > (listener=0x563989cf7be0,
> > as=0x563986ea4340 )
> > at softmmu/memory.c:2664
> > #2 0x56398603dd30 in memory_listener_register
> > (listener=0x563989cf7be0,
> > as=0x563986ea4340 )
> > at softmmu/memory.c:2740
> > #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
> > opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
> > busyloop_timeout=0)
> > at hw/virtio/vhost.c:1385
> > #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
> > at hw/block/vhost-user-blk.c:315
> > #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
> > event=CHR_EVENT_OPENED)
> > at hw/block/vhost-user-blk.c:379
> > Update the vhost-user-blk device with the internal started field which
> > will be used for initialization and clean up. The disconnect event will
> > set the overall VHOST device to the stopped state, so it can be used by
> > the vhost_migration_log routine.
> > Such approach could be propogated to the other vhost-user devices, but
> > better idea is just to make the same connect/disconnect code for all the
> > vhost-user devices.
> >
> > This migration issue was slightly discussed earlier:
> > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
> > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
> >
>
> What you’re doing here on a structural level is adding an additional
> flag “started” to VhostUserBlk to track whether the device really
> needs to be stopped and cleaned up on a vhost level on a disconnect.
> Can you make that more clear in the commit message as you have in the
> comments?
Sounds reasonable, will update the commit message.
>
>
> > Signed-off-by: Dima Stepanov
> > ---
> > hw/block/vhost-user-blk.c | 19 ---
> > hw/virtio/vhost.c | 27 ---
> > include/hw/virtio/vhost-user-blk.h | 10 ++
> > 3 files changed, 50 insertions(+), 6 deletions(-)
> >
> > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > index a00b854..5573e89 100644
> > --- a/hw/block/vhost-user-blk.c
> > +++ b/hw/block/vhost-user-blk.c
> > @@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
> > error_report("Error starting vhost: %d", -ret);
> > goto err_guest_notifiers;
> > }
> > +s->started = true;
> >
> > /* guest_notifier_mask/pending not used yet, so just unmask
> > * everything here. virtio-pci will do the right thing by
> > @@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
> > VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
> > int ret;
> >
> > +if (!s->started) {
> > +return;
> > +}
> > +s->started = false;
> > +
> > if (!k->set_guest_notifiers) {
> > return;
> > }
> > @@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > }
> > s->connected = false;
> >
> > -if (s->dev.started) {
> > -vhost_user_blk_stop(vdev);
> > -}
> > +vhost_user_blk_stop(vdev);
> >
> > vhost_dev_cleanup(>dev);
> > }
> > @@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
> > QEMUChrEvent event)
> > NULL, NULL, false);
> > aio_bh_schedule_onesho
[PATCH v2 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
Add support for the vhost-user-blk-pci device. This node can be used by
the vhost-user-blk tests. Tests for the vhost-user-blk device are added
in the following patches.
Signed-off-by: Dima Stepanov
---
tests/qtest/libqos/virtio-blk.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/tests/qtest/libqos/virtio-blk.c b/tests/qtest/libqos/virtio-blk.c
index 5da0259..959c5dc 100644
--- a/tests/qtest/libqos/virtio-blk.c
+++ b/tests/qtest/libqos/virtio-blk.c
@@ -36,6 +36,9 @@ static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
if (!g_strcmp0(interface, "virtio")) {
return v_blk->vdev;
}
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
fprintf(stderr, "%s not present in virtio-blk-device\n", interface);
g_assert_not_reached();
@@ -120,6 +123,17 @@ static void virtio_blk_register_nodes(void)
qos_node_produces("virtio-blk-pci", "virtio-blk");
g_free(arg);
+
+/* vhost-user-blk-pci */
+arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
+
+g_free(arg);
}
libqos_init(virtio_blk_register_nodes);
--
2.7.4
[PATCH v2 7/7] tests/qtest/vhost-user-test: enable the reconnect tests
For now a QTEST_VHOST_USER_FIXME environment variable is used to
separate reconnect tests for the vhost-user-net device. Looks like the
reconnect functionality is pretty stable, so this separation is
deprecated.
Remove it and enable these tests for the default run.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +++--
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 6b4c147..5fbfa02 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -1141,20 +1141,17 @@ static void register_vhost_user_test(void)
"virtio-net",
test_migrate, );
-/* keeps failing on build-system since Aug 15 2017 */
-if (getenv("QTEST_VHOST_USER_FIXME")) {
-opts.before = vhost_user_test_setup_reconnect;
-qos_add_test("vhost-user/reconnect", "virtio-net",
- test_reconnect, );
-
-opts.before = vhost_user_test_setup_connect_fail;
-qos_add_test("vhost-user/connect-fail", "virtio-net",
- test_vhost_user_started, );
-
-opts.before = vhost_user_test_setup_flags_mismatch;
-qos_add_test("vhost-user/flags-mismatch", "virtio-net",
- test_vhost_user_started, );
-}
+opts.before = vhost_user_test_setup_reconnect;
+qos_add_test("vhost-user/reconnect", "virtio-net",
+ test_reconnect, );
+
+opts.before = vhost_user_test_setup_connect_fail;
+qos_add_test("vhost-user/connect-fail", "virtio-net",
+ test_vhost_user_started, );
+
+opts.before = vhost_user_test_setup_flags_mismatch;
+qos_add_test("vhost-user/flags-mismatch", "virtio-net",
+ test_vhost_user_started, );
opts.before = vhost_user_test_setup_multiqueue;
opts.edge.extra_device_opts = "mq=on";
--
2.7.4
[PATCH v2 6/7] tests/qtest/vhost-user-test: add migrate_reconnect test
Add new migrate_reconnect test for the vhost-user-blk device. Perform a
disconnect after sending response for the VHOST_USER_SET_LOG_BASE
command.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +
1 file changed, 25 insertions(+)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9b6e202..6b4c147 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -146,6 +146,7 @@ static VhostUserMsg m __attribute__ ((unused));
enum {
TEST_FLAGS_OK,
TEST_FLAGS_DISCONNECT,
+TEST_FLAGS_MIGRATE_DISCONNECT,
TEST_FLAGS_BAD,
TEST_FLAGS_END,
};
@@ -436,6 +437,15 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE);
g_cond_broadcast(>data_cond);
+/*
+ * Perform disconnect after sending a response. In this
+ * case the next write command on the QEMU side (for now
+ * it is SET_FEATURES will return -1, because of disconnect.
+ */
+if (s->test_flags == TEST_FLAGS_MIGRATE_DISCONNECT) {
+qemu_chr_fe_disconnect(chr);
+s->test_flags = TEST_FLAGS_BAD;
+}
break;
case VHOST_USER_SET_VRING_BASE:
@@ -737,6 +747,17 @@ static void *vhost_user_test_setup_memfd(GString
*cmd_line, void *arg)
return server;
}
+static void *vhost_user_test_setup_migrate_reconnect(GString *cmd_line,
+void *arg)
+{
+TestServer *server;
+
+server = vhost_user_test_setup_memfd(cmd_line, arg);
+server->test_flags = TEST_FLAGS_MIGRATE_DISCONNECT;
+
+return server;
+}
+
static void test_read_guest_mem(void *obj, void *arg, QGuestAllocator *alloc)
{
TestServer *server = arg;
@@ -1151,5 +1172,9 @@ static void register_vhost_user_test(void)
opts.before = vhost_user_test_setup_memfd;
qos_add_test("migrate", "vhost-user-blk",
test_migrate, );
+
+opts.before = vhost_user_test_setup_migrate_reconnect;
+qos_add_test("migrate_reconnect", "vhost-user-blk",
+ test_migrate, );
}
libqos_init(register_vhost_user_test);
--
2.7.4
[PATCH v2 2/7] vhost: check queue state in the vhost_dev_set_log routine
If the vhost-user-blk daemon provides only one virtqueue, but device was
added with several queues, then QEMU will send more VHOST-USER command
than expected by daemon side. The vhost_virtqueue_start() routine
handles such case by checking the return value from the
virtio_queue_get_desc_addr() function call. Add the same check to the
vhost_dev_set_log() routine.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 12
1 file changed, 12 insertions(+)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ffef7ab..a33ffd4 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -825,12 +825,24 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+hwaddr addr;
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
}
for (i = 0; i < dev->nvqs; ++i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
+addr = virtio_queue_get_desc_addr(dev->vdev, idx);
+if (!addr) {
+/*
+ * The queue might not be ready for start. If this
+ * is the case there is no reason to continue the process.
+ * The similar logic is used by the vhost_virtqueue_start()
+ * routine.
+ */
+break;
+}
r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
enable_log);
if (r < 0) {
--
2.7.4
[PATCH v2 3/7] tests/qtest/vhost-user-test: prepare the tests for adding new dev class
For now only vhost-user-net device is supported by the test. Other
vhost-user devices are not tested. As a first step make source code
refactoring so new devices can reuse the same test routines. To make
this provide a new vhost_user_ops structure with the methods to
initialize device, its command line or make a proper vhost-user
responses.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 105 ++
1 file changed, 76 insertions(+), 29 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9ee0f1e..3df5322 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -135,6 +135,10 @@ enum {
TEST_FLAGS_END,
};
+enum {
+VHOST_USER_NET,
+};
+
typedef struct TestServer {
gchar *socket_path;
gchar *mig_path;
@@ -154,10 +158,25 @@ typedef struct TestServer {
bool test_fail;
int test_flags;
int queues;
+struct vhost_user_ops *vu_ops;
} TestServer;
+struct vhost_user_ops {
+/* Device types. */
+int type;
+void (*append_opts)(TestServer *s, GString *cmd_line,
+const char *chr_opts);
+
+/* VHOST-USER commands. */
+void (*set_features)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
+void (*get_protocol_features)(TestServer *s,
+CharBackend *chr, VhostUserMsg *msg);
+};
+
static const char *init_hugepagefs(void);
-static TestServer *test_server_new(const gchar *name);
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops);
static void test_server_free(TestServer *server);
static void test_server_listen(TestServer *server);
@@ -167,7 +186,7 @@ enum test_memfd {
TEST_MEMFD_NO,
};
-static void append_vhost_opts(TestServer *s, GString *cmd_line,
+static void append_vhost_net_opts(TestServer *s, GString *cmd_line,
const char *chr_opts)
{
g_string_append_printf(cmd_line, QEMU_CMD_CHR QEMU_CMD_NETDEV,
@@ -332,25 +351,15 @@ static void chr_read(void *opaque, const uint8_t *buf,
int size)
break;
case VHOST_USER_SET_FEATURES:
-g_assert_cmpint(msg.payload.u64 & (0x1ULL <<
VHOST_USER_F_PROTOCOL_FEATURES),
-!=, 0ULL);
-if (s->test_flags == TEST_FLAGS_DISCONNECT) {
-qemu_chr_fe_disconnect(chr);
-s->test_flags = TEST_FLAGS_BAD;
+if (s->vu_ops->set_features) {
+s->vu_ops->set_features(s, chr, );
}
break;
case VHOST_USER_GET_PROTOCOL_FEATURES:
-/* send back features to qemu */
-msg.flags |= VHOST_USER_REPLY_MASK;
-msg.size = sizeof(m.payload.u64);
-msg.payload.u64 = 1 << VHOST_USER_PROTOCOL_F_LOG_SHMFD;
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_CROSS_ENDIAN;
-if (s->queues > 1) {
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_MQ;
+if (s->vu_ops->get_protocol_features) {
+s->vu_ops->get_protocol_features(s, chr, );
}
-p = (uint8_t *)
-qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
case VHOST_USER_GET_VRING_BASE:
@@ -467,7 +476,8 @@ static const char *init_hugepagefs(void)
#endif
}
-static TestServer *test_server_new(const gchar *name)
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops)
{
TestServer *server = g_new0(TestServer, 1);
char template[] = "/tmp/vhost-test-XX";
@@ -495,6 +505,7 @@ static TestServer *test_server_new(const gchar *name)
server->log_fd = -1;
server->queues = 1;
+server->vu_ops = ops;
return server;
}
@@ -669,11 +680,11 @@ static void vhost_user_test_cleanup(void *s)
static void *vhost_user_test_setup(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_AUTO);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -682,11 +693,11 @@ static void *vhost_user_test_setup(GString *cmd_line,
void *arg)
static void *vhost_user_test_setup_memfd(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_YES);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -720,7 +73
[PATCH v2 1/7] vhost: recheck dev state in the vhost_migration_log routine
vhost-user devices can get a disconnect in the middle of the VHOST-USER
handshake on the migration start. If disconnect event happened right
before sending next VHOST-USER command, then the vhost_dev_set_log()
call in the vhost_migration_log() function will return error. This error
will lead to the assert() and close the QEMU migration source process.
For the vhost-user devices the disconnect event should not break the
migration process, because:
- the device will be in the stopped state, so it will not be changed
during migration
- if reconnect will be made the migration log will be reinitialized as
part of reconnect/init process:
#0 vhost_log_global_start (listener=0x563989cf7be0)
at hw/virtio/vhost.c:920
#1 0x56398603d8bc in listener_add_address_space
(listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2664
#2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2740
#3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=0)
at hw/virtio/vhost.c:1385
#4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
at hw/block/vhost-user-blk.c:315
#5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
event=CHR_EVENT_OPENED)
at hw/block/vhost-user-blk.c:379
Update the vhost-user-blk device with the internal started field which
will be used for initialization and clean up. The disconnect event will
set the overall VHOST device to the stopped state, so it can be used by
the vhost_migration_log routine.
Such approach could be propogated to the other vhost-user devices, but
better idea is just to make the same connect/disconnect code for all the
vhost-user devices.
This migration issue was slightly discussed earlier:
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 19 ---
hw/virtio/vhost.c | 27 ---
include/hw/virtio/vhost-user-blk.h | 10 ++
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index a00b854..5573e89 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
error_report("Error starting vhost: %d", -ret);
goto err_guest_notifiers;
}
+s->started = true;
/* guest_notifier_mask/pending not used yet, so just unmask
* everything here. virtio-pci will do the right thing by
@@ -175,6 +176,11 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!s->started) {
+return;
+}
+s->started = false;
+
if (!k->set_guest_notifiers) {
return;
}
@@ -341,9 +347,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
@@ -399,6 +403,15 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
NULL, NULL, false);
aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
}
+
+/*
+ * Move vhost device to the stopped state. The vhost-user device
+ * will be clean up and disconnected in BH. This can be useful in
+ * the vhost migration code. If disconnect was caught there is an
+ * option for the general vhost code to get the dev state without
+ * knowing its type (in this case vhost-user).
+ */
+s->dev.started = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1a1384e..ffef7ab 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -861,21 +861,42 @@ static int vhost_migration_log(MemoryListener *listener,
bool enable)
dev->log_enabled = enable;
return 0;
}
+
+r = 0;
if (!enable) {
r = vhost_dev_set_log(dev, false);
if (r < 0) {
-return r;
+goto check_dev_state;
}
vhost_log_put(dev, false);
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
if (r < 0) {
-return r;
+goto check_dev_state;
}
}
+
+check_dev_state:
dev->log_enabled = enable;
-return 0;
+/*
+ * vhost-user-* devices could change their state during log
+ * initialization due to disc
[PATCH v2 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
Add vhost_user_ops structure for the vhost-user-blk device class. Add
the test_reconnect and test_migrate tests for this device.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 140 +-
1 file changed, 138 insertions(+), 2 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 3df5322..9b6e202 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -24,6 +24,7 @@
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
#include "libqos/virtio-pci.h"
+#include "libqos/virtio-blk.h"
#include "libqos/malloc-pc.h"
#include "hw/virtio/virtio-net.h"
@@ -31,6 +32,7 @@
#include "standard-headers/linux/vhost_types.h"
#include "standard-headers/linux/virtio_ids.h"
#include "standard-headers/linux/virtio_net.h"
+#include "standard-headers/linux/virtio_blk.h"
#ifdef CONFIG_LINUX
#include
@@ -43,6 +45,7 @@
" -numa node,memdev=mem"
#define QEMU_CMD_CHR" -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=hs0,chardev=%s,vhostforce"
+#define QEMU_CMD_BLKCHR " -chardev socket,id=chdev0,path=%s%s"
#define HUGETLBFS_MAGIC 0x958458f6
@@ -55,6 +58,7 @@
#define VHOST_USER_PROTOCOL_F_MQ 0
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+#define VHOST_USER_PROTOCOL_F_CONFIG 9
#define VHOST_LOG_PAGE 0x1000
@@ -78,6 +82,8 @@ typedef enum VhostUserRequest {
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
+VHOST_USER_GET_CONFIG = 24,
+VHOST_USER_SET_CONFIG = 25,
VHOST_USER_MAX
} VhostUserRequest;
@@ -99,6 +105,14 @@ typedef struct VhostUserLog {
uint64_t mmap_offset;
} VhostUserLog;
+#define VHOST_USER_MAX_CONFIG_SIZE 256
+typedef struct VhostUserConfig {
+uint32_t offset;
+uint32_t size;
+uint32_t flags;
+uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+} VhostUserConfig;
+
typedef struct VhostUserMsg {
VhostUserRequest request;
@@ -114,6 +128,7 @@ typedef struct VhostUserMsg {
struct vhost_vring_addr addr;
VhostUserMemory memory;
VhostUserLog log;
+VhostUserConfig config;
} payload;
} QEMU_PACKED VhostUserMsg;
@@ -137,6 +152,7 @@ enum {
enum {
VHOST_USER_NET,
+VHOST_USER_BLK,
};
typedef struct TestServer {
@@ -166,12 +182,15 @@ struct vhost_user_ops {
int type;
void (*append_opts)(TestServer *s, GString *cmd_line,
const char *chr_opts);
+void (*driver_init)(void *obj, QGuestAllocator *alloc);
/* VHOST-USER commands. */
void (*set_features)(TestServer *s, CharBackend *chr,
VhostUserMsg *msg);
void (*get_protocol_features)(TestServer *s,
CharBackend *chr, VhostUserMsg *msg);
+void (*get_config)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
};
static const char *init_hugepagefs(void);
@@ -194,6 +213,14 @@ static void append_vhost_net_opts(TestServer *s, GString
*cmd_line,
chr_opts, s->chr_name);
}
+static void append_vhost_blk_opts(TestServer *s, GString *cmd_line,
+ const char *chr_opts)
+{
+g_string_append_printf(cmd_line, QEMU_CMD_BLKCHR,
+ s->socket_path,
+ chr_opts);
+}
+
static void append_mem_opts(TestServer *server, GString *cmd_line,
int size, enum test_memfd memfd)
{
@@ -425,6 +452,12 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
+case VHOST_USER_GET_CONFIG:
+if (s->vu_ops->get_config) {
+s->vu_ops->get_config(s, chr, );
+}
+break;
+
default:
break;
}
@@ -727,6 +760,9 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
guint8 *log;
guint64 size;
+if (s->vu_ops->driver_init) {
+s->vu_ops->driver_init(obj, alloc);
+}
if (!wait_for_fds(s)) {
return;
}
@@ -796,6 +832,24 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
g_string_free(dest_cmdline, true);
}
+static void vu_blk_driver_init(void *obj, QGuestAllocator *alloc)
+{
+QVirtioBlk *blk_if;
+QVirtioDevice *dev;
+QVirtQueue *vq;
+uint64_t features;
+
+blk_if = obj;
+dev = blk_if->vdev;
+features = qvirtio_get_features(dev);
+qvirtio_set_features(dev, features);
+
+vq = qvirtqueue_setup(dev, alloc, 0);
+g_assert(vq);
+
+qvirtio_set_driver_ok(dev);
+}
+
static void wait_for_rings_started(TestServer *s, size_t count)
{
[PATCH v2 0/7] vhost-user-blk: fix the migration issue and enhance qtests
v1 -> v2: - add comments to connected/started fields in the header file - move the "s->started" logic from the vhost_user_blk_disconnect routine to the vhost_user_blk_stop routine Reference e-mail threads: - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html If vhost-user daemon is used as a backend for the vhost device, then we should consider a possibility of disconnect at any moment. There was a general question here: should we consider it as an error or okay state for the vhost-user devices during migration process? I think the disconnect event for the vhost-user devices should not break the migration process, because: - the device will be in the stopped state, so it will not be changed during migration - if reconnect will be made the migration log will be reinitialized as part of reconnect/init process: #0 vhost_log_global_start (listener=0x563989cf7be0) at hw/virtio/vhost.c:920 #1 0x56398603d8bc in listener_add_address_space (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2664 #2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2740 #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, busyloop_timeout=0) at hw/virtio/vhost.c:1385 #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) at hw/block/vhost-user-blk.c:315 #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, event=CHR_EVENT_OPENED) at hw/block/vhost-user-blk.c:379 The first patch in the patchset fixes this issue by setting vhost device to the stopped state in the disconnect handler and check it the vhost_migration_log() routine before returning from the function. qtest framework was updated to test vhost-user-blk functionality. The vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to reproduce the original issue found. Dima Stepanov (7): vhost: recheck dev state in the vhost_migration_log routine vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/libqos/virtio-blk: add support for vhost-user-blk tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test tests/qtest/vhost-user-test: enable the reconnect tests hw/block/vhost-user-blk.c | 19 ++- hw/virtio/vhost.c | 39 - include/hw/virtio/vhost-user-blk.h | 10 ++ tests/qtest/libqos/virtio-blk.c| 14 ++ tests/qtest/vhost-user-test.c | 291 +++-- 5 files changed, 324 insertions(+), 49 deletions(-) -- 2.7.4
Re: [PATCH v1 0/7] vhost-user-blk: fix the migration issue and enhance qtests
On Tue, Aug 04, 2020 at 10:19:17AM -0400, Michael S. Tsirkin wrote: > On Tue, Aug 04, 2020 at 01:36:45PM +0300, Dima Stepanov wrote: > > Reference e-mail threads: > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html > > - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html > > > > If vhost-user daemon is used as a backend for the vhost device, then we > > should consider a possibility of disconnect at any moment. There was a > > general > > question here: should we consider it as an error or okay state for the > > vhost-user > > devices during migration process? > > I think the disconnect event for the vhost-user devices should not break the > > migration process, because: > > - the device will be in the stopped state, so it will not be changed > > during migration > > - if reconnect will be made the migration log will be reinitialized as > > part of reconnect/init process: > > #0 vhost_log_global_start (listener=0x563989cf7be0) > > at hw/virtio/vhost.c:920 > > #1 0x56398603d8bc in listener_add_address_space > > (listener=0x563989cf7be0, > > as=0x563986ea4340 ) > > at softmmu/memory.c:2664 > > #2 0x56398603dd30 in memory_listener_register > > (listener=0x563989cf7be0, > > as=0x563986ea4340 ) > > at softmmu/memory.c:2740 > > #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, > > opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, > > busyloop_timeout=0) > > at hw/virtio/vhost.c:1385 > > #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) > > at hw/block/vhost-user-blk.c:315 > > #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, > > event=CHR_EVENT_OPENED) > > at hw/block/vhost-user-blk.c:379 > > The first patch in the patchset fixes this issue by setting vhost device to > > the > > stopped state in the disconnect handler and check it the > > vhost_migration_log() > > routine before returning from the function. > > So I'm a bit confused. Isn't the connected state sufficient for this? > If not, adding some code comments explaining when is each flag > set would be helpful. > Thanks! Well, not really. The "connected" field is used internally as the flag in the _connect/_disconnect routines. Because we made oneshot_bh for the disconnect routine we can't really use it. Also in general the vhost_log_global_start() routine doesn't know anything about the device type (in this case vhost-user), so it is not correct to use this variable here. So what i want to reflect that vhost-user-blk code should change the state of the device to stopped state and not the general vhost code should check the connection status. Because of it i've update the general (struct vhost_dev)->started field with the stopped state. But yes, it is a good idea to update the comments in include/hw/virtio/vhost-user-blk.h. Will do it in v2. > > qtest framework was updated to test vhost-user-blk functionality. The > > vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to > > reproduce > > the original issue found. > > > > Dima Stepanov (7): > > vhost: recheck dev state in the vhost_migration_log routine > > vhost: check queue state in the vhost_dev_set_log routine > > tests/qtest/vhost-user-test: prepare the tests for adding new dev > > class > > tests/qtest/libqos/virtio-blk: add support for vhost-user-blk > > tests/qtest/vhost-user-test: add support for the vhost-user-blk device > > tests/qtest/vhost-user-test: add migrate_reconnect test > > tests/qtest/vhost-user-test: enable the reconnect tests > > > > hw/block/vhost-user-blk.c | 13 +- > > hw/virtio/vhost.c | 39 - > > include/hw/virtio/vhost-user-blk.h | 1 + > > tests/qtest/libqos/virtio-blk.c| 14 ++ > > tests/qtest/vhost-user-test.c | 291 > > +++-- > > 5 files changed, 311 insertions(+), 47 deletions(-) > > > > -- > > 2.7.4 >
[PATCH v1 5/7] tests/qtest/vhost-user-test: add support for the vhost-user-blk device
Add vhost_user_ops structure for the vhost-user-blk device class. Add
the test_reconnect and test_migrate tests for this device.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 140 +-
1 file changed, 138 insertions(+), 2 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 3df5322..9b6e202 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -24,6 +24,7 @@
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
#include "libqos/virtio-pci.h"
+#include "libqos/virtio-blk.h"
#include "libqos/malloc-pc.h"
#include "hw/virtio/virtio-net.h"
@@ -31,6 +32,7 @@
#include "standard-headers/linux/vhost_types.h"
#include "standard-headers/linux/virtio_ids.h"
#include "standard-headers/linux/virtio_net.h"
+#include "standard-headers/linux/virtio_blk.h"
#ifdef CONFIG_LINUX
#include
@@ -43,6 +45,7 @@
" -numa node,memdev=mem"
#define QEMU_CMD_CHR" -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=hs0,chardev=%s,vhostforce"
+#define QEMU_CMD_BLKCHR " -chardev socket,id=chdev0,path=%s%s"
#define HUGETLBFS_MAGIC 0x958458f6
@@ -55,6 +58,7 @@
#define VHOST_USER_PROTOCOL_F_MQ 0
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+#define VHOST_USER_PROTOCOL_F_CONFIG 9
#define VHOST_LOG_PAGE 0x1000
@@ -78,6 +82,8 @@ typedef enum VhostUserRequest {
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
+VHOST_USER_GET_CONFIG = 24,
+VHOST_USER_SET_CONFIG = 25,
VHOST_USER_MAX
} VhostUserRequest;
@@ -99,6 +105,14 @@ typedef struct VhostUserLog {
uint64_t mmap_offset;
} VhostUserLog;
+#define VHOST_USER_MAX_CONFIG_SIZE 256
+typedef struct VhostUserConfig {
+uint32_t offset;
+uint32_t size;
+uint32_t flags;
+uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+} VhostUserConfig;
+
typedef struct VhostUserMsg {
VhostUserRequest request;
@@ -114,6 +128,7 @@ typedef struct VhostUserMsg {
struct vhost_vring_addr addr;
VhostUserMemory memory;
VhostUserLog log;
+VhostUserConfig config;
} payload;
} QEMU_PACKED VhostUserMsg;
@@ -137,6 +152,7 @@ enum {
enum {
VHOST_USER_NET,
+VHOST_USER_BLK,
};
typedef struct TestServer {
@@ -166,12 +182,15 @@ struct vhost_user_ops {
int type;
void (*append_opts)(TestServer *s, GString *cmd_line,
const char *chr_opts);
+void (*driver_init)(void *obj, QGuestAllocator *alloc);
/* VHOST-USER commands. */
void (*set_features)(TestServer *s, CharBackend *chr,
VhostUserMsg *msg);
void (*get_protocol_features)(TestServer *s,
CharBackend *chr, VhostUserMsg *msg);
+void (*get_config)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
};
static const char *init_hugepagefs(void);
@@ -194,6 +213,14 @@ static void append_vhost_net_opts(TestServer *s, GString
*cmd_line,
chr_opts, s->chr_name);
}
+static void append_vhost_blk_opts(TestServer *s, GString *cmd_line,
+ const char *chr_opts)
+{
+g_string_append_printf(cmd_line, QEMU_CMD_BLKCHR,
+ s->socket_path,
+ chr_opts);
+}
+
static void append_mem_opts(TestServer *server, GString *cmd_line,
int size, enum test_memfd memfd)
{
@@ -425,6 +452,12 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
+case VHOST_USER_GET_CONFIG:
+if (s->vu_ops->get_config) {
+s->vu_ops->get_config(s, chr, );
+}
+break;
+
default:
break;
}
@@ -727,6 +760,9 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
guint8 *log;
guint64 size;
+if (s->vu_ops->driver_init) {
+s->vu_ops->driver_init(obj, alloc);
+}
if (!wait_for_fds(s)) {
return;
}
@@ -796,6 +832,24 @@ static void test_migrate(void *obj, void *arg,
QGuestAllocator *alloc)
g_string_free(dest_cmdline, true);
}
+static void vu_blk_driver_init(void *obj, QGuestAllocator *alloc)
+{
+QVirtioBlk *blk_if;
+QVirtioDevice *dev;
+QVirtQueue *vq;
+uint64_t features;
+
+blk_if = obj;
+dev = blk_if->vdev;
+features = qvirtio_get_features(dev);
+qvirtio_set_features(dev, features);
+
+vq = qvirtqueue_setup(dev, alloc, 0);
+g_assert(vq);
+
+qvirtio_set_driver_ok(dev);
+}
+
static void wait_for_rings_started(TestServer *s, size_t count)
{
[PATCH v1 7/7] tests/qtest/vhost-user-test: enable the reconnect tests
For now a QTEST_VHOST_USER_FIXME environment variable is used to
separate reconnect tests for the vhost-user-net device. Looks like the
reconnect functionality is pretty stable, so this separation is
deprecated.
Remove it and enable these tests for the default run.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +++--
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 6b4c147..5fbfa02 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -1141,20 +1141,17 @@ static void register_vhost_user_test(void)
"virtio-net",
test_migrate, );
-/* keeps failing on build-system since Aug 15 2017 */
-if (getenv("QTEST_VHOST_USER_FIXME")) {
-opts.before = vhost_user_test_setup_reconnect;
-qos_add_test("vhost-user/reconnect", "virtio-net",
- test_reconnect, );
-
-opts.before = vhost_user_test_setup_connect_fail;
-qos_add_test("vhost-user/connect-fail", "virtio-net",
- test_vhost_user_started, );
-
-opts.before = vhost_user_test_setup_flags_mismatch;
-qos_add_test("vhost-user/flags-mismatch", "virtio-net",
- test_vhost_user_started, );
-}
+opts.before = vhost_user_test_setup_reconnect;
+qos_add_test("vhost-user/reconnect", "virtio-net",
+ test_reconnect, );
+
+opts.before = vhost_user_test_setup_connect_fail;
+qos_add_test("vhost-user/connect-fail", "virtio-net",
+ test_vhost_user_started, );
+
+opts.before = vhost_user_test_setup_flags_mismatch;
+qos_add_test("vhost-user/flags-mismatch", "virtio-net",
+ test_vhost_user_started, );
opts.before = vhost_user_test_setup_multiqueue;
opts.edge.extra_device_opts = "mq=on";
--
2.7.4
[PATCH v1 3/7] tests/qtest/vhost-user-test: prepare the tests for adding new dev class
For now only vhost-user-net device is supported by the test. Other
vhost-user devices are not tested. As a first step make source code
refactoring so new devices can reuse the same test routines. To make
this provide a new vhost_user_ops structure with the methods to
initialize device, its command line or make a proper vhost-user
responses.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 105 ++
1 file changed, 76 insertions(+), 29 deletions(-)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9ee0f1e..3df5322 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -135,6 +135,10 @@ enum {
TEST_FLAGS_END,
};
+enum {
+VHOST_USER_NET,
+};
+
typedef struct TestServer {
gchar *socket_path;
gchar *mig_path;
@@ -154,10 +158,25 @@ typedef struct TestServer {
bool test_fail;
int test_flags;
int queues;
+struct vhost_user_ops *vu_ops;
} TestServer;
+struct vhost_user_ops {
+/* Device types. */
+int type;
+void (*append_opts)(TestServer *s, GString *cmd_line,
+const char *chr_opts);
+
+/* VHOST-USER commands. */
+void (*set_features)(TestServer *s, CharBackend *chr,
+VhostUserMsg *msg);
+void (*get_protocol_features)(TestServer *s,
+CharBackend *chr, VhostUserMsg *msg);
+};
+
static const char *init_hugepagefs(void);
-static TestServer *test_server_new(const gchar *name);
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops);
static void test_server_free(TestServer *server);
static void test_server_listen(TestServer *server);
@@ -167,7 +186,7 @@ enum test_memfd {
TEST_MEMFD_NO,
};
-static void append_vhost_opts(TestServer *s, GString *cmd_line,
+static void append_vhost_net_opts(TestServer *s, GString *cmd_line,
const char *chr_opts)
{
g_string_append_printf(cmd_line, QEMU_CMD_CHR QEMU_CMD_NETDEV,
@@ -332,25 +351,15 @@ static void chr_read(void *opaque, const uint8_t *buf,
int size)
break;
case VHOST_USER_SET_FEATURES:
-g_assert_cmpint(msg.payload.u64 & (0x1ULL <<
VHOST_USER_F_PROTOCOL_FEATURES),
-!=, 0ULL);
-if (s->test_flags == TEST_FLAGS_DISCONNECT) {
-qemu_chr_fe_disconnect(chr);
-s->test_flags = TEST_FLAGS_BAD;
+if (s->vu_ops->set_features) {
+s->vu_ops->set_features(s, chr, );
}
break;
case VHOST_USER_GET_PROTOCOL_FEATURES:
-/* send back features to qemu */
-msg.flags |= VHOST_USER_REPLY_MASK;
-msg.size = sizeof(m.payload.u64);
-msg.payload.u64 = 1 << VHOST_USER_PROTOCOL_F_LOG_SHMFD;
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_CROSS_ENDIAN;
-if (s->queues > 1) {
-msg.payload.u64 |= 1 << VHOST_USER_PROTOCOL_F_MQ;
+if (s->vu_ops->get_protocol_features) {
+s->vu_ops->get_protocol_features(s, chr, );
}
-p = (uint8_t *)
-qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE + msg.size);
break;
case VHOST_USER_GET_VRING_BASE:
@@ -467,7 +476,8 @@ static const char *init_hugepagefs(void)
#endif
}
-static TestServer *test_server_new(const gchar *name)
+static TestServer *test_server_new(const gchar *name,
+struct vhost_user_ops *ops)
{
TestServer *server = g_new0(TestServer, 1);
char template[] = "/tmp/vhost-test-XX";
@@ -495,6 +505,7 @@ static TestServer *test_server_new(const gchar *name)
server->log_fd = -1;
server->queues = 1;
+server->vu_ops = ops;
return server;
}
@@ -669,11 +680,11 @@ static void vhost_user_test_cleanup(void *s)
static void *vhost_user_test_setup(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_AUTO);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -682,11 +693,11 @@ static void *vhost_user_test_setup(GString *cmd_line,
void *arg)
static void *vhost_user_test_setup_memfd(GString *cmd_line, void *arg)
{
-TestServer *server = test_server_new("vhost-user-test");
+TestServer *server = test_server_new("vhost-user-test", arg);
test_server_listen(server);
append_mem_opts(server, cmd_line, 256, TEST_MEMFD_YES);
-append_vhost_opts(server, cmd_line, "");
+server->vu_ops->append_opts(server, cmd_line, "");
g_test_queue_destroy(vhost_user_test_cleanup, server);
@@ -720,7 +73
[PATCH v1 6/7] tests/qtest/vhost-user-test: add migrate_reconnect test
Add new migrate_reconnect test for the vhost-user-blk device. Perform a
disconnect after sending response for the VHOST_USER_SET_LOG_BASE
command.
Signed-off-by: Dima Stepanov
---
tests/qtest/vhost-user-test.c | 25 +
1 file changed, 25 insertions(+)
diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
index 9b6e202..6b4c147 100644
--- a/tests/qtest/vhost-user-test.c
+++ b/tests/qtest/vhost-user-test.c
@@ -146,6 +146,7 @@ static VhostUserMsg m __attribute__ ((unused));
enum {
TEST_FLAGS_OK,
TEST_FLAGS_DISCONNECT,
+TEST_FLAGS_MIGRATE_DISCONNECT,
TEST_FLAGS_BAD,
TEST_FLAGS_END,
};
@@ -436,6 +437,15 @@ static void chr_read(void *opaque, const uint8_t *buf, int
size)
qemu_chr_fe_write_all(chr, p, VHOST_USER_HDR_SIZE);
g_cond_broadcast(>data_cond);
+/*
+ * Perform disconnect after sending a response. In this
+ * case the next write command on the QEMU side (for now
+ * it is SET_FEATURES will return -1, because of disconnect.
+ */
+if (s->test_flags == TEST_FLAGS_MIGRATE_DISCONNECT) {
+qemu_chr_fe_disconnect(chr);
+s->test_flags = TEST_FLAGS_BAD;
+}
break;
case VHOST_USER_SET_VRING_BASE:
@@ -737,6 +747,17 @@ static void *vhost_user_test_setup_memfd(GString
*cmd_line, void *arg)
return server;
}
+static void *vhost_user_test_setup_migrate_reconnect(GString *cmd_line,
+void *arg)
+{
+TestServer *server;
+
+server = vhost_user_test_setup_memfd(cmd_line, arg);
+server->test_flags = TEST_FLAGS_MIGRATE_DISCONNECT;
+
+return server;
+}
+
static void test_read_guest_mem(void *obj, void *arg, QGuestAllocator *alloc)
{
TestServer *server = arg;
@@ -1151,5 +1172,9 @@ static void register_vhost_user_test(void)
opts.before = vhost_user_test_setup_memfd;
qos_add_test("migrate", "vhost-user-blk",
test_migrate, );
+
+opts.before = vhost_user_test_setup_migrate_reconnect;
+qos_add_test("migrate_reconnect", "vhost-user-blk",
+ test_migrate, );
}
libqos_init(register_vhost_user_test);
--
2.7.4
[PATCH v1 1/7] vhost: recheck dev state in the vhost_migration_log routine
vhost-user devices can get a disconnect in the middle of the VHOST-USER
handshake on the migration start. If disconnect event happened right
before sending next VHOST-USER command, then the vhost_dev_set_log()
call in the vhost_migration_log() function will return error. This error
will lead to the assert() and close the QEMU migration source process.
For the vhost-user devices the disconnect event should not break the
migration process, because:
- the device will be in the stopped state, so it will not be changed
during migration
- if reconnect will be made the migration log will be reinitialized as
part of reconnect/init process:
#0 vhost_log_global_start (listener=0x563989cf7be0)
at hw/virtio/vhost.c:920
#1 0x56398603d8bc in listener_add_address_space
(listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2664
#2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0,
as=0x563986ea4340 )
at softmmu/memory.c:2740
#3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8,
opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=0)
at hw/virtio/vhost.c:1385
#4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990)
at hw/block/vhost-user-blk.c:315
#5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990,
event=CHR_EVENT_OPENED)
at hw/block/vhost-user-blk.c:379
Update the vhost-user-blk device with the internal started field which
will be used for initialization and clean up. The disconnect event will
set the overall VHOST device to the stopped state, so it can be used by
the vhost_migration_log routine.
Such approach could be propogated to the other vhost-user devices, but
better idea is just to make the same connect/disconnect code for all the
vhost-user devices.
This migration issue was slightly discussed earlier:
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html
- https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 13 -
hw/virtio/vhost.c | 27 ---
include/hw/virtio/vhost-user-blk.h | 1 +
3 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index a00b854..9da08ec 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -150,6 +150,7 @@ static int vhost_user_blk_start(VirtIODevice *vdev)
error_report("Error starting vhost: %d", -ret);
goto err_guest_notifiers;
}
+s->started = true;
/* guest_notifier_mask/pending not used yet, so just unmask
* everything here. virtio-pci will do the right thing by
@@ -341,8 +342,9 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
+if (s->started) {
vhost_user_blk_stop(vdev);
+s->started = false;
}
vhost_dev_cleanup(>dev);
@@ -399,6 +401,15 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
NULL, NULL, false);
aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
}
+
+/*
+ * Move vhost device to the stopped state. The vhost-user device
+ * will be clean up and disconnected in BH. This can be useful in
+ * the vhost migration code. If disconnect was caught there is an
+ * option for the general vhost code to get the dev state without
+ * knowing its type (in this case vhost-user).
+ */
+s->dev.started = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1a1384e..ffef7ab 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -861,21 +861,42 @@ static int vhost_migration_log(MemoryListener *listener,
bool enable)
dev->log_enabled = enable;
return 0;
}
+
+r = 0;
if (!enable) {
r = vhost_dev_set_log(dev, false);
if (r < 0) {
-return r;
+goto check_dev_state;
}
vhost_log_put(dev, false);
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
if (r < 0) {
-return r;
+goto check_dev_state;
}
}
+
+check_dev_state:
dev->log_enabled = enable;
-return 0;
+/*
+ * vhost-user-* devices could change their state during log
+ * initialization due to disconnect. So check dev state after
+ * vhost communication.
+ */
+if (!dev->started) {
+/*
+ * Since device is in the stopped state, it is okay for
+ * migration. Return success.
+ */
+r = 0;
+}
+if (r) {
+/* An
[PATCH v1 0/7] vhost-user-blk: fix the migration issue and enhance qtests
Reference e-mail threads: - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg01509.html - https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05241.html If vhost-user daemon is used as a backend for the vhost device, then we should consider a possibility of disconnect at any moment. There was a general question here: should we consider it as an error or okay state for the vhost-user devices during migration process? I think the disconnect event for the vhost-user devices should not break the migration process, because: - the device will be in the stopped state, so it will not be changed during migration - if reconnect will be made the migration log will be reinitialized as part of reconnect/init process: #0 vhost_log_global_start (listener=0x563989cf7be0) at hw/virtio/vhost.c:920 #1 0x56398603d8bc in listener_add_address_space (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2664 #2 0x56398603dd30 in memory_listener_register (listener=0x563989cf7be0, as=0x563986ea4340 ) at softmmu/memory.c:2740 #3 0x563985fd6956 in vhost_dev_init (hdev=0x563989cf7bd8, opaque=0x563989cf7e30, backend_type=VHOST_BACKEND_TYPE_USER, busyloop_timeout=0) at hw/virtio/vhost.c:1385 #4 0x563985f7d0b8 in vhost_user_blk_connect (dev=0x563989cf7990) at hw/block/vhost-user-blk.c:315 #5 0x563985f7d3f6 in vhost_user_blk_event (opaque=0x563989cf7990, event=CHR_EVENT_OPENED) at hw/block/vhost-user-blk.c:379 The first patch in the patchset fixes this issue by setting vhost device to the stopped state in the disconnect handler and check it the vhost_migration_log() routine before returning from the function. qtest framework was updated to test vhost-user-blk functionality. The vhost-user-blk/vhost-user-blk-tests/migrate_reconnect test was added to reproduce the original issue found. Dima Stepanov (7): vhost: recheck dev state in the vhost_migration_log routine vhost: check queue state in the vhost_dev_set_log routine tests/qtest/vhost-user-test: prepare the tests for adding new dev class tests/qtest/libqos/virtio-blk: add support for vhost-user-blk tests/qtest/vhost-user-test: add support for the vhost-user-blk device tests/qtest/vhost-user-test: add migrate_reconnect test tests/qtest/vhost-user-test: enable the reconnect tests hw/block/vhost-user-blk.c | 13 +- hw/virtio/vhost.c | 39 - include/hw/virtio/vhost-user-blk.h | 1 + tests/qtest/libqos/virtio-blk.c| 14 ++ tests/qtest/vhost-user-test.c | 291 +++-- 5 files changed, 311 insertions(+), 47 deletions(-) -- 2.7.4
[PATCH v1 4/7] tests/qtest/libqos/virtio-blk: add support for vhost-user-blk
Add support for the vhost-user-blk-pci device. This node can be used by
the vhost-user-blk tests. Tests for the vhost-user-blk device are added
in the following patches.
Signed-off-by: Dima Stepanov
---
tests/qtest/libqos/virtio-blk.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/tests/qtest/libqos/virtio-blk.c b/tests/qtest/libqos/virtio-blk.c
index 5fc6940..c7878ab 100644
--- a/tests/qtest/libqos/virtio-blk.c
+++ b/tests/qtest/libqos/virtio-blk.c
@@ -36,6 +36,9 @@ static void *qvirtio_blk_get_driver(QVirtioBlk *v_blk,
if (!g_strcmp0(interface, "virtio")) {
return v_blk->vdev;
}
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
fprintf(stderr, "%s not present in virtio-blk-device\n", interface);
g_assert_not_reached();
@@ -120,6 +123,17 @@ static void virtio_blk_register_nodes(void)
qos_node_produces("virtio-blk-pci", "virtio-blk");
g_free(arg);
+
+/* vhost-user-blk-pci */
+arg = g_strdup_printf("id=drv0,chardev=chdev0,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", virtio_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-user-blk-pci", "vhost-user-blk");
+
+g_free(arg);
}
libqos_init(virtio_blk_register_nodes);
--
2.7.4
Re: [PATCH v4 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
On Sat, May 30, 2020 at 08:55:30PM -0400, Raphael Norwitz wrote:
> On Thu, May 28, 2020 at 5:13 AM Dima Stepanov wrote:
> >
> > A socket write during vhost-user communication may trigger a disconnect
> > event, calling vhost_user_blk_disconnect() and clearing all the
> > vhost_dev structures holding data that vhost-user functions expect to
> > remain valid to roll back initialization correctly. Delay the cleanup to
> > keep vhost_dev structure valid.
> > There are two possible states to handle:
> > 1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
> > the caller routine.
> > 2. RUN_STATE_RUNNING: delay by using bh
> >
> > BH changes are based on the similar changes for the vhost-user-net
> > device:
> > commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
> > "vhost-user: delay vhost_user_stop"
> >
> > Signed-off-by: Dima Stepanov
>
> Reviewed-by: Raphael Norwitz
>
> Li Feng - would you also like to sign off here?
Raphael,
Will you take this patchset for merging or what is the next step? )
Thanks, Dima.
>
> > ---
> > hw/block/vhost-user-blk.c | 38 +-
> > 1 file changed, 37 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > index 9d8c0b3..76838e7 100644
> > --- a/hw/block/vhost-user-blk.c
> > +++ b/hw/block/vhost-user-blk.c
> > @@ -349,6 +349,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > vhost_dev_cleanup(>dev);
> > }
> >
> > +static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
> > +
> > +static void vhost_user_blk_chr_closed_bh(void *opaque)
> > +{
> > +DeviceState *dev = opaque;
> > +VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > +VHostUserBlk *s = VHOST_USER_BLK(vdev);
> > +
> > +vhost_user_blk_disconnect(dev);
> > +qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
> > +NULL, opaque, NULL, true);
> > +}
> > +
> > static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
> > {
> > DeviceState *dev = opaque;
> > @@ -363,7 +376,30 @@ static void vhost_user_blk_event(void *opaque,
> > QEMUChrEvent event)
> > }
> > break;
> > case CHR_EVENT_CLOSED:
> > -vhost_user_blk_disconnect(dev);
> > +/*
> > + * A close event may happen during a read/write, but vhost
> > + * code assumes the vhost_dev remains setup, so delay the
> > + * stop & clear. There are two possible paths to hit this
> > + * disconnect event:
> > + * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
> > + * vhost_user_blk_device_realize() is a caller.
> > + * 2. In tha main loop phase after VM start.
> > + *
> > + * For p2 the disconnect event will be delayed. We can't
> > + * do the same for p1, because we are not running the loop
> > + * at this moment. So just skip this step and perform
> > + * disconnect in the caller function.
> > + *
> > + * TODO: maybe it is a good idea to make the same fix
> > + * for other vhost-user devices.
> > + */
> > +if (runstate_is_running()) {
> > +AioContext *ctx = qemu_get_current_aio_context();
> > +
> > +qemu_chr_fe_set_handlers(>chardev, NULL, NULL, NULL, NULL,
> > +NULL, NULL, false);
> > +aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh,
> > opaque);
> > +}
> > break;
> > case CHR_EVENT_BREAK:
> > case CHR_EVENT_MUX_IN:
> > --
> > 2.7.4
> >
> >
[PATCH v4 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
A socket write during vhost-user communication may trigger a disconnect
event, calling vhost_user_blk_disconnect() and clearing all the
vhost_dev structures holding data that vhost-user functions expect to
remain valid to roll back initialization correctly. Delay the cleanup to
keep vhost_dev structure valid.
There are two possible states to handle:
1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
the caller routine.
2. RUN_STATE_RUNNING: delay by using bh
BH changes are based on the similar changes for the vhost-user-net
device:
commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
"vhost-user: delay vhost_user_stop"
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 38 +-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 9d8c0b3..76838e7 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -349,6 +349,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
vhost_dev_cleanup(>dev);
}
+static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
+
+static void vhost_user_blk_chr_closed_bh(void *opaque)
+{
+DeviceState *dev = opaque;
+VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+VHostUserBlk *s = VHOST_USER_BLK(vdev);
+
+vhost_user_blk_disconnect(dev);
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
+NULL, opaque, NULL, true);
+}
+
static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
{
DeviceState *dev = opaque;
@@ -363,7 +376,30 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
}
break;
case CHR_EVENT_CLOSED:
-vhost_user_blk_disconnect(dev);
+/*
+ * A close event may happen during a read/write, but vhost
+ * code assumes the vhost_dev remains setup, so delay the
+ * stop & clear. There are two possible paths to hit this
+ * disconnect event:
+ * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
+ * vhost_user_blk_device_realize() is a caller.
+ * 2. In tha main loop phase after VM start.
+ *
+ * For p2 the disconnect event will be delayed. We can't
+ * do the same for p1, because we are not running the loop
+ * at this moment. So just skip this step and perform
+ * disconnect in the caller function.
+ *
+ * TODO: maybe it is a good idea to make the same fix
+ * for other vhost-user devices.
+ */
+if (runstate_is_running()) {
+AioContext *ctx = qemu_get_current_aio_context();
+
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, NULL, NULL,
+NULL, NULL, false);
+aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
+}
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
--
2.7.4
[PATCH v4 0/2] vhost-user reconnect issues during vhost initialization
Changes is v4: - Update the "[PATCH v4 2/2] vhost-user-blk: delay vhost_user_blk_disconnect" patch based on Raphael's comment and Li Feng previous commit: https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg02255.html Don't change the vhost_user_blk_device_realize() function. Update the comment for the CHR_EVENT_CLOSED event. Changes in v3: - "[PATCH v3 1/2] char-socket: return -1 in case of disconnect during tcp_chr_write" made a small cleanup suggested by Li Feng. Added "Reviewed-by: Marc-André Lureau" - Rework the vhost_user_blk_disconnect call logic to delay it. - Remove the migration patch from the patch set, since we are still having some discussion about it. In general the current idea is good, but need to make some more investigation of how to handle reconnect during migration properly Changes in v2: - Add to CC list: Li Feng , since it looks like that we are working on pretty similar issues - Remove [RFC PATCH v1 1/7] contrib/vhost-user-blk: add option to simulate disconnect on init. Going to send this functionality in the separate patch, with the LIBVHOST_USER_DEBUG rework. Need to think how to reuse this option and silence the messages first. - Remove [RFC PATCH v1 3/7] char-socket: initialize reconnect timer only if close is emitted. This will be handled in the separate patchset: [PATCH 3/4] char-socket: avoid double call tcp_chr_free_connection by Li Feng v1: During vhost-user reconnect functionality we hit several issues, if vhost-user-blk daemon is "crashed" or made disconnect during vhost initialization. The general scenario is as follows: - vhost start routine is called - vhost write failed due to SIGPIPE - this call the disconnect routine and vhost_dev_cleanup routine which set to 0 all the field of the vhost_dev structure - return back to vhost start routine with the error - on the fail path vhost start routine tries to rollback the changes by using vhost_dev struct fields which were already reset - sometimes this leads to SIGSEGV, sometimes to SIGABRT Before revising the vhost-user initialization code, we suggest adding the sanity checks to be aware of the possible disconnect event and that the vhost_dev structure can be in "uninitialized" state. The vhost-user-blk daemon is updated with the additional "--simulate-disconnect-stage=CASENUM" argument to simulate disconnect during VHOST device initialization. For instance: 1. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=1 This command will simulate disconnect in the SET_VRING_CALL handler. In this case the vhost device in QEMU is not set the started field to true. 2. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=2 This command will simulate disconnect in the SET_VRING_NUM handler. In this case the started field is set to true. These two cases test different QEMU parts. Also to trigger different code paths disconnect should be simulated in two ways: - before any successful initialization - make successful initialization once and try to simulate disconnects Also we catch SIGABRT on the migration start if vhost-user daemon disconnected during vhost-user set log commands communication. Dima Stepanov (2): char-socket: return -1 in case of disconnect during tcp_chr_write vhost-user-blk: delay vhost_user_blk_disconnect chardev/char-socket.c | 7 --- hw/block/vhost-user-blk.c | 38 +- 2 files changed, 41 insertions(+), 4 deletions(-) -- 2.7.4
[PATCH v4 1/2] char-socket: return -1 in case of disconnect during tcp_chr_write
During testing of the vhost-user-blk reconnect functionality the qemu
SIGSEGV was triggered:
start qemu as:
x86_64-softmmu/qemu-system-x86_64 -m 1024M -M q35 \
-object
memory-backend-file,id=ram-node0,size=1024M,mem-path=/dev/shm/qemu,share=on \
-numa node,cpus=0,memdev=ram-node0 \
-chardev socket,id=chardev0,path=./vhost.sock,noserver,reconnect=1 \
-device vhost-user-blk-pci,chardev=chardev0,num-queues=4 --enable-kvm
start vhost-user-blk daemon:
./vhost-user-blk -s ./vhost.sock -b test-img.raw
If vhost-user-blk will be killed during the vhost initialization
process, for instance after getting VHOST_SET_VRING_CALL command, then
QEMU will fail with the following backtrace:
Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
0x559272bb in vhost_user_read (dev=0x7fffef2d53e0, msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
260 CharBackend *chr = u->user->chr;
#0 0x559272bb in vhost_user_read (dev=0x7fffef2d53e0,
msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
#1 0x5592acb8 in vhost_user_get_config (dev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost-user.c:1645
#2 0x55925525 in vhost_dev_get_config (hdev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost.c:1490
#3 0x558cc46b in vhost_user_blk_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd8f0)
at ./hw/block/vhost-user-blk.c:429
#4 0x55920090 in virtio_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd948)
at ./hw/virtio/virtio.c:3615
#5 0x55a9779c in device_set_realized (obj=0x7fffef2d51a0, value=true,
errp=0x7fffdb88)
at ./hw/core/qdev.c:891
...
The problem is that vhost_user_write doesn't get an error after
disconnect and try to call vhost_user_read(). The tcp_chr_write()
routine should return -1 in case of disconnect. Indicate the EIO error
if this routine is called in the disconnected state.
Signed-off-by: Dima Stepanov
Reviewed-by: Marc-André Lureau
---
chardev/char-socket.c | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index 232e0a8..c2462e0 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -174,15 +174,16 @@ static int tcp_chr_write(Chardev *chr, const uint8_t
*buf, int len)
if (ret < 0 && errno != EAGAIN) {
if (tcp_chr_read_poll(chr) <= 0) {
+/* Perform disconnect and return error. */
tcp_chr_disconnect_locked(chr);
-return len;
} /* else let the read handler finish it properly */
}
return ret;
} else {
-/* XXX: indicate an error ? */
-return len;
+/* Indicate an error. */
+errno = EIO;
+return -1;
}
}
--
2.7.4
Re: [PATCH v3 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
On Wed, May 20, 2020 at 06:53:13PM +0300, Dima Stepanov wrote:
> A socket write during vhost-user communication may trigger a disconnect
> event, calling vhost_user_blk_disconnect() and clearing all the
> vhost_dev structures holding data that vhost-user functions expect to
> remain valid to roll back initialization correctly. Delay the cleanup to
> keep vhost_dev structure valid.
> There are two possible states to handle:
> 1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
> the caller routine.
> 2. RUN_STATE_RUNNING: delay by using bh
>
> BH changes are based on the similar changes for the vhost-user-net
> device:
> commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
> "vhost-user: delay vhost_user_stop"
>
> Signed-off-by: Dima Stepanov
I've reworked the patch based on Raphael's comment and send it for
review. Also i added a TODO comment in the vhost_user_blk_event()
routine. After review i'll send a v4 patch set.
---
hw/block/vhost-user-blk.c | 38 +-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 9d8c0b3..76838e7 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -349,6 +349,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
vhost_dev_cleanup(>dev);
}
+static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
+
+static void vhost_user_blk_chr_closed_bh(void *opaque)
+{
+DeviceState *dev = opaque;
+VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+VHostUserBlk *s = VHOST_USER_BLK(vdev);
+
+vhost_user_blk_disconnect(dev);
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
+NULL, opaque, NULL, true);
+}
+
static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
{
DeviceState *dev = opaque;
@@ -363,7 +376,30 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
}
break;
case CHR_EVENT_CLOSED:
-vhost_user_blk_disconnect(dev);
+/*
+ * A close event may happen during a read/write, but vhost
+ * code assumes the vhost_dev remains setup, so delay the
+ * stop & clear. There are two possible paths to hit this
+ * disconnect event:
+ * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
+ * vhost_user_blk_device_realize() is a caller.
+ * 2. In tha main loop phase after VM start.
+ *
+ * For p2 the disconnect event will be delayed. We can't
+ * do the same for p1, because we are not running the loop
+ * at this moment. So just skip this step and perform
+ * disconnect in the caller function.
+ *
+ * TODO: maybe it is a good idea to make the same fix
+ * for other vhost-user devices.
+ */
+if (runstate_is_running()) {
+AioContext *ctx = qemu_get_current_aio_context();
+
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, NULL, NULL,
+NULL, NULL, false);
+aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
+}
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
--
2.7.4
Re: [PATCH v3 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
On Mon, May 25, 2020 at 12:00:10AM -0400, Raphael Norwitz wrote:
> I'm mostly happy with this. A couple comments.
>
> On Wed, May 20, 2020 at 11:54 AM Dima Stepanov
> wrote:
> >
> > A socket write during vhost-user communication may trigger a disconnect
> > event, calling vhost_user_blk_disconnect() and clearing all the
> > vhost_dev structures holding data that vhost-user functions expect to
> > remain valid to roll back initialization correctly. Delay the cleanup to
> > keep vhost_dev structure valid.
> > There are two possible states to handle:
> > 1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
> > the caller routine.
> > 2. RUN_STATE_RUNNING: delay by using bh
> >
> > BH changes are based on the similar changes for the vhost-user-net
> > device:
> > commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
> > "vhost-user: delay vhost_user_stop"
> >
> I'd also give credit to Li Feng here - he sent a similar patch:
>
> https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg02255.html
Yes, thanks for pointing me to it.
>
> > Signed-off-by: Dima Stepanov
> > ---
> > hw/block/vhost-user-blk.c | 49
> > +--
> > 1 file changed, 43 insertions(+), 6 deletions(-)
> >
> > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > index 9d8c0b3..447fc9c 100644
> > --- a/hw/block/vhost-user-blk.c
> > +++ b/hw/block/vhost-user-blk.c
> > @@ -337,11 +337,6 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > VHostUserBlk *s = VHOST_USER_BLK(vdev);
> >
> > -if (!s->connected) {
> > -return;
> > -}
> > -s->connected = false;
> > -
> > if (s->dev.started) {
> > vhost_user_blk_stop(vdev);
> > }
> > @@ -349,6 +344,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > vhost_dev_cleanup(>dev);
> > }
> >
> > +static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
> > +
> > +static void vhost_user_blk_chr_closed_bh(void *opaque)
> > +{
> > +DeviceState *dev = opaque;
> > +VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > +VHostUserBlk *s = VHOST_USER_BLK(vdev);
> > +
> > +vhost_user_blk_disconnect(dev);
> > +qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
> > +NULL, opaque, NULL, true);
> > +}
> > +
> > static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
> > {
> > DeviceState *dev = opaque;
> > @@ -363,7 +371,28 @@ static void vhost_user_blk_event(void *opaque,
> > QEMUChrEvent event)
> > }
> > break;
> > case CHR_EVENT_CLOSED:
> > -vhost_user_blk_disconnect(dev);
> > +/*
> > + * A close event may happen during a read/write, but vhost
> > + * code assumes the vhost_dev remains setup, so delay the
> > + * stop & clear. There are two possible paths to hit this
> > + * disconnect event:
> > + * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
> > + * vhost_user_blk_device_realize() is a caller.
> > + * 2. In tha main loop phase after VM start.
> > + *
> > + * For p2 the disconnect event will be delayed. We can't
> > + * do the same for p1, because we are not running the loop
> > + * at this moment. So just skip this step and perform
> > + * disconnect in the caller function.
> > + */
> > +if (s->connected && runstate_is_running()) {
> > +AioContext *ctx = qemu_get_current_aio_context();
> > +
> > +qemu_chr_fe_set_handlers(>chardev, NULL, NULL, NULL, NULL,
> > +NULL, NULL, false);
> > +aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh,
> > opaque);
> > +}
> > +s->connected = false;
> > break;
> > case CHR_EVENT_BREAK:
> > case CHR_EVENT_MUX_IN:
> > @@ -428,6 +457,14 @@ reconnect:
> >
> > ret = vhost_dev_get_config(>dev, (uint8_t *)>blkcfg,
> > sizeof(struct virtio_blk_config));
>
> I find checking s->connected before ret a little confusing. I think we
> should also enforce a reconnect if s->connected is false. AFIK if
> s->connected is false, ret must also be less than 0, but to be safe
> I’d prefer
Re: [PATCH v3 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
On Mon, May 25, 2020 at 11:31:16AM +0800, Jason Wang wrote:
>
> On 2020/5/20 下午11:53, Dima Stepanov wrote:
> >A socket write during vhost-user communication may trigger a disconnect
> >event, calling vhost_user_blk_disconnect() and clearing all the
> >vhost_dev structures holding data that vhost-user functions expect to
> >remain valid to roll back initialization correctly. Delay the cleanup to
> >keep vhost_dev structure valid.
> >There are two possible states to handle:
> >1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
> >the caller routine.
> >2. RUN_STATE_RUNNING: delay by using bh
> >
> >BH changes are based on the similar changes for the vhost-user-net
> >device:
> > commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
> > "vhost-user: delay vhost_user_stop"
>
>
> It's better to explain why we don't need to deal with case 1 in the
> vhost-user-net case.
In general i believe we should have similar change for all the
vhost-user devices. But i don't have a tests for it right now. So as we
discussed in v2 e-mail thread, i decided to stick only with the
vhost-user-blk changes. Also it seems to me that the problem is a little
bit wider, we have the changes like:
- only vhost-user-net: e7c83a885f865128ae3cf1946f8cb538b63cbfba
"vhost-user: delay vhost_user_stop"
- only vhost-user-blk: 9d283f85d755285bf1b1bfcb1ab275239dbf2c7b
"fix vhost_user_blk_watch crash"
- only vhost-user-blk: my change
At least what i knew (maybe more, because i can miss smth). So maybe
this "vhost_user_event()" routine should be generic for all vhost-user
devices with the internal method calls to specific devices. I want to
try making a rework for this, but don't want to do it in this patch set.
Because it will require more investigation and testing, since more
devices will be touched during refactoring.
>
> And do we still need patch 1 if we had this patch??
Yes, we still need it. The first patch is about getting an error from
the low level to the upper initialization error. So for example when we
call smth like get_config() and failed because of disconnect, we will stop
initialization. Without patch 1 we will try to call next initialization
function and such behaviour looks incorrect.
Thanks, Dima.
No other comments mixed in below.
>
> Thanks
>
>
> >
> >Signed-off-by: Dima Stepanov
> >---
> > hw/block/vhost-user-blk.c | 49
> > +--
> > 1 file changed, 43 insertions(+), 6 deletions(-)
> >
> >diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> >index 9d8c0b3..447fc9c 100644
> >--- a/hw/block/vhost-user-blk.c
> >+++ b/hw/block/vhost-user-blk.c
> >@@ -337,11 +337,6 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > VHostUserBlk *s = VHOST_USER_BLK(vdev);
> >-if (!s->connected) {
> >-return;
> >-}
> >-s->connected = false;
> >-
> > if (s->dev.started) {
> > vhost_user_blk_stop(vdev);
> > }
> >@@ -349,6 +344,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
> > vhost_dev_cleanup(>dev);
> > }
> >+static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
> >+
> >+static void vhost_user_blk_chr_closed_bh(void *opaque)
> >+{
> >+DeviceState *dev = opaque;
> >+VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> >+VHostUserBlk *s = VHOST_USER_BLK(vdev);
> >+
> >+vhost_user_blk_disconnect(dev);
> >+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
> >+NULL, opaque, NULL, true);
> >+}
> >+
> > static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
> > {
> > DeviceState *dev = opaque;
> >@@ -363,7 +371,28 @@ static void vhost_user_blk_event(void *opaque,
> >QEMUChrEvent event)
> > }
> > break;
> > case CHR_EVENT_CLOSED:
> >-vhost_user_blk_disconnect(dev);
> >+/*
> >+ * A close event may happen during a read/write, but vhost
> >+ * code assumes the vhost_dev remains setup, so delay the
> >+ * stop & clear. There are two possible paths to hit this
> >+ * disconnect event:
> >+ * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
> >+ * vhost_user_blk_device_realize() is a caller.
> >+ * 2. In tha main loop phase after VM start.
> >+ *
> >+ * For p2 the disconnect event will be delayed. We can't
> >+
[PATCH v3 1/2] char-socket: return -1 in case of disconnect during tcp_chr_write
During testing of the vhost-user-blk reconnect functionality the qemu
SIGSEGV was triggered:
start qemu as:
x86_64-softmmu/qemu-system-x86_64 -m 1024M -M q35 \
-object
memory-backend-file,id=ram-node0,size=1024M,mem-path=/dev/shm/qemu,share=on \
-numa node,cpus=0,memdev=ram-node0 \
-chardev socket,id=chardev0,path=./vhost.sock,noserver,reconnect=1 \
-device vhost-user-blk-pci,chardev=chardev0,num-queues=4 --enable-kvm
start vhost-user-blk daemon:
./vhost-user-blk -s ./vhost.sock -b test-img.raw
If vhost-user-blk will be killed during the vhost initialization
process, for instance after getting VHOST_SET_VRING_CALL command, then
QEMU will fail with the following backtrace:
Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
0x559272bb in vhost_user_read (dev=0x7fffef2d53e0, msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
260 CharBackend *chr = u->user->chr;
#0 0x559272bb in vhost_user_read (dev=0x7fffef2d53e0,
msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
#1 0x5592acb8 in vhost_user_get_config (dev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost-user.c:1645
#2 0x55925525 in vhost_dev_get_config (hdev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost.c:1490
#3 0x558cc46b in vhost_user_blk_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd8f0)
at ./hw/block/vhost-user-blk.c:429
#4 0x55920090 in virtio_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd948)
at ./hw/virtio/virtio.c:3615
#5 0x55a9779c in device_set_realized (obj=0x7fffef2d51a0, value=true,
errp=0x7fffdb88)
at ./hw/core/qdev.c:891
...
The problem is that vhost_user_write doesn't get an error after
disconnect and try to call vhost_user_read(). The tcp_chr_write()
routine should return -1 in case of disconnect. Indicate the EIO error
if this routine is called in the disconnected state.
Signed-off-by: Dima Stepanov
Reviewed-by: Marc-André Lureau
---
chardev/char-socket.c | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index 232e0a8..c2462e0 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -174,15 +174,16 @@ static int tcp_chr_write(Chardev *chr, const uint8_t
*buf, int len)
if (ret < 0 && errno != EAGAIN) {
if (tcp_chr_read_poll(chr) <= 0) {
+/* Perform disconnect and return error. */
tcp_chr_disconnect_locked(chr);
-return len;
} /* else let the read handler finish it properly */
}
return ret;
} else {
-/* XXX: indicate an error ? */
-return len;
+/* Indicate an error. */
+errno = EIO;
+return -1;
}
}
--
2.7.4
[PATCH v3 2/2] vhost-user-blk: delay vhost_user_blk_disconnect
A socket write during vhost-user communication may trigger a disconnect
event, calling vhost_user_blk_disconnect() and clearing all the
vhost_dev structures holding data that vhost-user functions expect to
remain valid to roll back initialization correctly. Delay the cleanup to
keep vhost_dev structure valid.
There are two possible states to handle:
1. RUN_STATE_PRELAUNCH: skip bh oneshot call and perform disconnect in
the caller routine.
2. RUN_STATE_RUNNING: delay by using bh
BH changes are based on the similar changes for the vhost-user-net
device:
commit e7c83a885f865128ae3cf1946f8cb538b63cbfba
"vhost-user: delay vhost_user_stop"
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 49 +--
1 file changed, 43 insertions(+), 6 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 9d8c0b3..447fc9c 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -337,11 +337,6 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
VirtIODevice *vdev = VIRTIO_DEVICE(dev);
VHostUserBlk *s = VHOST_USER_BLK(vdev);
-if (!s->connected) {
-return;
-}
-s->connected = false;
-
if (s->dev.started) {
vhost_user_blk_stop(vdev);
}
@@ -349,6 +344,19 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
vhost_dev_cleanup(>dev);
}
+static void vhost_user_blk_event(void *opaque, QEMUChrEvent event);
+
+static void vhost_user_blk_chr_closed_bh(void *opaque)
+{
+DeviceState *dev = opaque;
+VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+VHostUserBlk *s = VHOST_USER_BLK(vdev);
+
+vhost_user_blk_disconnect(dev);
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, vhost_user_blk_event,
+NULL, opaque, NULL, true);
+}
+
static void vhost_user_blk_event(void *opaque, QEMUChrEvent event)
{
DeviceState *dev = opaque;
@@ -363,7 +371,28 @@ static void vhost_user_blk_event(void *opaque,
QEMUChrEvent event)
}
break;
case CHR_EVENT_CLOSED:
-vhost_user_blk_disconnect(dev);
+/*
+ * A close event may happen during a read/write, but vhost
+ * code assumes the vhost_dev remains setup, so delay the
+ * stop & clear. There are two possible paths to hit this
+ * disconnect event:
+ * 1. When VM is in the RUN_STATE_PRELAUNCH state. The
+ * vhost_user_blk_device_realize() is a caller.
+ * 2. In tha main loop phase after VM start.
+ *
+ * For p2 the disconnect event will be delayed. We can't
+ * do the same for p1, because we are not running the loop
+ * at this moment. So just skip this step and perform
+ * disconnect in the caller function.
+ */
+if (s->connected && runstate_is_running()) {
+AioContext *ctx = qemu_get_current_aio_context();
+
+qemu_chr_fe_set_handlers(>chardev, NULL, NULL, NULL, NULL,
+NULL, NULL, false);
+aio_bh_schedule_oneshot(ctx, vhost_user_blk_chr_closed_bh, opaque);
+}
+s->connected = false;
break;
case CHR_EVENT_BREAK:
case CHR_EVENT_MUX_IN:
@@ -428,6 +457,14 @@ reconnect:
ret = vhost_dev_get_config(>dev, (uint8_t *)>blkcfg,
sizeof(struct virtio_blk_config));
+if (!s->connected) {
+/*
+ * Perform disconnect before making reconnect. More detailed
+ * comment why it was delayed is in the vhost_user_blk_event()
+ * routine.
+ */
+vhost_user_blk_disconnect(dev);
+}
if (ret < 0) {
error_report("vhost-user-blk: get block config failed");
goto reconnect;
--
2.7.4
[PATCH v3 0/2] vhost-user reconnect issues during vhost initialization
Changes in v3: - "[PATCH v3 1/2] char-socket: return -1 in case of disconnect during tcp_chr_write" made a small cleanup suggested by Li Feng. Added "Reviewed-by: Marc-André Lureau" - Rework the vhost_user_blk_disconnect call logic to delay it. - Remove the migration patch from the patch set, since we are still having some discussion about it. In general the current idea is good, but need to make some more investigation of how to handle reconnect during migration properly Changes in v2: - Add to CC list: Li Feng , since it looks like that we are working on pretty similar issues - Remove [RFC PATCH v1 1/7] contrib/vhost-user-blk: add option to simulate disconnect on init. Going to send this functionality in the separate patch, with the LIBVHOST_USER_DEBUG rework. Need to think how to reuse this option and silence the messages first. - Remove [RFC PATCH v1 3/7] char-socket: initialize reconnect timer only if close is emitted. This will be handled in the separate patchset: [PATCH 3/4] char-socket: avoid double call tcp_chr_free_connection by Li Feng v1: During vhost-user reconnect functionality we hit several issues, if vhost-user-blk daemon is "crashed" or made disconnect during vhost initialization. The general scenario is as follows: - vhost start routine is called - vhost write failed due to SIGPIPE - this call the disconnect routine and vhost_dev_cleanup routine which set to 0 all the field of the vhost_dev structure - return back to vhost start routine with the error - on the fail path vhost start routine tries to rollback the changes by using vhost_dev struct fields which were already reset - sometimes this leads to SIGSEGV, sometimes to SIGABRT Before revising the vhost-user initialization code, we suggest adding the sanity checks to be aware of the possible disconnect event and that the vhost_dev structure can be in "uninitialized" state. The vhost-user-blk daemon is updated with the additional "--simulate-disconnect-stage=CASENUM" argument to simulate disconnect during VHOST device initialization. For instance: 1. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=1 This command will simulate disconnect in the SET_VRING_CALL handler. In this case the vhost device in QEMU is not set the started field to true. 2. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=2 This command will simulate disconnect in the SET_VRING_NUM handler. In this case the started field is set to true. These two cases test different QEMU parts. Also to trigger different code paths disconnect should be simulated in two ways: - before any successful initialization - make successful initialization once and try to simulate disconnects Also we catch SIGABRT on the migration start if vhost-user daemon disconnected during vhost-user set log commands communication. Dima Stepanov (2): char-socket: return -1 in case of disconnect during tcp_chr_write vhost-user-blk: delay vhost_user_blk_disconnect chardev/char-socket.c | 7 --- hw/block/vhost-user-blk.c | 49 +-- 2 files changed, 47 insertions(+), 9 deletions(-) -- 2.7.4
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Wed, May 13, 2020 at 01:56:18PM +0800, Jason Wang wrote:
>
> On 2020/5/13 下午12:15, Michael S. Tsirkin wrote:
> >On Tue, May 12, 2020 at 12:35:30PM +0300, Dima Stepanov wrote:
> >>On Tue, May 12, 2020 at 11:32:50AM +0800, Jason Wang wrote:
> >>>On 2020/5/11 下午5:25, Dima Stepanov wrote:
> >>>>On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> >>>>>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>>>>If vhost-user daemon is used as a backend for the vhost device, then we
> >>>>>>should consider a possibility of disconnect at any moment. If such
> >>>>>>disconnect happened in the vhost_migration_log() routine the vhost
> >>>>>>device structure will be clean up.
> >>>>>>At the start of the vhost_migration_log() function there is a check:
> >>>>>> if (!dev->started) {
> >>>>>> dev->log_enabled = enable;
> >>>>>> return 0;
> >>>>>> }
> >>>>>>To be consistent with this check add the same check after calling the
> >>>>>>vhost_dev_set_log() routine. This in general help not to break a
> >>>>>>migration due the assert() message. But it looks like that this code
> >>>>>>should be revised to handle these errors more carefully.
> >>>>>>
> >>>>>>In case of vhost-user device backend the fail paths should consider the
> >>>>>>state of the device. In this case we should skip some function calls
> >>>>>>during rollback on the error paths, so not to get the NULL dereference
> >>>>>>errors.
> >>>>>>
> >>>>>>Signed-off-by: Dima Stepanov
> >>>>>>---
> >>>>>> hw/virtio/vhost.c | 39 +++
> >>>>>> 1 file changed, 35 insertions(+), 4 deletions(-)
> >>>>>>
> >>>>>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>>>>index 3ee50c4..d5ab96d 100644
> >>>>>>--- a/hw/virtio/vhost.c
> >>>>>>+++ b/hw/virtio/vhost.c
> >>>>>>@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> >>>>>>*dev,
> >>>>>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>>>>> {
> >>>>>> int r, i, idx;
> >>>>>>+
> >>>>>>+if (!dev->started) {
> >>>>>>+/*
> >>>>>>+ * If vhost-user daemon is used as a backend for the
> >>>>>>+ * device and the connection is broken, then the vhost_dev
> >>>>>>+ * structure will be reset all its values to 0.
> >>>>>>+ * Add additional check for the device state.
> >>>>>>+ */
> >>>>>>+return -1;
> >>>>>>+}
> >>>>>>+
> >>>>>> r = vhost_dev_set_features(dev, enable_log);
> >>>>>> if (r < 0) {
> >>>>>> goto err_features;
> >>>>>>@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev
> >>>>>>*dev, bool enable_log)
> >>>>>> }
> >>>>>> return 0;
> >>>>>> err_vq:
> >>>>>>-for (; i >= 0; --i) {
> >>>>>>+/*
> >>>>>>+ * Disconnect with the vhost-user daemon can lead to the
> >>>>>>+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >>>>>>+ * structure.
> >>>>>>+ */
> >>>>>>+for (; dev->started && (i >= 0); --i) {
> >>>>>> idx = dev->vhost_ops->vhost_get_vq_index(
> >>>>>Why need the check of dev->started here, can started be modified outside
> >>>>>mainloop? If yes, I don't get the check of !dev->started in the
> >>>>>beginning of
> >>>>>this function.
> >>>>>
> >>>>No dev->started can't change outside the mainloop. The main problem is
> >>>>only for the vhost_user_blk daemon. Consider the case when we
> >>>>successfully pass the dev->started check at the beginning of the
> >>>
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Mon, May 18, 2020 at 10:53:59AM +0100, Dr. David Alan Gilbert wrote:
> * Dima Stepanov (dimas...@yandex-team.ru) wrote:
> > On Mon, May 18, 2020 at 10:50:39AM +0800, Jason Wang wrote:
> > >
> > > On 2020/5/16 上午12:54, Dima Stepanov wrote:
> > > >On Thu, May 14, 2020 at 03:34:24PM +0800, Jason Wang wrote:
> > > >>On 2020/5/13 下午5:47, Dima Stepanov wrote:
> > > >>>>> case CHR_EVENT_CLOSED:
> > > >>>>> /* a close event may happen during a read/write, but vhost
> > > >>>>> * code assumes the vhost_dev remains setup, so delay the
> > > >>>>> * stop & clear to idle.
> > > >>>>> * FIXME: better handle failure in vhost code, remove bh
> > > >>>>> */
> > > >>>>> if (s->watch) {
> > > >>>>> AioContext *ctx = qemu_get_current_aio_context();
> > > >>>>>
> > > >>>>> g_source_remove(s->watch);
> > > >>>>> s->watch = 0;
> > > >>>>> qemu_chr_fe_set_handlers(>chr, NULL, NULL, NULL,
> > > >>>>> NULL,
> > > >>>>> NULL, NULL, false);
> > > >>>>>
> > > >>>>> aio_bh_schedule_oneshot(ctx, chr_closed_bh, opaque);
> > > >>>>> }
> > > >>>>> break;
> > > >>>>>
> > > >>>>>I think it's time we dropped the FIXME and moved the handling to
> > > >>>>>common
> > > >>>>>code. Jason? Marc-André?
> > > >>>>I agree. Just to confirm, do you prefer bh or doing changes like what
> > > >>>>is
> > > >>>>done in this series? It looks to me bh can have more easier codes.
> > > >>>Could it be a good idea just to make disconnect in the char device but
> > > >>>postphone clean up in the vhost-user-blk (or any other vhost-user
> > > >>>device) itself? So we are moving the postphone logic and decision from
> > > >>>the char device to vhost-user device. One of the idea i have is as
> > > >>>follows:
> > > >>> - Put ourself in the INITIALIZATION state
> > > >>> - Start these vhost-user "handshake" commands
> > > >>> - If we got a disconnect error, perform disconnect, but don't clean
> > > >>> up
> > > >>> device (it will be clean up on the roll back). I can be done by
> > > >>> checking the state in vhost_user_..._disconnect routine or smth
> > > >>> like it
> > > >>
> > > >>Any issue you saw just using the aio bh as Michael posted above.
> > > >>
> > > >>Then we don't need to deal with the silent vhost_dev_stop() and we will
> > > >>have
> > > >>codes that is much more easier to understand.
> > > >I've implemented this solution inside
> > > >hw/block/vhost-user-blk.c:vhost_user_blk_event() in the similar way by
> > > >using the s->connected field. Looks good and more correct fix ). I have
> > > >two questions here before i'll rework the fixes:
> > > >1. Is it okay to make the similar fix inside vhost_user_blk_event() or
> > > >we are looking for more generic vhost-user solution? What do you think?
> > >
> > >
> > > I think I agree with Michael, it's better to have a generic vhost-user
> > > solution. But if it turns out to be not easy, we can start from fixing
> > > vhost-user-blk.
> > I also agree, but as i see it right now the connect/disconnect events
> > are handled inside each vhost-user device implementation file. So it will
> > need some global refactoring. So i suggest having this fix first and
> > after it refactoring the code:
> > - more devices will be involved
> > - i see there is some difference in device handling
>
> I'm following bits of this discussion, some thoughts;
> if your device doesn't support reconnect, then if, at the start of
> migration you find that you can't start the log what is the correct
> behaviour?
I'm not sure here, but it looks like that in this case the device state
will be:
disconnect -> stopped (will not be changed during migration, because
reconnect isn't supported)
And because of it
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Mon, May 18, 2020 at 10:50:39AM +0800, Jason Wang wrote:
>
> On 2020/5/16 上午12:54, Dima Stepanov wrote:
> >On Thu, May 14, 2020 at 03:34:24PM +0800, Jason Wang wrote:
> >>On 2020/5/13 下午5:47, Dima Stepanov wrote:
> >>>>> case CHR_EVENT_CLOSED:
> >>>>> /* a close event may happen during a read/write, but vhost
> >>>>> * code assumes the vhost_dev remains setup, so delay the
> >>>>> * stop & clear to idle.
> >>>>> * FIXME: better handle failure in vhost code, remove bh
> >>>>> */
> >>>>> if (s->watch) {
> >>>>> AioContext *ctx = qemu_get_current_aio_context();
> >>>>>
> >>>>> g_source_remove(s->watch);
> >>>>> s->watch = 0;
> >>>>> qemu_chr_fe_set_handlers(>chr, NULL, NULL, NULL, NULL,
> >>>>> NULL, NULL, false);
> >>>>>
> >>>>> aio_bh_schedule_oneshot(ctx, chr_closed_bh, opaque);
> >>>>> }
> >>>>> break;
> >>>>>
> >>>>>I think it's time we dropped the FIXME and moved the handling to common
> >>>>>code. Jason? Marc-André?
> >>>>I agree. Just to confirm, do you prefer bh or doing changes like what is
> >>>>done in this series? It looks to me bh can have more easier codes.
> >>>Could it be a good idea just to make disconnect in the char device but
> >>>postphone clean up in the vhost-user-blk (or any other vhost-user
> >>>device) itself? So we are moving the postphone logic and decision from
> >>>the char device to vhost-user device. One of the idea i have is as
> >>>follows:
> >>> - Put ourself in the INITIALIZATION state
> >>> - Start these vhost-user "handshake" commands
> >>> - If we got a disconnect error, perform disconnect, but don't clean up
> >>> device (it will be clean up on the roll back). I can be done by
> >>> checking the state in vhost_user_..._disconnect routine or smth like
> >>> it
> >>
> >>Any issue you saw just using the aio bh as Michael posted above.
> >>
> >>Then we don't need to deal with the silent vhost_dev_stop() and we will have
> >>codes that is much more easier to understand.
> >I've implemented this solution inside
> >hw/block/vhost-user-blk.c:vhost_user_blk_event() in the similar way by
> >using the s->connected field. Looks good and more correct fix ). I have
> >two questions here before i'll rework the fixes:
> >1. Is it okay to make the similar fix inside vhost_user_blk_event() or
> >we are looking for more generic vhost-user solution? What do you think?
>
>
> I think I agree with Michael, it's better to have a generic vhost-user
> solution. But if it turns out to be not easy, we can start from fixing
> vhost-user-blk.
I also agree, but as i see it right now the connect/disconnect events
are handled inside each vhost-user device implementation file. So it will
need some global refactoring. So i suggest having this fix first and
after it refactoring the code:
- more devices will be involved
- i see there is some difference in device handling
>
>
> >2. For migration we require an additional information that for the
> >vhost-user device it isn't an error, because i'm trigerring the
> >following assert error:
> > Core was generated by `x86_64-softmmu/qemu-system-x86_64 -nodefaults
> > -no-user-config -M q35,sata=false'.
> > Program terminated with signal SIGABRT, Aborted.
> > #0 0x7fb56e729428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> > [Current thread is 1 (Thread 0x7fb486ef5700 (LWP 527734))]
> >
> > (gdb) bt
> > #0 0x7fb56e729428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> > #1 0x7fb56e72b02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
> > #2 0x5648ea376ee6 in vhost_log_global_start
> > (listener=0x5648ece4eb08) at ./hw/virtio/vhost.c:857
> > #3 0x5648ea2dde7e in memory_global_dirty_log_start ()
> > at ./memory.c:2611
> > #4 0x5648ea2e68e7 in ram_init_bitmaps (rs=0x7fb4740008c0)
> > at ./migration/ram.c:2305
> > #5 0x5648ea2e698b in ram_init_all (rsp=0x5648eb1f0f20 )
> > at ./migration/ram.c:2323
> > #6 0x5648ea2e6cc5 in ram_save_setup (f=0x5648ec609e00,
> > opaque=0x5648eb1f
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Mon, May 18, 2020 at 10:52:08AM +0800, Jason Wang wrote:
>
> On 2020/5/16 上午11:20, Li Feng wrote:
> >Hi, Dima.
> >This abort is what I have mentioned in my previous email.
> >I have triggered this crash without any fix a week ago.
> >And I have written a test patch to let vhost_log_global_start return
> >int and propagate the error to up layer.
> >However, my change is a little large, because the origin callback
> >return void, and don't do some rollback.
> >After test, the migration could migrate to dst successfully, and fio
> >is still running perfectly, but the src vm is still stuck here, no
> >crash.
> >
> >Is it right to return this error to the up layer?
>
>
> That could be a solution or we may ask David for more suggestion.
>
> Another thing that might be useful is to block re connection during
> migration.
I've written a little more information as answer to Feng's mail. But
what if add some new callback to get the device started state (started or not).
And for the vhost-user (or at least vhost-usr-blk) devices it will use
the connected field also to return the device state:
- disconnect -> not started
For other devices we can just return the started field value as it is
right now.
No other comments mixed in below.
>
> Thanks
>
>
> >
> >Thanks,
> >Feng Li
> >
> >Dima Stepanov 于2020年5月16日周六 上午12:55写道：
> >>On Thu, May 14, 2020 at 03:34:24PM +0800, Jason Wang wrote:
> >>>On 2020/5/13 下午5:47, Dima Stepanov wrote:
> >>>>>> case CHR_EVENT_CLOSED:
> >>>>>> /* a close event may happen during a read/write, but vhost
> >>>>>> * code assumes the vhost_dev remains setup, so delay the
> >>>>>> * stop & clear to idle.
> >>>>>> * FIXME: better handle failure in vhost code, remove bh
> >>>>>> */
> >>>>>> if (s->watch) {
> >>>>>> AioContext *ctx = qemu_get_current_aio_context();
> >>>>>>
> >>>>>> g_source_remove(s->watch);
> >>>>>> s->watch = 0;
> >>>>>> qemu_chr_fe_set_handlers(>chr, NULL, NULL, NULL, NULL,
> >>>>>> NULL, NULL, false);
> >>>>>>
> >>>>>> aio_bh_schedule_oneshot(ctx, chr_closed_bh, opaque);
> >>>>>> }
> >>>>>> break;
> >>>>>>
> >>>>>>I think it's time we dropped the FIXME and moved the handling to common
> >>>>>>code. Jason? Marc-André?
> >>>>>I agree. Just to confirm, do you prefer bh or doing changes like what is
> >>>>>done in this series? It looks to me bh can have more easier codes.
> >>>>Could it be a good idea just to make disconnect in the char device but
> >>>>postphone clean up in the vhost-user-blk (or any other vhost-user
> >>>>device) itself? So we are moving the postphone logic and decision from
> >>>>the char device to vhost-user device. One of the idea i have is as
> >>>>follows:
> >>>> - Put ourself in the INITIALIZATION state
> >>>> - Start these vhost-user "handshake" commands
> >>>> - If we got a disconnect error, perform disconnect, but don't clean up
> >>>> device (it will be clean up on the roll back). I can be done by
> >>>> checking the state in vhost_user_..._disconnect routine or smth like
> >>>> it
> >>>
> >>>Any issue you saw just using the aio bh as Michael posted above.
> >>>
> >>>Then we don't need to deal with the silent vhost_dev_stop() and we will
> >>>have
> >>>codes that is much more easier to understand.
> >>I've implemented this solution inside
> >>hw/block/vhost-user-blk.c:vhost_user_blk_event() in the similar way by
> >>using the s->connected field. Looks good and more correct fix ). I have
> >>two questions here before i'll rework the fixes:
> >>1. Is it okay to make the similar fix inside vhost_user_blk_event() or
> >>we are looking for more generic vhost-user solution? What do you think?
> >>2. For migration we require an additional information that for the
> >>vhost-user device it isn't an error, because i'm trigerring the
> >>following assert error:
> >> Core was generated by `x86_64-softmmu/qemu-s
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Sat, May 16, 2020 at 11:20:03AM +0800, Li Feng wrote:
> Hi, Dima.
> This abort is what I have mentioned in my previous email.
Yes, i understood it and this abort() message was fixed by the previous
patch. But since we try new postphone approach this patch isn't working
and we need to get the device state somehow:
- vhost-user disconnect => device not started
> I have triggered this crash without any fix a week ago.
> And I have written a test patch to let vhost_log_global_start return
> int and propagate the error to up layer.
> However, my change is a little large, because the origin callback
> return void, and don't do some rollback.
> After test, the migration could migrate to dst successfully, and fio
> is still running perfectly, but the src vm is still stuck here, no
> crash.
>
> Is it right to return this error to the up layer?
Well it is the question we talk about with you, i'm also not sure. I can
only summarize some of the statements i used:
- device state: not started -> is okay for migration
- device state: vhost-user disconnect, this is the same as "not
started" -> is okay for migration
- at least my internal migration tests passed
So my idea for the fix here is smth like:
Add device callback, for instance vhost_dev_started() which will
return device state. And for the vhost-user device (or at least
vhost-user-blk) device this callback will consider the connected field
and return true or false.
As a result vhost_migration_log() will check device state at the start
of the routine and before return.
But if the disconnect state isn't okay for migration, then we should
return an error.
No other comments mixed in below.
>
> Thanks,
> Feng Li
>
> Dima Stepanov 于2020年5月16日周六 上午12:55写道：
> >
> > On Thu, May 14, 2020 at 03:34:24PM +0800, Jason Wang wrote:
> > >
> > > On 2020/5/13 下午5:47, Dima Stepanov wrote:
> > > >>> case CHR_EVENT_CLOSED:
> > > >>> /* a close event may happen during a read/write, but vhost
> > > >>> * code assumes the vhost_dev remains setup, so delay the
> > > >>> * stop & clear to idle.
> > > >>> * FIXME: better handle failure in vhost code, remove bh
> > > >>> */
> > > >>> if (s->watch) {
> > > >>> AioContext *ctx = qemu_get_current_aio_context();
> > > >>>
> > > >>> g_source_remove(s->watch);
> > > >>> s->watch = 0;
> > > >>> qemu_chr_fe_set_handlers(>chr, NULL, NULL, NULL, NULL,
> > > >>> NULL, NULL, false);
> > > >>>
> > > >>> aio_bh_schedule_oneshot(ctx, chr_closed_bh, opaque);
> > > >>> }
> > > >>> break;
> > > >>>
> > > >>>I think it's time we dropped the FIXME and moved the handling to common
> > > >>>code. Jason? Marc-André?
> > > >>I agree. Just to confirm, do you prefer bh or doing changes like what is
> > > >>done in this series? It looks to me bh can have more easier codes.
> > > >Could it be a good idea just to make disconnect in the char device but
> > > >postphone clean up in the vhost-user-blk (or any other vhost-user
> > > >device) itself? So we are moving the postphone logic and decision from
> > > >the char device to vhost-user device. One of the idea i have is as
> > > >follows:
> > > > - Put ourself in the INITIALIZATION state
> > > > - Start these vhost-user "handshake" commands
> > > > - If we got a disconnect error, perform disconnect, but don't clean up
> > > > device (it will be clean up on the roll back). I can be done by
> > > > checking the state in vhost_user_..._disconnect routine or smth
> > > > like it
> > >
> > >
> > > Any issue you saw just using the aio bh as Michael posted above.
> > >
> > > Then we don't need to deal with the silent vhost_dev_stop() and we will
> > > have
> > > codes that is much more easier to understand.
> > I've implemented this solution inside
> > hw/block/vhost-user-blk.c:vhost_user_blk_event() in the similar way by
> > using the s->connected field. Looks good and more correct fix ). I have
> > two questions here before i'll rework the fixes:
> > 1. Is it okay to make the similar fix inside vhost_user_blk_event() or
> > we are looking for more generic vhost-user solution?
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Thu, May 14, 2020 at 03:34:24PM +0800, Jason Wang wrote:
>
> On 2020/5/13 下午5:47, Dima Stepanov wrote:
> >>> case CHR_EVENT_CLOSED:
> >>> /* a close event may happen during a read/write, but vhost
> >>> * code assumes the vhost_dev remains setup, so delay the
> >>> * stop & clear to idle.
> >>> * FIXME: better handle failure in vhost code, remove bh
> >>> */
> >>> if (s->watch) {
> >>> AioContext *ctx = qemu_get_current_aio_context();
> >>>
> >>> g_source_remove(s->watch);
> >>> s->watch = 0;
> >>> qemu_chr_fe_set_handlers(>chr, NULL, NULL, NULL, NULL,
> >>> NULL, NULL, false);
> >>>
> >>> aio_bh_schedule_oneshot(ctx, chr_closed_bh, opaque);
> >>> }
> >>> break;
> >>>
> >>>I think it's time we dropped the FIXME and moved the handling to common
> >>>code. Jason? Marc-André?
> >>I agree. Just to confirm, do you prefer bh or doing changes like what is
> >>done in this series? It looks to me bh can have more easier codes.
> >Could it be a good idea just to make disconnect in the char device but
> >postphone clean up in the vhost-user-blk (or any other vhost-user
> >device) itself? So we are moving the postphone logic and decision from
> >the char device to vhost-user device. One of the idea i have is as
> >follows:
> > - Put ourself in the INITIALIZATION state
> > - Start these vhost-user "handshake" commands
> > - If we got a disconnect error, perform disconnect, but don't clean up
> > device (it will be clean up on the roll back). I can be done by
> > checking the state in vhost_user_..._disconnect routine or smth like it
>
>
> Any issue you saw just using the aio bh as Michael posted above.
>
> Then we don't need to deal with the silent vhost_dev_stop() and we will have
> codes that is much more easier to understand.
I've implemented this solution inside
hw/block/vhost-user-blk.c:vhost_user_blk_event() in the similar way by
using the s->connected field. Looks good and more correct fix ). I have
two questions here before i'll rework the fixes:
1. Is it okay to make the similar fix inside vhost_user_blk_event() or
we are looking for more generic vhost-user solution? What do you think?
2. For migration we require an additional information that for the
vhost-user device it isn't an error, because i'm trigerring the
following assert error:
Core was generated by `x86_64-softmmu/qemu-system-x86_64 -nodefaults
-no-user-config -M q35,sata=false'.
Program terminated with signal SIGABRT, Aborted.
#0 0x7fb56e729428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fb486ef5700 (LWP 527734))]
(gdb) bt
#0 0x7fb56e729428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x7fb56e72b02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x5648ea376ee6 in vhost_log_global_start
(listener=0x5648ece4eb08) at ./hw/virtio/vhost.c:857
#3 0x5648ea2dde7e in memory_global_dirty_log_start ()
at ./memory.c:2611
#4 0x5648ea2e68e7 in ram_init_bitmaps (rs=0x7fb4740008c0)
at ./migration/ram.c:2305
#5 0x5648ea2e698b in ram_init_all (rsp=0x5648eb1f0f20 )
at ./migration/ram.c:2323
#6 0x5648ea2e6cc5 in ram_save_setup (f=0x5648ec609e00,
opaque=0x5648eb1f0f20 )
at ./migration/ram.c:2436
#7 0x5648ea67b7d3 in qemu_savevm_state_setup (f=0x5648ec609e00) at
migration/savevm.c:1176
#8 0x5648ea674511 in migration_thread (opaque=0x5648ec031ff0) at
migration/migration.c:3416
#9 0x5648ea85d65d in qemu_thread_start (args=0x5648ec6057f0) at
util/qemu-thread-posix.c:519
#10 0x7fb56eac56ba in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7fb56e7fb41d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) frame 2
#2 0x5648ea376ee6 in vhost_log_global_start
(listener=0x5648ece4eb08) at ./hw/virtio/vhost.c:857
857 abort();
(gdb) list
852 {
853 int r;
854
855 r = vhost_migration_log(listener, true);
856 if (r < 0) {
857 abort();
858 }
859 }
860
861 static void vhost_log_global_stop(MemoryListener *listener)
Since bh postphone the clean up, we can't use the ->started field.
Do we have any mechanism to get the device type/state in the common
vhost_migration_log() routine? So for example for the vhost-user/disconnect
device we will be able to return 0. Or should we implement it and introduce
it in this patch set?
Thanks, Dima.
>
> Thank
>
>
> > - vhost-user command returns error back to the _start() routine
> > - Rollback in one place in the start() routine, by calling this
> > postphoned clean up for the disconnect
> >
>
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Wed, May 13, 2020 at 01:56:18PM +0800, Jason Wang wrote:
>
> On 2020/5/13 下午12:15, Michael S. Tsirkin wrote:
> >On Tue, May 12, 2020 at 12:35:30PM +0300, Dima Stepanov wrote:
> >>On Tue, May 12, 2020 at 11:32:50AM +0800, Jason Wang wrote:
> >>>On 2020/5/11 下午5:25, Dima Stepanov wrote:
> >>>>On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> >>>>>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>>>>If vhost-user daemon is used as a backend for the vhost device, then we
> >>>>>>should consider a possibility of disconnect at any moment. If such
> >>>>>>disconnect happened in the vhost_migration_log() routine the vhost
> >>>>>>device structure will be clean up.
> >>>>>>At the start of the vhost_migration_log() function there is a check:
> >>>>>> if (!dev->started) {
> >>>>>> dev->log_enabled = enable;
> >>>>>> return 0;
> >>>>>> }
> >>>>>>To be consistent with this check add the same check after calling the
> >>>>>>vhost_dev_set_log() routine. This in general help not to break a
> >>>>>>migration due the assert() message. But it looks like that this code
> >>>>>>should be revised to handle these errors more carefully.
> >>>>>>
> >>>>>>In case of vhost-user device backend the fail paths should consider the
> >>>>>>state of the device. In this case we should skip some function calls
> >>>>>>during rollback on the error paths, so not to get the NULL dereference
> >>>>>>errors.
> >>>>>>
> >>>>>>Signed-off-by: Dima Stepanov
> >>>>>>---
> >>>>>> hw/virtio/vhost.c | 39 +++
> >>>>>> 1 file changed, 35 insertions(+), 4 deletions(-)
> >>>>>>
> >>>>>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>>>>index 3ee50c4..d5ab96d 100644
> >>>>>>--- a/hw/virtio/vhost.c
> >>>>>>+++ b/hw/virtio/vhost.c
> >>>>>>@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> >>>>>>*dev,
> >>>>>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>>>>> {
> >>>>>> int r, i, idx;
> >>>>>>+
> >>>>>>+if (!dev->started) {
> >>>>>>+/*
> >>>>>>+ * If vhost-user daemon is used as a backend for the
> >>>>>>+ * device and the connection is broken, then the vhost_dev
> >>>>>>+ * structure will be reset all its values to 0.
> >>>>>>+ * Add additional check for the device state.
> >>>>>>+ */
> >>>>>>+return -1;
> >>>>>>+}
> >>>>>>+
> >>>>>> r = vhost_dev_set_features(dev, enable_log);
> >>>>>> if (r < 0) {
> >>>>>> goto err_features;
> >>>>>>@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev
> >>>>>>*dev, bool enable_log)
> >>>>>> }
> >>>>>> return 0;
> >>>>>> err_vq:
> >>>>>>-for (; i >= 0; --i) {
> >>>>>>+/*
> >>>>>>+ * Disconnect with the vhost-user daemon can lead to the
> >>>>>>+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >>>>>>+ * structure.
> >>>>>>+ */
> >>>>>>+for (; dev->started && (i >= 0); --i) {
> >>>>>> idx = dev->vhost_ops->vhost_get_vq_index(
> >>>>>Why need the check of dev->started here, can started be modified outside
> >>>>>mainloop? If yes, I don't get the check of !dev->started in the
> >>>>>beginning of
> >>>>>this function.
> >>>>>
> >>>>No dev->started can't change outside the mainloop. The main problem is
> >>>>only for the vhost_user_blk daemon. Consider the case when we
> >>>>successfully pass the dev->started check at the beginning of the
> >>>
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Wed, May 13, 2020 at 11:20:50AM +0800, Jason Wang wrote:
>
> On 2020/5/12 下午5:35, Dima Stepanov wrote:
> >On Tue, May 12, 2020 at 11:32:50AM +0800, Jason Wang wrote:
> >>On 2020/5/11 下午5:25, Dima Stepanov wrote:
> >>>On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> >>>>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>>>If vhost-user daemon is used as a backend for the vhost device, then we
> >>>>>should consider a possibility of disconnect at any moment. If such
> >>>>>disconnect happened in the vhost_migration_log() routine the vhost
> >>>>>device structure will be clean up.
> >>>>>At the start of the vhost_migration_log() function there is a check:
> >>>>> if (!dev->started) {
> >>>>> dev->log_enabled = enable;
> >>>>> return 0;
> >>>>> }
> >>>>>To be consistent with this check add the same check after calling the
> >>>>>vhost_dev_set_log() routine. This in general help not to break a
> >>>>>migration due the assert() message. But it looks like that this code
> >>>>>should be revised to handle these errors more carefully.
> >>>>>
> >>>>>In case of vhost-user device backend the fail paths should consider the
> >>>>>state of the device. In this case we should skip some function calls
> >>>>>during rollback on the error paths, so not to get the NULL dereference
> >>>>>errors.
> >>>>>
> >>>>>Signed-off-by: Dima Stepanov
> >>>>>---
> >>>>> hw/virtio/vhost.c | 39 +++
> >>>>> 1 file changed, 35 insertions(+), 4 deletions(-)
> >>>>>
> >>>>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>>>index 3ee50c4..d5ab96d 100644
> >>>>>--- a/hw/virtio/vhost.c
> >>>>>+++ b/hw/virtio/vhost.c
> >>>>>@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> >>>>>*dev,
> >>>>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>>>> {
> >>>>> int r, i, idx;
> >>>>>+
> >>>>>+if (!dev->started) {
> >>>>>+/*
> >>>>>+ * If vhost-user daemon is used as a backend for the
> >>>>>+ * device and the connection is broken, then the vhost_dev
> >>>>>+ * structure will be reset all its values to 0.
> >>>>>+ * Add additional check for the device state.
> >>>>>+ */
> >>>>>+return -1;
> >>>>>+}
> >>>>>+
> >>>>> r = vhost_dev_set_features(dev, enable_log);
> >>>>> if (r < 0) {
> >>>>> goto err_features;
> >>>>>@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev
> >>>>>*dev, bool enable_log)
> >>>>> }
> >>>>> return 0;
> >>>>> err_vq:
> >>>>>-for (; i >= 0; --i) {
> >>>>>+/*
> >>>>>+ * Disconnect with the vhost-user daemon can lead to the
> >>>>>+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >>>>>+ * structure.
> >>>>>+ */
> >>>>>+for (; dev->started && (i >= 0); --i) {
> >>>>> idx = dev->vhost_ops->vhost_get_vq_index(
> >>>>Why need the check of dev->started here, can started be modified outside
> >>>>mainloop? If yes, I don't get the check of !dev->started in the beginning
> >>>>of
> >>>>this function.
> >>>>
> >>>No dev->started can't change outside the mainloop. The main problem is
> >>>only for the vhost_user_blk daemon. Consider the case when we
> >>>successfully pass the dev->started check at the beginning of the
> >>>function, but after it we hit the disconnect on the next call on the
> >>>second or third iteration:
> >>> r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx, enable_log);
> >>>The unix socket backend device will call the disconnect routine for this
> >>>device and reset the structure. So the structure will be
Re: [PATCH v2 4/5] vhost: check vring address before calling unmap
On Wed, May 13, 2020 at 11:00:38AM +0800, Jason Wang wrote:
>
> On 2020/5/12 下午5:08, Dima Stepanov wrote:
> >On Tue, May 12, 2020 at 11:26:11AM +0800, Jason Wang wrote:
> >>On 2020/5/11 下午5:11, Dima Stepanov wrote:
> >>>On Mon, May 11, 2020 at 11:05:58AM +0800, Jason Wang wrote:
> >>>>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>>>Since disconnect can happen at any time during initialization not all
> >>>>>vring buffers (for instance used vring) can be intialized successfully.
> >>>>>If the buffer was not initialized then vhost_memory_unmap call will lead
> >>>>>to SIGSEGV. Add checks for the vring address value before calling unmap.
> >>>>>Also add assert() in the vhost_memory_unmap() routine.
> >>>>>
> >>>>>Signed-off-by: Dima Stepanov
> >>>>>---
> >>>>> hw/virtio/vhost.c | 27 +--
> >>>>> 1 file changed, 21 insertions(+), 6 deletions(-)
> >>>>>
> >>>>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>>>index ddbdc53..3ee50c4 100644
> >>>>>--- a/hw/virtio/vhost.c
> >>>>>+++ b/hw/virtio/vhost.c
> >>>>>@@ -314,6 +314,8 @@ static void vhost_memory_unmap(struct vhost_dev
> >>>>>*dev, void *buffer,
> >>>>> hwaddr len, int is_write,
> >>>>> hwaddr access_len)
> >>>>> {
> >>>>>+assert(buffer);
> >>>>>+
> >>>>> if (!vhost_dev_has_iommu(dev)) {
> >>>>> cpu_physical_memory_unmap(buffer, len, is_write, access_len);
> >>>>> }
> >>>>>@@ -1132,12 +1134,25 @@ static void vhost_virtqueue_stop(struct
> >>>>>vhost_dev *dev,
> >>>>> vhost_vq_index);
> >>>>> }
> >>>>>-vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev,
> >>>>>idx),
> >>>>>- 1, virtio_queue_get_used_size(vdev, idx));
> >>>>>-vhost_memory_unmap(dev, vq->avail,
> >>>>>virtio_queue_get_avail_size(vdev, idx),
> >>>>>- 0, virtio_queue_get_avail_size(vdev, idx));
> >>>>>-vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev,
> >>>>>idx),
> >>>>>- 0, virtio_queue_get_desc_size(vdev, idx));
> >>>>>+/*
> >>>>>+ * Since the vhost-user disconnect can happen during initialization
> >>>>>+ * check if vring was initialized, before making unmap.
> >>>>>+ */
> >>>>>+if (vq->used) {
> >>>>>+vhost_memory_unmap(dev, vq->used,
> >>>>>+ virtio_queue_get_used_size(vdev, idx),
> >>>>>+ 1, virtio_queue_get_used_size(vdev, idx));
> >>>>>+}
> >>>>>+if (vq->avail) {
> >>>>>+vhost_memory_unmap(dev, vq->avail,
> >>>>>+ virtio_queue_get_avail_size(vdev, idx),
> >>>>>+ 0, virtio_queue_get_avail_size(vdev, idx));
> >>>>>+}
> >>>>>+if (vq->desc) {
> >>>>>+vhost_memory_unmap(dev, vq->desc,
> >>>>>+ virtio_queue_get_desc_size(vdev, idx),
> >>>>>+ 0, virtio_queue_get_desc_size(vdev, idx));
> >>>>>+}
> >>>>Any reason not checking hdev->started instead? vhost_dev_start() will set
> >>>>it
> >>>>to true if virtqueues were correctly mapped.
> >>>>
> >>>>Thanks
> >>>Well i see it a little bit different:
> >>> - vhost_dev_start() sets hdev->started to true before starting
> >>>virtqueues
> >>> - vhost_virtqueue_start() maps all the memory
> >>>If we hit the vhost disconnect at the start of the
> >>>vhost_virtqueue_start(), for instance for this call:
> >>> r = dev->vhost_ops->vhost_set_vring_base(dev, );
> >>>Then we will call vhost_user_blk_disconnect:
> >>> vhost_user_blk_disconnect()->
> >>>
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Tue, May 12, 2020 at 11:32:50AM +0800, Jason Wang wrote:
>
> On 2020/5/11 下午5:25, Dima Stepanov wrote:
> >On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> >>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>If vhost-user daemon is used as a backend for the vhost device, then we
> >>>should consider a possibility of disconnect at any moment. If such
> >>>disconnect happened in the vhost_migration_log() routine the vhost
> >>>device structure will be clean up.
> >>>At the start of the vhost_migration_log() function there is a check:
> >>> if (!dev->started) {
> >>> dev->log_enabled = enable;
> >>> return 0;
> >>> }
> >>>To be consistent with this check add the same check after calling the
> >>>vhost_dev_set_log() routine. This in general help not to break a
> >>>migration due the assert() message. But it looks like that this code
> >>>should be revised to handle these errors more carefully.
> >>>
> >>>In case of vhost-user device backend the fail paths should consider the
> >>>state of the device. In this case we should skip some function calls
> >>>during rollback on the error paths, so not to get the NULL dereference
> >>>errors.
> >>>
> >>>Signed-off-by: Dima Stepanov
> >>>---
> >>> hw/virtio/vhost.c | 39 +++
> >>> 1 file changed, 35 insertions(+), 4 deletions(-)
> >>>
> >>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>index 3ee50c4..d5ab96d 100644
> >>>--- a/hw/virtio/vhost.c
> >>>+++ b/hw/virtio/vhost.c
> >>>@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> >>>*dev,
> >>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>> {
> >>> int r, i, idx;
> >>>+
> >>>+if (!dev->started) {
> >>>+/*
> >>>+ * If vhost-user daemon is used as a backend for the
> >>>+ * device and the connection is broken, then the vhost_dev
> >>>+ * structure will be reset all its values to 0.
> >>>+ * Add additional check for the device state.
> >>>+ */
> >>>+return -1;
> >>>+}
> >>>+
> >>> r = vhost_dev_set_features(dev, enable_log);
> >>> if (r < 0) {
> >>> goto err_features;
> >>>@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev *dev,
> >>>bool enable_log)
> >>> }
> >>> return 0;
> >>> err_vq:
> >>>-for (; i >= 0; --i) {
> >>>+/*
> >>>+ * Disconnect with the vhost-user daemon can lead to the
> >>>+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >>>+ * structure.
> >>>+ */
> >>>+for (; dev->started && (i >= 0); --i) {
> >>> idx = dev->vhost_ops->vhost_get_vq_index(
> >>
> >>Why need the check of dev->started here, can started be modified outside
> >>mainloop? If yes, I don't get the check of !dev->started in the beginning of
> >>this function.
> >>
> >No dev->started can't change outside the mainloop. The main problem is
> >only for the vhost_user_blk daemon. Consider the case when we
> >successfully pass the dev->started check at the beginning of the
> >function, but after it we hit the disconnect on the next call on the
> >second or third iteration:
> > r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx, enable_log);
> >The unix socket backend device will call the disconnect routine for this
> >device and reset the structure. So the structure will be reset (and
> >dev->started set to false) inside this set_addr() call.
>
>
> I still don't get here. I think the disconnect can not happen in the middle
> of vhost_dev_set_log() since both of them were running in mainloop. And even
> if it can, we probably need other synchronization mechanism other than
> simple check here.
Disconnect isn't happened in the separate thread it is happened in this
routine inside vhost_dev_set_log. When for instance vhost_user_write()
call failed:
vhost_user_set_log_base()
vhost_user_write()
vhost_user_blk_disconnect()
vhost_dev_cleanup()
vhost_user_backend_cleanup()
So the point is that if we somehow got a disconnect with the
vhos
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Tue, May 12, 2020 at 11:47:34AM +0800, Li Feng wrote:
> Hi, Dima.
>
> If vhost_migration_log return < 0, then vhost_log_global_start will
> trigger a crash.
> Does your patch have process this abort?
> If a disconnect happens in the migration stage, the correct operation
> is to stop the migration, right?
>
> 841 static void vhost_log_global_start(MemoryListener *listener)
> 842 {
> 843 int r;
> 844
> 845 r = vhost_migration_log(listener, true);
> 846 if (r < 0) {
> 847 abort();
> 848 }
> 849 }
Yes, my patch process it by not returning an error ). That is one of the
point we've talked about with Raphael and Michael in this thread. First
of all in my patches i'm still following the same logic which has been
already in upstream ./hw/virtio/vhost.c:vhost_migration_log():
...
820 if (!dev->started) {
821 dev->log_enabled = enable;
822 return 0;
823 }
...
It means, that if device not started, then continue migration without
returning any error. So i followed the same logic, if we got a
disconnect, then it will mean that device isn't started and we can
continue migration. As a result no error is returned and assert() isn't
hit.
Also there is a question from Raphael to Michael about it you can find
it in this thread, by i will add it also:
> Subject: Re: [PATCH v2 5/5] vhost: add device started check in
> migration set log
> On Wed, May 06, 2020 at 06:08:34PM -0400, Raphael Norwitz wrote:
>> In particular, we need to decide whether a migration should be
>> allowed to continue if a device disconnects durning the migration
>> stage.
>>
>> mst, any thoughts?
> Why not? It can't change state while disconnected, so it just makes
> things easier.
So it looks like a correct way to handle it. Also our internal tests
passed. Some words about our tests:
- run src VM with vhost-usr-blk daemon used
- run fio inside it
- perform reconnect every X seconds (just kill and restart
daemon), X is random
- run dst VM
- perform migration
- fio should complete in dst VM
And we cycle this test like forever. At least for now we see no new
issues.
No other comments mixed in below.
>
> Thanks,
>
> Feng Li
>
> Jason Wang 于2020年5月12日周二 上午11:33写道：
> >
> >
> > On 2020/5/11 下午5:25, Dima Stepanov wrote:
> > > On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> > >> On 2020/4/30 下午9:36, Dima Stepanov wrote:
> > >>> If vhost-user daemon is used as a backend for the vhost device, then we
> > >>> should consider a possibility of disconnect at any moment. If such
> > >>> disconnect happened in the vhost_migration_log() routine the vhost
> > >>> device structure will be clean up.
> > >>> At the start of the vhost_migration_log() function there is a check:
> > >>>if (!dev->started) {
> > >>>dev->log_enabled = enable;
> > >>>return 0;
> > >>>}
> > >>> To be consistent with this check add the same check after calling the
> > >>> vhost_dev_set_log() routine. This in general help not to break a
> > >>> migration due the assert() message. But it looks like that this code
> > >>> should be revised to handle these errors more carefully.
> > >>>
> > >>> In case of vhost-user device backend the fail paths should consider the
> > >>> state of the device. In this case we should skip some function calls
> > >>> during rollback on the error paths, so not to get the NULL dereference
> > >>> errors.
> > >>>
> > >>> Signed-off-by: Dima Stepanov
> > >>> ---
> > >>> hw/virtio/vhost.c | 39 +++
> > >>> 1 file changed, 35 insertions(+), 4 deletions(-)
> > >>>
> > >>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > >>> index 3ee50c4..d5ab96d 100644
> > >>> --- a/hw/virtio/vhost.c
> > >>> +++ b/hw/virtio/vhost.c
> > >>> @@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> > >>> *dev,
> > >>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> > >>> {
> > >>> int r, i, idx;
> > >>> +
> > >>> +if (!dev->started) {
> > >>> +/*
> > >>> + * If vhost-user daemon is used as a backend for the
> > >>> + * device and the connection is broken, then the vhost_dev
> > >&
Re: [PATCH v2 4/5] vhost: check vring address before calling unmap
On Tue, May 12, 2020 at 11:26:11AM +0800, Jason Wang wrote:
>
> On 2020/5/11 下午5:11, Dima Stepanov wrote:
> >On Mon, May 11, 2020 at 11:05:58AM +0800, Jason Wang wrote:
> >>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>Since disconnect can happen at any time during initialization not all
> >>>vring buffers (for instance used vring) can be intialized successfully.
> >>>If the buffer was not initialized then vhost_memory_unmap call will lead
> >>>to SIGSEGV. Add checks for the vring address value before calling unmap.
> >>>Also add assert() in the vhost_memory_unmap() routine.
> >>>
> >>>Signed-off-by: Dima Stepanov
> >>>---
> >>> hw/virtio/vhost.c | 27 +--
> >>> 1 file changed, 21 insertions(+), 6 deletions(-)
> >>>
> >>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>index ddbdc53..3ee50c4 100644
> >>>--- a/hw/virtio/vhost.c
> >>>+++ b/hw/virtio/vhost.c
> >>>@@ -314,6 +314,8 @@ static void vhost_memory_unmap(struct vhost_dev *dev,
> >>>void *buffer,
> >>> hwaddr len, int is_write,
> >>> hwaddr access_len)
> >>> {
> >>>+assert(buffer);
> >>>+
> >>> if (!vhost_dev_has_iommu(dev)) {
> >>> cpu_physical_memory_unmap(buffer, len, is_write, access_len);
> >>> }
> >>>@@ -1132,12 +1134,25 @@ static void vhost_virtqueue_stop(struct vhost_dev
> >>>*dev,
> >>> vhost_vq_index);
> >>> }
> >>>-vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev,
> >>>idx),
> >>>- 1, virtio_queue_get_used_size(vdev, idx));
> >>>-vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev,
> >>>idx),
> >>>- 0, virtio_queue_get_avail_size(vdev, idx));
> >>>-vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev,
> >>>idx),
> >>>- 0, virtio_queue_get_desc_size(vdev, idx));
> >>>+/*
> >>>+ * Since the vhost-user disconnect can happen during initialization
> >>>+ * check if vring was initialized, before making unmap.
> >>>+ */
> >>>+if (vq->used) {
> >>>+vhost_memory_unmap(dev, vq->used,
> >>>+ virtio_queue_get_used_size(vdev, idx),
> >>>+ 1, virtio_queue_get_used_size(vdev, idx));
> >>>+}
> >>>+if (vq->avail) {
> >>>+vhost_memory_unmap(dev, vq->avail,
> >>>+ virtio_queue_get_avail_size(vdev, idx),
> >>>+ 0, virtio_queue_get_avail_size(vdev, idx));
> >>>+}
> >>>+if (vq->desc) {
> >>>+vhost_memory_unmap(dev, vq->desc,
> >>>+ virtio_queue_get_desc_size(vdev, idx),
> >>>+ 0, virtio_queue_get_desc_size(vdev, idx));
> >>>+}
> >>
> >>Any reason not checking hdev->started instead? vhost_dev_start() will set it
> >>to true if virtqueues were correctly mapped.
> >>
> >>Thanks
> >Well i see it a little bit different:
> > - vhost_dev_start() sets hdev->started to true before starting
> >virtqueues
> > - vhost_virtqueue_start() maps all the memory
> >If we hit the vhost disconnect at the start of the
> >vhost_virtqueue_start(), for instance for this call:
> > r = dev->vhost_ops->vhost_set_vring_base(dev, );
> >Then we will call vhost_user_blk_disconnect:
> > vhost_user_blk_disconnect()->
> > vhost_user_blk_stop()->
> > vhost_dev_stop()->
> > vhost_virtqueue_stop()
> >As a result we will come in this routine with the hdev->started still
> >set to true, but if used/avail/desc fields still uninitialized and set
> >to 0.
>
>
> I may miss something, but consider both vhost_dev_start() and
> vhost_user_blk_disconnect() were serialized in main loop. Can this really
> happen?
Yes, consider the case when we start the vhost-user-blk device:
vhost_dev_start->
vhost_virtqueue_start
And we got a disconnect in the middle of vhost_virtqueue_start()
routine, for instance:
1000 vq->num = state.num = virtio_queue_get_num
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Sun, May 10, 2020 at 08:03:39PM -0400, Raphael Norwitz wrote:
> On Thu, May 7, 2020 at 11:35 AM Dima Stepanov wrote:
> >
> > What do you think?
> >
>
> Apologies - I tripped over the if (dev->started && r < 0) check.
> Never-mind my point with race conditions and failing migrations.
>
> Rather than modifying vhost_dev_set_log(), it may be clearer to put a
> check after vhost_dev_log_resize()? Something like:
>
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -829,11 +829,22 @@ static int vhost_migration_log(MemoryListener
> *listener, int enable)
> vhost_log_put(dev, false);
> } else {
> vhost_dev_log_resize(dev, vhost_get_log_size(dev));
> +/*
> + * A device can be stopped because of backend disconnect inside
> + * vhost_dev_log_resize(). In this case we should mark logging
> + * enabled and return without attempting to set the backend
> + * logging state.
> + */
> +if (!dev->started) {
> +goto out_success;
> +}
> r = vhost_dev_set_log(dev, true);
> if (r < 0) {
> return r;
> }
> }
> +
> +out_success:
> dev->log_enabled = enable;
> return 0;
> }
This patch will not fix all the issues. Consider the case than you will
hit disconnect inside vhost_dev_set_log. For instance for the 3rd
virtqueue, for the following call:
vhost_virtqueue_set_addr(...)
Maybe i didn't explain very clearly the problem. The problem i've tried
to fix is only for the vhost-user-blk devices. This issue can be hit
during VHOST_USER commands "handshake". If we hit disconnect on any step
of this "handshake" then we will try to make clean up twice:
1. First during disconnect cleanup (unix socket backend).
2. Second as roll back for initialization.
If this is the case, then we shouldn't call p2, as everything was clean
up on p1. And the complicated thing is that there are several VHOST_USER
commands and we should consider the state after each. And even more,
initialization could fail because of some other reason and we hit
disconnect inside roll back clean up, in this case we should complete
clean up in the disconnect function and stop rolling back.
Hope it helps ).
>
> This seems harmless enough to me, and I see how it fixes your
> particular crash, but I would still prefer we worked towards a more
> robust solution. In particular I think we could handle this inside
> vhost-user-blk if we let the device state persist between connections
> (i.e. call vhost_dev_cleanup() inside vhost_user_blk_connect() before
> vhost_dev_init() on reconnect). This should also fix some of the
> crashes Li Feng has hit, and probably others which haven’t been
> reported yet. What do you think?
Yes, this looks like a good direction. Because all my patches are only
workarounds and i believe there can be some other issues which haven't
been reported or will be introduced ).
I still think that these patches are good to submit and to think about
more complicated/refactoring solution as the next step.
>
> If that’s unworkable I guess we will need to add these vhost level
> checks.
At least for now, i don't think its unworkable, i just think that it
will take some time to figure out how to refactor it properly. But the
SIGSEGV issue is real.
> In that case I would still prefer we add a “disconnected” flag
> in struct vhost_dev struct, and make sure it isn’t cleared by
> vhost_dev_cleanup(). That way we don’t conflate stopping a device with
> backend disconnect at the vhost level and potentially regress behavior
> for other device types.
It is also possible, but should be analyzed and properly tested. So as i
said it will take some time to figure out how to refactor it properly.
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
>
> On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >If vhost-user daemon is used as a backend for the vhost device, then we
> >should consider a possibility of disconnect at any moment. If such
> >disconnect happened in the vhost_migration_log() routine the vhost
> >device structure will be clean up.
> >At the start of the vhost_migration_log() function there is a check:
> > if (!dev->started) {
> > dev->log_enabled = enable;
> > return 0;
> > }
> >To be consistent with this check add the same check after calling the
> >vhost_dev_set_log() routine. This in general help not to break a
> >migration due the assert() message. But it looks like that this code
> >should be revised to handle these errors more carefully.
> >
> >In case of vhost-user device backend the fail paths should consider the
> >state of the device. In this case we should skip some function calls
> >during rollback on the error paths, so not to get the NULL dereference
> >errors.
> >
> >Signed-off-by: Dima Stepanov
> >---
> > hw/virtio/vhost.c | 39 +++
> > 1 file changed, 35 insertions(+), 4 deletions(-)
> >
> >diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >index 3ee50c4..d5ab96d 100644
> >--- a/hw/virtio/vhost.c
> >+++ b/hw/virtio/vhost.c
> >@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
> > static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> > {
> > int r, i, idx;
> >+
> >+if (!dev->started) {
> >+/*
> >+ * If vhost-user daemon is used as a backend for the
> >+ * device and the connection is broken, then the vhost_dev
> >+ * structure will be reset all its values to 0.
> >+ * Add additional check for the device state.
> >+ */
> >+return -1;
> >+}
> >+
> > r = vhost_dev_set_features(dev, enable_log);
> > if (r < 0) {
> > goto err_features;
> >@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev *dev,
> >bool enable_log)
> > }
> > return 0;
> > err_vq:
> >-for (; i >= 0; --i) {
> >+/*
> >+ * Disconnect with the vhost-user daemon can lead to the
> >+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >+ * structure.
> >+ */
> >+for (; dev->started && (i >= 0); --i) {
> > idx = dev->vhost_ops->vhost_get_vq_index(
>
>
> Why need the check of dev->started here, can started be modified outside
> mainloop? If yes, I don't get the check of !dev->started in the beginning of
> this function.
>
No dev->started can't change outside the mainloop. The main problem is
only for the vhost_user_blk daemon. Consider the case when we
successfully pass the dev->started check at the beginning of the
function, but after it we hit the disconnect on the next call on the
second or third iteration:
r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx, enable_log);
The unix socket backend device will call the disconnect routine for this
device and reset the structure. So the structure will be reset (and
dev->started set to false) inside this set_addr() call. So
we shouldn't call the clean up calls because this virtqueues were clean
up in the disconnect call. But we should protect these calls somehow, so
it will not hit SIGSEGV and we will be able to pass migration.
Just to summarize it:
For the vhost-user-blk devices we ca hit clean up calls twice in case of
vhost disconnect:
1. The first time during the disconnect process. The clean up is called
inside it.
2. The second time during roll back clean up.
So if it is the case we should skip p2.
>
> >dev, dev->vq_index + i);
> > vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
> > dev->log_enabled);
> > }
> >-vhost_dev_set_features(dev, dev->log_enabled);
> >+if (dev->started) {
> >+vhost_dev_set_features(dev, dev->log_enabled);
> >+}
> > err_features:
> > return r;
> > }
> >@@ -832,7 +850,15 @@ static int vhost_migration_log(MemoryListener
> >*listener, int enable)
> > } else {
> > vhost_dev_log_resize(dev, vhost_get_log_size(dev));
> > r = vhost_dev_set_log(dev, true);
> >-if (r < 0) {
> >+/*
> >+ * The dev log resize can fail, because of disconnect
> >+ * with the vhost-user-blk daemon.
Re: [PATCH v2 4/5] vhost: check vring address before calling unmap
On Mon, May 11, 2020 at 11:05:58AM +0800, Jason Wang wrote:
>
> On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >Since disconnect can happen at any time during initialization not all
> >vring buffers (for instance used vring) can be intialized successfully.
> >If the buffer was not initialized then vhost_memory_unmap call will lead
> >to SIGSEGV. Add checks for the vring address value before calling unmap.
> >Also add assert() in the vhost_memory_unmap() routine.
> >
> >Signed-off-by: Dima Stepanov
> >---
> > hw/virtio/vhost.c | 27 +--
> > 1 file changed, 21 insertions(+), 6 deletions(-)
> >
> >diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >index ddbdc53..3ee50c4 100644
> >--- a/hw/virtio/vhost.c
> >+++ b/hw/virtio/vhost.c
> >@@ -314,6 +314,8 @@ static void vhost_memory_unmap(struct vhost_dev *dev,
> >void *buffer,
> > hwaddr len, int is_write,
> > hwaddr access_len)
> > {
> >+assert(buffer);
> >+
> > if (!vhost_dev_has_iommu(dev)) {
> > cpu_physical_memory_unmap(buffer, len, is_write, access_len);
> > }
> >@@ -1132,12 +1134,25 @@ static void vhost_virtqueue_stop(struct vhost_dev
> >*dev,
> > vhost_vq_index);
> > }
> >-vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
> >- 1, virtio_queue_get_used_size(vdev, idx));
> >-vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev,
> >idx),
> >- 0, virtio_queue_get_avail_size(vdev, idx));
> >-vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
> >- 0, virtio_queue_get_desc_size(vdev, idx));
> >+/*
> >+ * Since the vhost-user disconnect can happen during initialization
> >+ * check if vring was initialized, before making unmap.
> >+ */
> >+if (vq->used) {
> >+vhost_memory_unmap(dev, vq->used,
> >+ virtio_queue_get_used_size(vdev, idx),
> >+ 1, virtio_queue_get_used_size(vdev, idx));
> >+}
> >+if (vq->avail) {
> >+vhost_memory_unmap(dev, vq->avail,
> >+ virtio_queue_get_avail_size(vdev, idx),
> >+ 0, virtio_queue_get_avail_size(vdev, idx));
> >+}
> >+if (vq->desc) {
> >+vhost_memory_unmap(dev, vq->desc,
> >+ virtio_queue_get_desc_size(vdev, idx),
> >+ 0, virtio_queue_get_desc_size(vdev, idx));
> >+}
>
>
> Any reason not checking hdev->started instead? vhost_dev_start() will set it
> to true if virtqueues were correctly mapped.
>
> Thanks
Well i see it a little bit different:
- vhost_dev_start() sets hdev->started to true before starting
virtqueues
- vhost_virtqueue_start() maps all the memory
If we hit the vhost disconnect at the start of the
vhost_virtqueue_start(), for instance for this call:
r = dev->vhost_ops->vhost_set_vring_base(dev, );
Then we will call vhost_user_blk_disconnect:
vhost_user_blk_disconnect()->
vhost_user_blk_stop()->
vhost_dev_stop()->
vhost_virtqueue_stop()
As a result we will come in this routine with the hdev->started still
set to true, but if used/avail/desc fields still uninitialized and set
to 0.
>
>
> > }
> > static void vhost_eventfd_add(MemoryListener *listener,
>
Re: [PATCH v2 2/5] vhost: introduce wrappers to set guest notifiers for virtio device
On Mon, May 11, 2020 at 11:03:01AM +0800, Jason Wang wrote:
>
> On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >Introduce new wrappers to set/reset guest notifiers for the virtio
> >device in the vhost device module:
> > vhost_dev_assign_guest_notifiers
> > ->set_guest_notifiers(..., ..., true);
> > vhost_dev_drop_guest_notifiers
> > ->set_guest_notifiers(..., ..., false);
> >This is a preliminary step to refactor code,
>
>
> Maybe I miss something, I don't see any add-on patch to modify the new
> wrapper in this series?
Hi, in fact the next 3/5 patch:
"[PATCH v2 3/5] vhost-user-blk: add mechanism to track the guest
notifiers init state"
is about using these wrappers. But disregard it, i decided to follow
Raphael suggestion. So we will fix the vhost-user-blk case first, so i
will not introduce these wrappers. And the code will be more easier to
read and straightforward.
I will send v3 as soon as we decide what to do with the migration fix
in this patchset.
No other comments mixed in below.
>
>
> > so the set_guest_notifiers
> >methods could be called based on the vhost device state.
> >Update all vhost used devices to use these wrappers instead of direct
> >method call.
> >
> >Signed-off-by: Dima Stepanov
> >---
> > backends/cryptodev-vhost.c | 26 +++---
> > backends/vhost-user.c | 16 +---
> > hw/block/vhost-user-blk.c | 15 +--
> > hw/net/vhost_net.c | 30 +-
> > hw/scsi/vhost-scsi-common.c | 15 +--
> > hw/virtio/vhost-user-fs.c | 17 +++--
> > hw/virtio/vhost-vsock.c | 18 --
> > hw/virtio/vhost.c | 38 ++
> > hw/virtio/virtio.c | 13 +
> > include/hw/virtio/vhost.h | 4
> > include/hw/virtio/virtio.h | 1 +
> > 11 files changed, 118 insertions(+), 75 deletions(-)
> >
> >diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c
> >index 8337c9a..4522195 100644
> >--- a/backends/cryptodev-vhost.c
> >+++ b/backends/cryptodev-vhost.c
> >@@ -169,16 +169,13 @@ vhost_set_vring_enable(CryptoDevBackendClient *cc,
> > int cryptodev_vhost_start(VirtIODevice *dev, int total_queues)
> > {
> > VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
> >-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
> >-VirtioBusState *vbus = VIRTIO_BUS(qbus);
> >-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
> > int r, e;
> > int i;
> > CryptoDevBackend *b = vcrypto->cryptodev;
> > CryptoDevBackendVhost *vhost_crypto;
> > CryptoDevBackendClient *cc;
> >-if (!k->set_guest_notifiers) {
> >+if (!virtio_device_guest_notifiers_initialized(dev)) {
> > error_report("binding does not support guest notifiers");
> > return -ENOSYS;
> > }
> >@@ -198,9 +195,13 @@ int cryptodev_vhost_start(VirtIODevice *dev, int
> >total_queues)
> > }
> > }
> >-r = k->set_guest_notifiers(qbus->parent, total_queues, true);
> >+/*
> >+ * Since all the states are handled by one vhost device,
> >+ * use the first one in array.
> >+ */
> >+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
> >+r = vhost_dev_assign_guest_notifiers(_crypto->dev, dev,
> >total_queues);
> > if (r < 0) {
> >-error_report("error binding guest notifier: %d", -r);
> > goto err;
> > }
> >@@ -232,7 +233,8 @@ err_start:
> > vhost_crypto = cryptodev_get_vhost(cc, b, i);
> > cryptodev_vhost_stop_one(vhost_crypto, dev);
> > }
> >-e = k->set_guest_notifiers(qbus->parent, total_queues, false);
> >+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
> >+e = vhost_dev_drop_guest_notifiers(_crypto->dev, dev,
> >total_queues);
> > if (e < 0) {
> > error_report("vhost guest notifier cleanup failed: %d", e);
> > }
> >@@ -242,9 +244,6 @@ err:
> > void cryptodev_vhost_stop(VirtIODevice *dev, int total_queues)
> > {
> >-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
> >-VirtioBusState *vbus = VIRTIO_BUS(qbus);
> >-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
> > VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
> > CryptoDevBackend *b = vcrypto->cryptodev;
> > CryptoDevBackendVhost *vhost_crypto;
> >@@
Re: [PATCH v2 5/5] vhost: add device started check in migration set log
On Wed, May 06, 2020 at 06:08:34PM -0400, Raphael Norwitz wrote:
> As you correctly point out, this code needs to be looked at more
> carefully so that
> if the device does disconnect in the background we can handle the migration
> path
> gracefully. In particular, we need to decide whether a migration
> should be allowed
> to continue if a device disconnects durning the migration stage.
>From what i see from the code it is allowed. At the start of the
hw/virtio/vhost.c:vhost_migration_log() routine there is a check:
if (!dev->started) {
dev->log_enabled = enable;
return 0;
}
So our changes had the same idea. If device isn't started then 0 can be
returned. Please note, that if we want to return error here then the
following assert will be hit (hw/virtio/vhost.c)
static void vhost_log_global_start(MemoryListener *listener)
{
int r;
r = vhost_migration_log(listener, true);
if (r < 0) {
abort();
}
}
But as i mentioned we didn't change this logic, we just propogate it on
the whole migration start process during vhost handshake. After it our
tests passed successfully.
>
> mst, any thoughts?
>
> Have you looked at the suggestion I gave Li Feng to move vhost_dev_cleanup()
> into the connection path in vhost-user-blk? I’m not sure if he’s
> actively working on it,
> but I would prefer if we can find a way to keep some state around
> between reconnects
> so we aren’t constantly checking dev->started. A device can be stopped
> for reasons
> other than backend disconnect so I’d rather not reuse this field to
> check for backend
> disconnect failures.
In fact i didn't try to use >started field to signal about disconnect.
What i tried to follow is that if device not started (because of
disconnect or any other reason), there is no need to continue
initialization and we can proceed with the next migration step.
>
> On Thu, Apr 30, 2020 at 9:57 AM Dima Stepanov wrote:
> >
> > If vhost-user daemon is used as a backend for the vhost device, then we
> > should consider a possibility of disconnect at any moment. If such
> > disconnect happened in the vhost_migration_log() routine the vhost
> > device structure will be clean up.
> > At the start of the vhost_migration_log() function there is a check:
> > if (!dev->started) {
> > dev->log_enabled = enable;
> > return 0;
> > }
> > To be consistent with this check add the same check after calling the
> > vhost_dev_set_log() routine. This in general help not to break a
>
> Could you point to the specific asserts which are being triggered?
Just to be clear here. The assert message i mentioned is described
above. I wanted to explain why we followed the "(!dev->started) return 0"
logic. And in this case we didn't return error and return 0.
But the first error we hit during migration testing was SIGSEGV:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x56354db0a74a in vhost_dev_has_iommu (dev=0x563550562b00)
at hw/virtio/vhost.c:299
299 return vdev->dma_as != _space_memory &&
(gdb) p vdev
$1 = (VirtIODevice *) 0x0
(gdb) bt
#0 0x56354db0a74a in vhost_dev_has_iommu (dev=0x563550562b00)
at hw/virtio/vhost.c:299
#1 0x56354db0bb76 in vhost_dev_set_features (dev=0x563550562b00,
enable_log=true)
at hw/virtio/vhost.c:777
#2 0x56354db0bc1e in vhost_dev_set_log (dev=0x563550562b00,
enable_log=true)
at hw/virtio/vhost.c:790
#3 0x56354db0be58 in vhost_migration_log (listener=0x563550562b08,
enable=1)
at hw/virtio/vhost.c:834
#4 0x56354db0be9b in vhost_log_global_start (listener=0x563550562b08)
at hw/virtio/vhost.c:847
#5 0x56354da72e7e in memory_global_dirty_log_start ()
at memory.c:2611
...
>
> > migration due the assert() message. But it looks like that this code
> > should be revised to handle these errors more carefully.
> >
> > In case of vhost-user device backend the fail paths should consider the
> > state of the device. In this case we should skip some function calls
> > during rollback on the error paths, so not to get the NULL dereference
> > errors.
> >
> > Signed-off-by: Dima Stepanov
> > ---
> > hw/virtio/vhost.c | 39 +++
> > 1 file changed, 35 insertions(+), 4 deletions(-)
> >
> > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > index 3ee50c4..d5ab96d 100644
> > --- a/hw/virtio/vhost.c
> > +++ b/hw/virtio/vhost.c
> > @@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev
> > *dev,
> > static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> > {
> > int r, i, idx;
>
>
Re: [PATCH v2 2/5] vhost: introduce wrappers to set guest notifiers for virtio device
On Sun, May 03, 2020 at 08:36:45PM -0400, Raphael Norwitz wrote: > I’m happy from the vhost, vhost-user-blk and vhost-user-scsi side. For > other device types it looks pretty straightforward, but their maintainers > should probably confirm. > > Since you plan to change the behavior of these helpers in subsequent > patches, maybe consider sending the other device types separately > after the rest of the series has been merged? That way the changes to > individual devices will be much easier to review. Thanks for comments. Agree, will make a more straightforward fix only for vhost-user-blk. After it we can figure out how to propogate this change to other devices. > > On Thu, Apr 30, 2020 at 9:48 AM Dima Stepanov wrote: > > > > Introduce new wrappers to set/reset guest notifiers for the virtio > > device in the vhost device module: > > vhost_dev_assign_guest_notifiers > > ->set_guest_notifiers(..., ..., true); > > vhost_dev_drop_guest_notifiers > > ->set_guest_notifiers(..., ..., false); > > This is a preliminary step to refactor code, so the set_guest_notifiers > > methods could be called based on the vhost device state. > > Update all vhost used devices to use these wrappers instead of direct > > method call. > > > > Signed-off-by: Dima Stepanov > > --- > > backends/cryptodev-vhost.c | 26 +++--- > > backends/vhost-user.c | 16 +--- > > hw/block/vhost-user-blk.c | 15 +-- > > hw/net/vhost_net.c | 30 +- > > hw/scsi/vhost-scsi-common.c | 15 +-- > > hw/virtio/vhost-user-fs.c | 17 +++-- > > hw/virtio/vhost-vsock.c | 18 -- > > hw/virtio/vhost.c | 38 ++ > > hw/virtio/virtio.c | 13 + > > include/hw/virtio/vhost.h | 4 > > include/hw/virtio/virtio.h | 1 + > > 11 files changed, 118 insertions(+), 75 deletions(-) > >
Re: [PATCH v2 3/5] vhost-user-blk: add mechanism to track the guest notifiers init state
On Sun, May 03, 2020 at 09:06:38PM -0400, Raphael Norwitz wrote: > Apologies for mixing up patches last time. This looks good from a > vhost-user-blk perspective, but I worry that some of these changes > could impact other vhost device types. > > I agree with adding notifiers_set to struct vhost_dev, and setting it in > vhost_dev_enable/disable notifiers, but is there any reason notifiers_set > can’t be checked inside vhost-user-blk? Thanks for your review. I also have some concerns about changing current API, but my idea was that these issues will be triggered for all vhost-user/reconnect devices. But maybe you are right and first we should fix vhost-user-blk issues. I'll try to modify patch 2 and 3 in my patchset, so new notifiers_set field will be added, but no API change will be made. Will see how it looks.
[PATCH v2 4/5] vhost: check vring address before calling unmap
Since disconnect can happen at any time during initialization not all
vring buffers (for instance used vring) can be intialized successfully.
If the buffer was not initialized then vhost_memory_unmap call will lead
to SIGSEGV. Add checks for the vring address value before calling unmap.
Also add assert() in the vhost_memory_unmap() routine.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 27 +--
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ddbdc53..3ee50c4 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -314,6 +314,8 @@ static void vhost_memory_unmap(struct vhost_dev *dev, void
*buffer,
hwaddr len, int is_write,
hwaddr access_len)
{
+assert(buffer);
+
if (!vhost_dev_has_iommu(dev)) {
cpu_physical_memory_unmap(buffer, len, is_write, access_len);
}
@@ -1132,12 +1134,25 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
vhost_vq_index);
}
-vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
- 1, virtio_queue_get_used_size(vdev, idx));
-vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
- 0, virtio_queue_get_avail_size(vdev, idx));
-vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
- 0, virtio_queue_get_desc_size(vdev, idx));
+/*
+ * Since the vhost-user disconnect can happen during initialization
+ * check if vring was initialized, before making unmap.
+ */
+if (vq->used) {
+vhost_memory_unmap(dev, vq->used,
+ virtio_queue_get_used_size(vdev, idx),
+ 1, virtio_queue_get_used_size(vdev, idx));
+}
+if (vq->avail) {
+vhost_memory_unmap(dev, vq->avail,
+ virtio_queue_get_avail_size(vdev, idx),
+ 0, virtio_queue_get_avail_size(vdev, idx));
+}
+if (vq->desc) {
+vhost_memory_unmap(dev, vq->desc,
+ virtio_queue_get_desc_size(vdev, idx),
+ 0, virtio_queue_get_desc_size(vdev, idx));
+}
}
static void vhost_eventfd_add(MemoryListener *listener,
--
2.7.4
[PATCH v2 0/5] vhost-user reconnect issues during vhost initialization
Changes in v2: - Add to CC list: Li Feng , since it looks like that we are working on pretty similar issues - Remove [RFC PATCH v1 1/7] contrib/vhost-user-blk: add option to simulate disconnect on init. Going to send this functionality in the separate patch, with the LIBVHOST_USER_DEBUG rework. Need to think how to reuse this option and silence the messages first. - Remove [RFC PATCH v1 3/7] char-socket: initialize reconnect timer only if close is emitted. This will be handled in the separate patchset: [PATCH 3/4] char-socket: avoid double call tcp_chr_free_connection by Li Feng v1: During vhost-user reconnect functionality we hit several issues, if vhost-user-blk daemon is "crashed" or made disconnect during vhost initialization. The general scenario is as follows: - vhost start routine is called - vhost write failed due to SIGPIPE - this call the disconnect routine and vhost_dev_cleanup routine which set to 0 all the field of the vhost_dev structure - return back to vhost start routine with the error - on the fail path vhost start routine tries to rollback the changes by using vhost_dev struct fields which were already reset - sometimes this leads to SIGSEGV, sometimes to SIGABRT Before revising the vhost-user initialization code, we suggest adding the sanity checks to be aware of the possible disconnect event and that the vhost_dev structure can be in "uninitialized" state. The vhost-user-blk daemon is updated with the additional "--simulate-disconnect-stage=CASENUM" argument to simulate disconnect during VHOST device initialization. For instance: 1. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=1 This command will simulate disconnect in the SET_VRING_CALL handler. In this case the vhost device in QEMU is not set the started field to true. 2. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=2 This command will simulate disconnect in the SET_VRING_NUM handler. In this case the started field is set to true. These two cases test different QEMU parts. Also to trigger different code paths disconnect should be simulated in two ways: - before any successful initialization - make successful initialization once and try to simulate disconnects Also we catch SIGABRT on the migration start if vhost-user daemon disconnected during vhost-user set log commands communication. *** BLURB HERE *** Dima Stepanov (5): char-socket: return -1 in case of disconnect during tcp_chr_write vhost: introduce wrappers to set guest notifiers for virtio device vhost-user-blk: add mechanism to track the guest notifiers init state vhost: check vring address before calling unmap vhost: add device started check in migration set log backends/cryptodev-vhost.c | 26 +- backends/vhost-user.c | 16 ++ chardev/char-socket.c | 8 +-- hw/block/vhost-user-blk.c | 23 - hw/net/vhost_net.c | 30 +++- hw/scsi/vhost-scsi-common.c | 15 ++ hw/virtio/vhost-user-fs.c | 17 +++ hw/virtio/vhost-vsock.c | 18 +++ hw/virtio/vhost.c | 115 hw/virtio/virtio.c | 13 + include/hw/virtio/vhost.h | 5 ++ include/hw/virtio/virtio.h | 1 + 12 files changed, 195 insertions(+), 92 deletions(-) -- 2.7.4
[PATCH v2 1/5] char-socket: return -1 in case of disconnect during tcp_chr_write
During testing of the vhost-user-blk reconnect functionality the qemu
SIGSEGV was triggered:
start qemu as:
x86_64-softmmu/qemu-system-x86_64 -m 1024M -M q35 \
-object
memory-backend-file,id=ram-node0,size=1024M,mem-path=/dev/shm/qemu,share=on \
-numa node,cpus=0,memdev=ram-node0 \
-chardev socket,id=chardev0,path=./vhost.sock,noserver,reconnect=1 \
-device vhost-user-blk-pci,chardev=chardev0,num-queues=4 --enable-kvm
start vhost-user-blk daemon:
./vhost-user-blk -s ./vhost.sock -b test-img.raw
If vhost-user-blk will be killed during the vhost initialization
process, for instance after getting VHOST_SET_VRING_CALL command, then
QEMU will fail with the following backtrace:
Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
0x559272bb in vhost_user_read (dev=0x7fffef2d53e0, msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
260 CharBackend *chr = u->user->chr;
#0 0x559272bb in vhost_user_read (dev=0x7fffef2d53e0,
msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
#1 0x5592acb8 in vhost_user_get_config (dev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost-user.c:1645
#2 0x55925525 in vhost_dev_get_config (hdev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost.c:1490
#3 0x558cc46b in vhost_user_blk_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd8f0)
at ./hw/block/vhost-user-blk.c:429
#4 0x55920090 in virtio_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd948)
at ./hw/virtio/virtio.c:3615
#5 0x55a9779c in device_set_realized (obj=0x7fffef2d51a0, value=true,
errp=0x7fffdb88)
at ./hw/core/qdev.c:891
...
The problem is that vhost_user_write doesn't get an error after
disconnect and try to call vhost_user_read(). The tcp_chr_write()
routine should return -1 in case of disconnect. Indicate the EIO error
if this routine is called in the disconnected state.
Signed-off-by: Dima Stepanov
---
chardev/char-socket.c | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index 185fe38..c128cca 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -175,14 +175,16 @@ static int tcp_chr_write(Chardev *chr, const uint8_t
*buf, int len)
if (ret < 0 && errno != EAGAIN) {
if (tcp_chr_read_poll(chr) <= 0) {
tcp_chr_disconnect_locked(chr);
-return len;
+/* Return an error since we made a disconnect. */
+return ret;
} /* else let the read handler finish it properly */
}
return ret;
} else {
-/* XXX: indicate an error ? */
-return len;
+/* Indicate an error. */
+errno = EIO;
+return -1;
}
}
--
2.7.4
[PATCH v2 3/5] vhost-user-blk: add mechanism to track the guest notifiers init state
In case of the vhost-user devices the daemon can be killed at any
moment. Since QEMU supports the reconnet functionality the guest
notifiers should be reset and disabled after "disconnect" event. The
most issues were found if the "disconnect" event happened during vhost
device initialization step.
The disconnect event leads to the call of the vhost_dev_cleanup()
routine. Which memset to 0 a vhost device structure. Because of this, if
device was not started (dev.started == false) and the connection is
broken, then the set_guest_notifier method will produce assertion error.
Also connection can be broken after the dev.started field is set to
true.
A new notifiers_set field is added to the vhost_dev structure to track
the state of the guest notifiers during the initialization process.
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 8
hw/virtio/vhost.c | 11 +++
include/hw/virtio/vhost.h | 1 +
3 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 70d7842..5a3de0f 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -175,7 +175,9 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
return;
}
-vhost_dev_stop(>dev, vdev);
+if (s->dev.started) {
+vhost_dev_stop(>dev, vdev);
+}
ret = vhost_dev_drop_guest_notifiers(>dev, vdev, s->dev.nvqs);
if (ret < 0) {
@@ -337,9 +339,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index fa3da9c..ddbdc53 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1380,6 +1380,7 @@ int vhost_dev_enable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
goto fail_vq;
}
}
+hdev->notifiers_set = true;
return 0;
fail_vq:
@@ -1407,6 +1408,10 @@ void vhost_dev_disable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
int i, r;
+if (!hdev->notifiers_set) {
+return;
+}
+
for (i = 0; i < hdev->nvqs; ++i) {
r = virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index + i,
false);
@@ -1417,6 +1422,8 @@ void vhost_dev_disable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index + i);
}
virtio_device_release_ioeventfd(vdev);
+
+hdev->notifiers_set = false;
}
/*
@@ -1449,6 +1456,10 @@ int vhost_dev_drop_guest_notifiers(struct vhost_dev
*hdev,
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!hdev->notifiers_set) {
+return 0;
+}
+
ret = k->set_guest_notifiers(qbus->parent, nvqs, false);
if (ret < 0) {
error_report("Error reset guest notifier: %d", -ret);
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 4d0d2e2..e3711a7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -90,6 +90,7 @@ struct vhost_dev {
QLIST_HEAD(, vhost_iommu) iommu_list;
IOMMUNotifier n;
const VhostDevConfigOps *config_ops;
+bool notifiers_set;
};
int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
--
2.7.4
[PATCH v2 5/5] vhost: add device started check in migration set log
If vhost-user daemon is used as a backend for the vhost device, then we
should consider a possibility of disconnect at any moment. If such
disconnect happened in the vhost_migration_log() routine the vhost
device structure will be clean up.
At the start of the vhost_migration_log() function there is a check:
if (!dev->started) {
dev->log_enabled = enable;
return 0;
}
To be consistent with this check add the same check after calling the
vhost_dev_set_log() routine. This in general help not to break a
migration due the assert() message. But it looks like that this code
should be revised to handle these errors more carefully.
In case of vhost-user device backend the fail paths should consider the
state of the device. In this case we should skip some function calls
during rollback on the error paths, so not to get the NULL dereference
errors.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 39 +++
1 file changed, 35 insertions(+), 4 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 3ee50c4..d5ab96d 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+
+if (!dev->started) {
+/*
+ * If vhost-user daemon is used as a backend for the
+ * device and the connection is broken, then the vhost_dev
+ * structure will be reset all its values to 0.
+ * Add additional check for the device state.
+ */
+return -1;
+}
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev *dev, bool
enable_log)
}
return 0;
err_vq:
-for (; i >= 0; --i) {
+/*
+ * Disconnect with the vhost-user daemon can lead to the
+ * vhost_dev_cleanup() call which will clean up vhost_dev
+ * structure.
+ */
+for (; dev->started && (i >= 0); --i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
dev->log_enabled);
}
-vhost_dev_set_features(dev, dev->log_enabled);
+if (dev->started) {
+vhost_dev_set_features(dev, dev->log_enabled);
+}
err_features:
return r;
}
@@ -832,7 +850,15 @@ static int vhost_migration_log(MemoryListener *listener,
int enable)
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
-if (r < 0) {
+/*
+ * The dev log resize can fail, because of disconnect
+ * with the vhost-user-blk daemon. Check the device
+ * state before calling the vhost_dev_set_log()
+ * function.
+ * Don't return error if device isn't started to be
+ * consistent with the check above.
+ */
+if (dev->started && r < 0) {
return r;
}
}
@@ -1739,7 +1765,12 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev)
fail_log:
vhost_log_put(hdev, false);
fail_vq:
-while (--i >= 0) {
+/*
+ * Disconnect with the vhost-user daemon can lead to the
+ * vhost_dev_cleanup() call which will clean up vhost_dev
+ * structure.
+ */
+while ((--i >= 0) && (hdev->started)) {
vhost_virtqueue_stop(hdev,
vdev,
hdev->vqs + i,
--
2.7.4
[PATCH v2 2/5] vhost: introduce wrappers to set guest notifiers for virtio device
Introduce new wrappers to set/reset guest notifiers for the virtio
device in the vhost device module:
vhost_dev_assign_guest_notifiers
->set_guest_notifiers(..., ..., true);
vhost_dev_drop_guest_notifiers
->set_guest_notifiers(..., ..., false);
This is a preliminary step to refactor code, so the set_guest_notifiers
methods could be called based on the vhost device state.
Update all vhost used devices to use these wrappers instead of direct
method call.
Signed-off-by: Dima Stepanov
---
backends/cryptodev-vhost.c | 26 +++---
backends/vhost-user.c | 16 +---
hw/block/vhost-user-blk.c | 15 +--
hw/net/vhost_net.c | 30 +-
hw/scsi/vhost-scsi-common.c | 15 +--
hw/virtio/vhost-user-fs.c | 17 +++--
hw/virtio/vhost-vsock.c | 18 --
hw/virtio/vhost.c | 38 ++
hw/virtio/virtio.c | 13 +
include/hw/virtio/vhost.h | 4
include/hw/virtio/virtio.h | 1 +
11 files changed, 118 insertions(+), 75 deletions(-)
diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c
index 8337c9a..4522195 100644
--- a/backends/cryptodev-vhost.c
+++ b/backends/cryptodev-vhost.c
@@ -169,16 +169,13 @@ vhost_set_vring_enable(CryptoDevBackendClient *cc,
int cryptodev_vhost_start(VirtIODevice *dev, int total_queues)
{
VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
-VirtioBusState *vbus = VIRTIO_BUS(qbus);
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
int r, e;
int i;
CryptoDevBackend *b = vcrypto->cryptodev;
CryptoDevBackendVhost *vhost_crypto;
CryptoDevBackendClient *cc;
-if (!k->set_guest_notifiers) {
+if (!virtio_device_guest_notifiers_initialized(dev)) {
error_report("binding does not support guest notifiers");
return -ENOSYS;
}
@@ -198,9 +195,13 @@ int cryptodev_vhost_start(VirtIODevice *dev, int
total_queues)
}
}
-r = k->set_guest_notifiers(qbus->parent, total_queues, true);
+/*
+ * Since all the states are handled by one vhost device,
+ * use the first one in array.
+ */
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+r = vhost_dev_assign_guest_notifiers(_crypto->dev, dev,
total_queues);
if (r < 0) {
-error_report("error binding guest notifier: %d", -r);
goto err;
}
@@ -232,7 +233,8 @@ err_start:
vhost_crypto = cryptodev_get_vhost(cc, b, i);
cryptodev_vhost_stop_one(vhost_crypto, dev);
}
-e = k->set_guest_notifiers(qbus->parent, total_queues, false);
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+e = vhost_dev_drop_guest_notifiers(_crypto->dev, dev, total_queues);
if (e < 0) {
error_report("vhost guest notifier cleanup failed: %d", e);
}
@@ -242,9 +244,6 @@ err:
void cryptodev_vhost_stop(VirtIODevice *dev, int total_queues)
{
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
-VirtioBusState *vbus = VIRTIO_BUS(qbus);
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
CryptoDevBackend *b = vcrypto->cryptodev;
CryptoDevBackendVhost *vhost_crypto;
@@ -259,7 +258,12 @@ void cryptodev_vhost_stop(VirtIODevice *dev, int
total_queues)
cryptodev_vhost_stop_one(vhost_crypto, dev);
}
-r = k->set_guest_notifiers(qbus->parent, total_queues, false);
+/*
+ * Since all the states are handled by one vhost device,
+ * use the first one in array.
+ */
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+r = vhost_dev_drop_guest_notifiers(_crypto->dev, dev, total_queues);
if (r < 0) {
error_report("vhost guest notifier cleanup failed: %d", r);
}
diff --git a/backends/vhost-user.c b/backends/vhost-user.c
index 2bf3406..e116bc6 100644
--- a/backends/vhost-user.c
+++ b/backends/vhost-user.c
@@ -60,15 +60,13 @@ vhost_user_backend_dev_init(VhostUserBackend *b,
VirtIODevice *vdev,
void
vhost_user_backend_start(VhostUserBackend *b)
{
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(b->vdev)));
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret, i ;
if (b->started) {
return;
}
-if (!k->set_guest_notifiers) {
+if (!virtio_device_guest_notifiers_initialized(b->vdev)) {
error_report("binding does not support guest notifiers");
return;
}
@@ -78,9 +76,8 @@ vhost_user_backend_start(VhostUserBackend *b)
return;
}
-ret = k->set_guest_notifiers(qbus->parent, b->dev.nvqs, true);
+ret = vhost_dev_assign_guest_notifiers(>dev, b->vdev, b->dev.n
Re: [RFC PATCH v1 3/7] char-socket: initialize reconnect timer only if close is emitted
On Sun, Apr 26, 2020 at 03:26:58PM +0800, Li Feng wrote:
> This patch is trying to fix the same issue with me.
> However, our fix is different.
>
> I think that check the s->reconnect_timer is better.
I also thought about your solution:
- if (s->reconnect_time) {
+ if (s->reconnect_time && !s->reconnect_timer) {
But was afraid of possible side effects. Since Marc-André approved your
fix, i'm also good with your approach. In this case i'll remove this
patch from the v2 patchset.
Thanks for handling it!
>
> Thanks,
> Feng Li
>
> Marc-André Lureau 于2020年4月24日周五 上午3:16写道：
>
>
> >
> > Hi
> >
> > On Thu, Apr 23, 2020 at 8:41 PM Dima Stepanov
> > wrote:
> > >
> > > During vhost-user reconnect functionality testing the following assert
> > > was hit:
> > > qemu-system-x86_64: chardev/char-socket.c:125:
> > > qemu_chr_socket_restart_timer: Assertion `!s->reconnect_timer' failed.
> > > Aborted (core dumped)
> >
> > That looks related to "[PATCH 3/4] char-socket: avoid double call
> > tcp_chr_free_connection"
> >
> > > This is observed only if the connection is closed by the vhost-user-blk
> > > daemon during the initialization routine. In this case the
> > > tcp_chr_disconnect_locked() routine is called twice. First time it is
> > > called in the tcp_chr_write() routine, after getting the SIGPIPE signal.
> > > Second time it is called when vhost_user_blk_connect() routine return
> > > error. In general it looks correct, because the initialization routine
> > > can return error in many cases.
> > > The tcp_chr_disconnect_locked() routine could be fixed. The timer will
> > > be restarted only if the close event is emitted.
> > >
> > > Signed-off-by: Dima Stepanov
> > > ---
> > > chardev/char-socket.c | 10 +-
> > > 1 file changed, 5 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/chardev/char-socket.c b/chardev/char-socket.c
> > > index c128cca..83ca4d9 100644
> > > --- a/chardev/char-socket.c
> > > +++ b/chardev/char-socket.c
> > > @@ -476,7 +476,7 @@ static void
> > > update_disconnected_filename(SocketChardev *s)
> > > static void tcp_chr_disconnect_locked(Chardev *chr)
> > > {
> > > SocketChardev *s = SOCKET_CHARDEV(chr);
> > > -bool emit_close = s->state == TCP_CHARDEV_STATE_CONNECTED;
> > > +bool was_connected = s->state == TCP_CHARDEV_STATE_CONNECTED;
> > >
> > > tcp_chr_free_connection(chr);
> > >
> > > @@ -485,11 +485,11 @@ static void tcp_chr_disconnect_locked(Chardev *chr)
> > >chr, NULL, chr->gcontext);
> > > }
> > > update_disconnected_filename(s);
> > > -if (emit_close) {
> > > +if (was_connected) {
> > > qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
> > > -}
> > > -if (s->reconnect_time) {
> > > -qemu_chr_socket_restart_timer(chr);
> > > +if (s->reconnect_time) {
> > > +qemu_chr_socket_restart_timer(chr);
> > > +}
> > > }
> > > }
> > >
> > > --
> > > 2.7.4
> > >
> > >
> >
> >
> > --
> > Marc-André Lureau
>
> --
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
>
[RFC PATCH v1 2/7] char-socket: return -1 in case of disconnect during tcp_chr_write
During testing of the vhost-user-blk reconnect functionality the qemu
SIGSEGV was triggered:
start qemu as:
x86_64-softmmu/qemu-system-x86_64 -m 1024M -M q35 \
-object
memory-backend-file,id=ram-node0,size=1024M,mem-path=/dev/shm/qemu,share=on \
-numa node,cpus=0,memdev=ram-node0 \
-chardev socket,id=chardev0,path=./vhost.sock,noserver,reconnect=1 \
-device vhost-user-blk-pci,chardev=chardev0,num-queues=4 --enable-kvm
start vhost-user-blk daemon:
./vhost-user-blk -s ./vhost.sock -b test-img.raw
If vhost-user-blk will be killed during the vhost initialization
process, for instance after getting VHOST_SET_VRING_CALL command, then
QEMU will fail with the following backtrace:
Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
0x559272bb in vhost_user_read (dev=0x7fffef2d53e0, msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
260 CharBackend *chr = u->user->chr;
#0 0x559272bb in vhost_user_read (dev=0x7fffef2d53e0,
msg=0x7fffd5b0)
at ./hw/virtio/vhost-user.c:260
#1 0x5592acb8 in vhost_user_get_config (dev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost-user.c:1645
#2 0x55925525 in vhost_dev_get_config (hdev=0x7fffef2d53e0,
config=0x7fffef2d5394 "", config_len=60)
at ./hw/virtio/vhost.c:1490
#3 0x558cc46b in vhost_user_blk_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd8f0)
at ./hw/block/vhost-user-blk.c:429
#4 0x55920090 in virtio_device_realize (dev=0x7fffef2d51a0,
errp=0x7fffd948)
at ./hw/virtio/virtio.c:3615
#5 0x55a9779c in device_set_realized (obj=0x7fffef2d51a0, value=true,
errp=0x7fffdb88)
at ./hw/core/qdev.c:891
...
The problem is that vhost_user_write doesn't get an error after
disconnect and try to call vhost_user_read(). The tcp_chr_write()
routine should return -1 in case of disconnect. Indicate the EIO error
if this routine is called in the disconnected state.
Signed-off-by: Dima Stepanov
---
chardev/char-socket.c | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index 185fe38..c128cca 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -175,14 +175,16 @@ static int tcp_chr_write(Chardev *chr, const uint8_t
*buf, int len)
if (ret < 0 && errno != EAGAIN) {
if (tcp_chr_read_poll(chr) <= 0) {
tcp_chr_disconnect_locked(chr);
-return len;
+/* Return an error since we made a disconnect. */
+return ret;
} /* else let the read handler finish it properly */
}
return ret;
} else {
-/* XXX: indicate an error ? */
-return len;
+/* Indicate an error. */
+errno = EIO;
+return -1;
}
}
--
2.7.4
[RFC PATCH v1 6/7] vhost: check vring address before calling unmap
Since disconnect can happen at any time during initialization not all
vring buffers (for instance used vring) can be intialized successfully.
If the buffer was not initialized then vhost_memory_unmap call will lead
to SIGSEGV. Add checks for the vring address value before calling unmap.
Also add assert() in the vhost_memory_unmap() routine.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 27 +--
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ddbdc53..3ee50c4 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -314,6 +314,8 @@ static void vhost_memory_unmap(struct vhost_dev *dev, void
*buffer,
hwaddr len, int is_write,
hwaddr access_len)
{
+assert(buffer);
+
if (!vhost_dev_has_iommu(dev)) {
cpu_physical_memory_unmap(buffer, len, is_write, access_len);
}
@@ -1132,12 +1134,25 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
vhost_vq_index);
}
-vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
- 1, virtio_queue_get_used_size(vdev, idx));
-vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
- 0, virtio_queue_get_avail_size(vdev, idx));
-vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
- 0, virtio_queue_get_desc_size(vdev, idx));
+/*
+ * Since the vhost-user disconnect can happen during initialization
+ * check if vring was initialized, before making unmap.
+ */
+if (vq->used) {
+vhost_memory_unmap(dev, vq->used,
+ virtio_queue_get_used_size(vdev, idx),
+ 1, virtio_queue_get_used_size(vdev, idx));
+}
+if (vq->avail) {
+vhost_memory_unmap(dev, vq->avail,
+ virtio_queue_get_avail_size(vdev, idx),
+ 0, virtio_queue_get_avail_size(vdev, idx));
+}
+if (vq->desc) {
+vhost_memory_unmap(dev, vq->desc,
+ virtio_queue_get_desc_size(vdev, idx),
+ 0, virtio_queue_get_desc_size(vdev, idx));
+}
}
static void vhost_eventfd_add(MemoryListener *listener,
--
2.7.4
[RFC PATCH v1 4/7] vhost: introduce wrappers to set guest notifiers for virtio device
Introduce new wrappers to set/reset guest notifiers for the virtio
device in the vhost device module:
vhost_dev_assign_guest_notifiers
->set_guest_notifiers(..., ..., true);
vhost_dev_drop_guest_notifiers
->set_guest_notifiers(..., ..., false);
This is a preliminary step to refactor code, so the set_guest_notifiers
methods could be called based on the vhost device state.
Update all vhost used devices to use these wrappers instead of direct
method call.
Signed-off-by: Dima Stepanov
---
backends/cryptodev-vhost.c | 26 +++---
backends/vhost-user.c | 16 +---
hw/block/vhost-user-blk.c | 15 +--
hw/net/vhost_net.c | 30 +-
hw/scsi/vhost-scsi-common.c | 15 +--
hw/virtio/vhost-user-fs.c | 17 +++--
hw/virtio/vhost-vsock.c | 18 --
hw/virtio/vhost.c | 38 ++
hw/virtio/virtio.c | 13 +
include/hw/virtio/vhost.h | 4
include/hw/virtio/virtio.h | 1 +
11 files changed, 118 insertions(+), 75 deletions(-)
diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c
index 8337c9a..4522195 100644
--- a/backends/cryptodev-vhost.c
+++ b/backends/cryptodev-vhost.c
@@ -169,16 +169,13 @@ vhost_set_vring_enable(CryptoDevBackendClient *cc,
int cryptodev_vhost_start(VirtIODevice *dev, int total_queues)
{
VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
-VirtioBusState *vbus = VIRTIO_BUS(qbus);
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
int r, e;
int i;
CryptoDevBackend *b = vcrypto->cryptodev;
CryptoDevBackendVhost *vhost_crypto;
CryptoDevBackendClient *cc;
-if (!k->set_guest_notifiers) {
+if (!virtio_device_guest_notifiers_initialized(dev)) {
error_report("binding does not support guest notifiers");
return -ENOSYS;
}
@@ -198,9 +195,13 @@ int cryptodev_vhost_start(VirtIODevice *dev, int
total_queues)
}
}
-r = k->set_guest_notifiers(qbus->parent, total_queues, true);
+/*
+ * Since all the states are handled by one vhost device,
+ * use the first one in array.
+ */
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+r = vhost_dev_assign_guest_notifiers(_crypto->dev, dev,
total_queues);
if (r < 0) {
-error_report("error binding guest notifier: %d", -r);
goto err;
}
@@ -232,7 +233,8 @@ err_start:
vhost_crypto = cryptodev_get_vhost(cc, b, i);
cryptodev_vhost_stop_one(vhost_crypto, dev);
}
-e = k->set_guest_notifiers(qbus->parent, total_queues, false);
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+e = vhost_dev_drop_guest_notifiers(_crypto->dev, dev, total_queues);
if (e < 0) {
error_report("vhost guest notifier cleanup failed: %d", e);
}
@@ -242,9 +244,6 @@ err:
void cryptodev_vhost_stop(VirtIODevice *dev, int total_queues)
{
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
-VirtioBusState *vbus = VIRTIO_BUS(qbus);
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
VirtIOCrypto *vcrypto = VIRTIO_CRYPTO(dev);
CryptoDevBackend *b = vcrypto->cryptodev;
CryptoDevBackendVhost *vhost_crypto;
@@ -259,7 +258,12 @@ void cryptodev_vhost_stop(VirtIODevice *dev, int
total_queues)
cryptodev_vhost_stop_one(vhost_crypto, dev);
}
-r = k->set_guest_notifiers(qbus->parent, total_queues, false);
+/*
+ * Since all the states are handled by one vhost device,
+ * use the first one in array.
+ */
+vhost_crypto = cryptodev_get_vhost(b->conf.peers.ccs[0], b, 0);
+r = vhost_dev_drop_guest_notifiers(_crypto->dev, dev, total_queues);
if (r < 0) {
error_report("vhost guest notifier cleanup failed: %d", r);
}
diff --git a/backends/vhost-user.c b/backends/vhost-user.c
index 2bf3406..e116bc6 100644
--- a/backends/vhost-user.c
+++ b/backends/vhost-user.c
@@ -60,15 +60,13 @@ vhost_user_backend_dev_init(VhostUserBackend *b,
VirtIODevice *vdev,
void
vhost_user_backend_start(VhostUserBackend *b)
{
-BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(b->vdev)));
-VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret, i ;
if (b->started) {
return;
}
-if (!k->set_guest_notifiers) {
+if (!virtio_device_guest_notifiers_initialized(b->vdev)) {
error_report("binding does not support guest notifiers");
return;
}
@@ -78,9 +76,8 @@ vhost_user_backend_start(VhostUserBackend *b)
return;
}
-ret = k->set_guest_notifiers(qbus->parent, b->dev.nvqs, true);
+ret = vhost_dev_assign_guest_notifiers(>dev, b->vdev, b->dev.n
[RFC PATCH v1 1/7] contrib/vhost-user-blk: add option to simulate disconnect on init
Add "--simulate-disconnect-stage" option for the testing purposes.
This option can be used to test the vhost-user reconnect functionality:
./vhost-user-blk ... --simulate-disconnect-stage=
In this case the daemon will "crash" in the middle of the VHOST comands
communication. Case nums are as follows:
1 - make assert in the handler of the SET_VRING_CALL command
2 - make assert in the handler of the SET_VRING_NUM command
Main purpose is to test QEMU reconnect functionality. Such fail
injection should not lead to QEMU crash and should be handled
successfully.
Also update the "GOptionEntry entries" definition with the final NULL
item according to API.
Signed-off-by: Dima Stepanov
---
contrib/libvhost-user/libvhost-user.c | 30 ++
contrib/libvhost-user/libvhost-user.h | 13 +
contrib/vhost-user-blk/vhost-user-blk.c | 14 +-
3 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/contrib/libvhost-user/libvhost-user.c
b/contrib/libvhost-user/libvhost-user.c
index 3bca996..5215214 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -73,6 +73,14 @@
#define VHOST_USER_VERSION 1
#define LIBVHOST_USER_DEBUG 0
+/*
+ * Inject fail in different places in daemon. This will trigger different
+ * paths in QEMU. Main purpose is to test the reconnect functionality
+ * during vhost initialization step.
+ */
+#define VHOST_SDISCONNECT_SET_VRING_CALL 1
+#define VHOST_SDISCONNECT_SET_VRING_NUM 2
+
#define DPRINT(...) \
do {\
if (LIBVHOST_USER_DEBUG) { \
@@ -861,6 +869,11 @@ vu_set_vring_num_exec(VuDev *dev, VhostUserMsg *vmsg)
DPRINT("State.index: %d\n", index);
DPRINT("State.num: %d\n", num);
dev->vq[index].vring.num = num;
+if (dev->simulate_init_disconnect == VHOST_SDISCONNECT_SET_VRING_NUM) {
+DPRINT("Simulate vhost daemon crash during initialization.\n");
+assert(0);
+return false;
+}
return false;
}
@@ -1161,6 +1174,13 @@ vu_set_vring_call_exec(VuDev *dev, VhostUserMsg *vmsg)
DPRINT("u64: 0x%016"PRIx64"\n", vmsg->payload.u64);
+/* Simulate crash during initialization. */
+if (dev->simulate_init_disconnect == VHOST_SDISCONNECT_SET_VRING_CALL) {
+DPRINT("Simulate vhost daemon crash during initialization.\n");
+assert(0);
+return false;
+}
+
if (!vu_check_queue_msg_file(dev, vmsg)) {
return false;
}
@@ -2073,6 +2093,16 @@ vu_queue_empty(VuDev *dev, VuVirtq *vq)
return vring_avail_idx(vq) == vq->last_avail_idx;
}
+/*
+ * Set the flag to simulate the vhost-user daemon crash during
+ * initialization. This is used to test reconnect functionality.
+ */
+void
+vu_simulate_init_disconnect(VuDev *dev, int should_simulate)
+{
+dev->simulate_init_disconnect = should_simulate;
+}
+
static bool
vring_notify(VuDev *dev, VuVirtq *vq)
{
diff --git a/contrib/libvhost-user/libvhost-user.h
b/contrib/libvhost-user/libvhost-user.h
index f30394f..9f75e86 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -388,6 +388,9 @@ struct VuDev {
/* Postcopy data */
int postcopy_ufd;
bool postcopy_listening;
+
+/* Fields to simulate test cases. */
+int simulate_init_disconnect;
};
typedef struct VuVirtqElement {
@@ -645,4 +648,14 @@ void vu_queue_get_avail_bytes(VuDev *vdev, VuVirtq *vq,
unsigned int *in_bytes,
bool vu_queue_avail_bytes(VuDev *dev, VuVirtq *vq, unsigned int in_bytes,
unsigned int out_bytes);
+/**
+ * vu_simulate_init_disconnect:
+ * @dev: a VuDev context
+ * @should_simulate: expected simulation behaviour (true or false)
+ *
+ * Set the flag to simulate the vhost-user daemon crash during
+ * initialization. This is used to test reconnect functionality.
+ */
+void vu_simulate_init_disconnect(VuDev *dev, int should_simulate);
+
#endif /* LIBVHOST_USER_H */
diff --git a/contrib/vhost-user-blk/vhost-user-blk.c
b/contrib/vhost-user-blk/vhost-user-blk.c
index 6fd91c7..6ac37ca 100644
--- a/contrib/vhost-user-blk/vhost-user-blk.c
+++ b/contrib/vhost-user-blk/vhost-user-blk.c
@@ -581,6 +581,7 @@ static char *opt_socket_path;
static char *opt_blk_file;
static gboolean opt_print_caps;
static gboolean opt_read_only;
+static gboolean opt_simulate_disconnect;
static GOptionEntry entries[] = {
{ "print-capabilities", 'c', 0, G_OPTION_ARG_NONE, _print_caps,
@@ -592,7 +593,14 @@ static GOptionEntry entries[] = {
{"blk-file", 'b', 0, G_OPTION_ARG_FILENAME, _blk_file,
"block device or file path", "PATH"},
{ "read-only", 'r', 0, G_OPTION_ARG_NONE, _read_only,
- "Enable read-only", NULL }
+ "Enable r
[RFC PATCH v1 3/7] char-socket: initialize reconnect timer only if close is emitted
During vhost-user reconnect functionality testing the following assert
was hit:
qemu-system-x86_64: chardev/char-socket.c:125:
qemu_chr_socket_restart_timer: Assertion `!s->reconnect_timer' failed.
Aborted (core dumped)
This is observed only if the connection is closed by the vhost-user-blk
daemon during the initialization routine. In this case the
tcp_chr_disconnect_locked() routine is called twice. First time it is
called in the tcp_chr_write() routine, after getting the SIGPIPE signal.
Second time it is called when vhost_user_blk_connect() routine return
error. In general it looks correct, because the initialization routine
can return error in many cases.
The tcp_chr_disconnect_locked() routine could be fixed. The timer will
be restarted only if the close event is emitted.
Signed-off-by: Dima Stepanov
---
chardev/char-socket.c | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index c128cca..83ca4d9 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -476,7 +476,7 @@ static void update_disconnected_filename(SocketChardev *s)
static void tcp_chr_disconnect_locked(Chardev *chr)
{
SocketChardev *s = SOCKET_CHARDEV(chr);
-bool emit_close = s->state == TCP_CHARDEV_STATE_CONNECTED;
+bool was_connected = s->state == TCP_CHARDEV_STATE_CONNECTED;
tcp_chr_free_connection(chr);
@@ -485,11 +485,11 @@ static void tcp_chr_disconnect_locked(Chardev *chr)
chr, NULL, chr->gcontext);
}
update_disconnected_filename(s);
-if (emit_close) {
+if (was_connected) {
qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
-}
-if (s->reconnect_time) {
-qemu_chr_socket_restart_timer(chr);
+if (s->reconnect_time) {
+qemu_chr_socket_restart_timer(chr);
+}
}
}
--
2.7.4
[RFC PATCH v1 5/7] vhost-user-blk: add mechanism to track the guest notifiers init state
In case of the vhost-user devices the daemon can be killed at any
moment. Since QEMU supports the reconnet functionality the guest
notifiers should be reset and disabled after "disconnect" event. The
most issues were found if the "disconnect" event happened during vhost
device initialization step.
The disconnect event leads to the call of the vhost_dev_cleanup()
routine. Which memset to 0 a vhost device structure. Because of this, if
device was not started (dev.started == false) and the connection is
broken, then the set_guest_notifier method will produce assertion error.
Also connection can be broken after the dev.started field is set to
true.
A new notifiers_set field is added to the vhost_dev structure to track
the state of the guest notifiers during the initialization process.
Signed-off-by: Dima Stepanov
---
hw/block/vhost-user-blk.c | 8
hw/virtio/vhost.c | 11 +++
include/hw/virtio/vhost.h | 1 +
3 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 70d7842..5a3de0f 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -175,7 +175,9 @@ static void vhost_user_blk_stop(VirtIODevice *vdev)
return;
}
-vhost_dev_stop(>dev, vdev);
+if (s->dev.started) {
+vhost_dev_stop(>dev, vdev);
+}
ret = vhost_dev_drop_guest_notifiers(>dev, vdev, s->dev.nvqs);
if (ret < 0) {
@@ -337,9 +339,7 @@ static void vhost_user_blk_disconnect(DeviceState *dev)
}
s->connected = false;
-if (s->dev.started) {
-vhost_user_blk_stop(vdev);
-}
+vhost_user_blk_stop(vdev);
vhost_dev_cleanup(>dev);
}
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index fa3da9c..ddbdc53 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1380,6 +1380,7 @@ int vhost_dev_enable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
goto fail_vq;
}
}
+hdev->notifiers_set = true;
return 0;
fail_vq:
@@ -1407,6 +1408,10 @@ void vhost_dev_disable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
int i, r;
+if (!hdev->notifiers_set) {
+return;
+}
+
for (i = 0; i < hdev->nvqs; ++i) {
r = virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index + i,
false);
@@ -1417,6 +1422,8 @@ void vhost_dev_disable_notifiers(struct vhost_dev *hdev,
VirtIODevice *vdev)
virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index + i);
}
virtio_device_release_ioeventfd(vdev);
+
+hdev->notifiers_set = false;
}
/*
@@ -1449,6 +1456,10 @@ int vhost_dev_drop_guest_notifiers(struct vhost_dev
*hdev,
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
int ret;
+if (!hdev->notifiers_set) {
+return 0;
+}
+
ret = k->set_guest_notifiers(qbus->parent, nvqs, false);
if (ret < 0) {
error_report("Error reset guest notifier: %d", -ret);
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 4d0d2e2..e3711a7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -90,6 +90,7 @@ struct vhost_dev {
QLIST_HEAD(, vhost_iommu) iommu_list;
IOMMUNotifier n;
const VhostDevConfigOps *config_ops;
+bool notifiers_set;
};
int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
--
2.7.4
[RFC PATCH v1 7/7] vhost: add device started check in migration set log
If vhost-user daemon is used as a backend for the vhost device, then we
should consider a possibility of disconnect at any moment. If such
disconnect happened in the vhost_migration_log() routine the vhost
device structure will be clean up.
At the start of the vhost_migration_log() function there is a check:
if (!dev->started) {
dev->log_enabled = enable;
return 0;
}
To be consistent with this check add the same check after calling the
vhost_dev_set_log() routine. This in general help not to break a
migration due the assert() message. But it looks like that this code
should be revised to handle these errors more carefully.
In case of vhost-user device backend the fail paths should consider the
state of the device. In this case we should skip some function calls
during rollback on the error paths, so not to get the NULL dereference
errors.
Signed-off-by: Dima Stepanov
---
hw/virtio/vhost.c | 39 +++
1 file changed, 35 insertions(+), 4 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 3ee50c4..d5ab96d 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
{
int r, i, idx;
+
+if (!dev->started) {
+/*
+ * If vhost-user daemon is used as a backend for the
+ * device and the connection is broken, then the vhost_dev
+ * structure will be reset all its values to 0.
+ * Add additional check for the device state.
+ */
+return -1;
+}
+
r = vhost_dev_set_features(dev, enable_log);
if (r < 0) {
goto err_features;
@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev *dev, bool
enable_log)
}
return 0;
err_vq:
-for (; i >= 0; --i) {
+/*
+ * Disconnect with the vhost-user daemon can lead to the
+ * vhost_dev_cleanup() call which will clean up vhost_dev
+ * structure.
+ */
+for (; dev->started && (i >= 0); --i) {
idx = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + i);
vhost_virtqueue_set_addr(dev, dev->vqs + i, idx,
dev->log_enabled);
}
-vhost_dev_set_features(dev, dev->log_enabled);
+if (dev->started) {
+vhost_dev_set_features(dev, dev->log_enabled);
+}
err_features:
return r;
}
@@ -832,7 +850,15 @@ static int vhost_migration_log(MemoryListener *listener,
int enable)
} else {
vhost_dev_log_resize(dev, vhost_get_log_size(dev));
r = vhost_dev_set_log(dev, true);
-if (r < 0) {
+/*
+ * The dev log resize can fail, because of disconnect
+ * with the vhost-user-blk daemon. Check the device
+ * state before calling the vhost_dev_set_log()
+ * function.
+ * Don't return error if device isn't started to be
+ * consistent with the check above.
+ */
+if (dev->started && r < 0) {
return r;
}
}
@@ -1739,7 +1765,12 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev)
fail_log:
vhost_log_put(hdev, false);
fail_vq:
-while (--i >= 0) {
+/*
+ * Disconnect with the vhost-user daemon can lead to the
+ * vhost_dev_cleanup() call which will clean up vhost_dev
+ * structure.
+ */
+while ((--i >= 0) && (hdev->started)) {
vhost_virtqueue_stop(hdev,
vdev,
hdev->vqs + i,
--
2.7.4
[RFC PATCH v1 0/7] vhost-user reconnect issues during vhost initialization
During vhost-user reconnect functionality we hit several issues, if vhost-user-blk daemon is "crashed" or made disconnect during vhost initialization. The general scenario is as follows: - vhost start routine is called - vhost write failed due to SIGPIPE - this call the disconnect routine and vhost_dev_cleanup routine which set to 0 all the field of the vhost_dev structure - return back to vhost start routine with the error - on the fail path vhost start routine tries to rollback the changes by using vhost_dev struct fields which were already reset - sometimes this leads to SIGSEGV, sometimes to SIGABRT Before revising the vhost-user initialization code, we suggest adding the sanity checks to be aware of the possible disconnect event and that the vhost_dev structure can be in "uninitialized" state. The vhost-user-blk daemon is updated with the additional "--simulate-disconnect-stage=CASENUM" argument to simulate disconnect during VHOST device initialization. For instance: 1. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=1 This command will simulate disconnect in the SET_VRING_CALL handler. In this case the vhost device in QEMU is not set the started field to true. 2. $ ./vhost-user-blk -s ./vhost.sock -b test-img.raw --simulate-disconnect-stage=2 This command will simulate disconnect in the SET_VRING_NUM handler. In this case the started field is set to true. These two cases test different QEMU parts. Also to trigger different code paths disconnect should be simulated in two ways: - before any successful initialization - make successful initialization once and try to simulate disconnects Also we catch SIGABRT on the migration start if vhost-user daemon disconnected during vhost-user set log commands communication. Dima Stepanov (7): contrib/vhost-user-blk: add option to simulate disconnect on init char-socket: return -1 in case of disconnect during tcp_chr_write char-socket: initialize reconnect timer only if close is emitted vhost: introduce wrappers to set guest notifiers for virtio device vhost-user-blk: add mechanism to track the guest notifiers init state vhost: check vring address before calling unmap vhost: add device started check in migration set log backends/cryptodev-vhost.c | 26 +--- backends/vhost-user.c | 16 ++--- chardev/char-socket.c | 18 ++--- contrib/libvhost-user/libvhost-user.c | 30 + contrib/libvhost-user/libvhost-user.h | 13 contrib/vhost-user-blk/vhost-user-blk.c | 14 +++- hw/block/vhost-user-blk.c | 23 +++ hw/net/vhost_net.c | 30 + hw/scsi/vhost-scsi-common.c | 15 ++--- hw/virtio/vhost-user-fs.c | 17 ++--- hw/virtio/vhost-vsock.c | 18 +++-- hw/virtio/vhost.c | 115 +--- hw/virtio/virtio.c | 13 include/hw/virtio/vhost.h | 5 ++ include/hw/virtio/virtio.h | 1 + 15 files changed, 256 insertions(+), 98 deletions(-) -- 2.7.4
