from:"Coiby Xu"


On Mon, Jul 12, 2021 at 06:20:33AM +0800, Coiby Xu wrote:

On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:

On Sat, 10 Jul 2021 at 14:30, Peter Maydell  wrote:


I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).

I think from grep that this must be the vhost-user-blk test.


I've also now seen this on qemu-system-i386 guest x86-64 Linux host:


Good to to know that! This makes it much easier for me to debug this
issue.


Which i386 image do you use for the guest? Could you share the download
link? I can't find a suitable i386 qcow2 image. For example, [1] is
outdated.

[1] http://people.debian.org/~aurel32/qemu

--
Best regards,
Coiby

Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host


On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:

On Sat, 10 Jul 2021 at 14:30, Peter Maydell  wrote:


I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).

I think from grep that this must be the vhost-user-blk test.


I've also now seen this on qemu-system-i386 guest x86-64 Linux host:


Good to to know that! This makes it much easier for me to debug this
issue.



Process tree:
petmay01 28992  0.0  0.0 123812  8612 ?Sl   14:46   0:01
  \_ tests/qtest/qos-test --tap -k -m quick
petmay01 30068  0.0  0.0 379204 20580 ?Sl   14:46   0:00
  |   \_ ./storage-daemon/qemu-storage-daemon
--blockdev driver=file,node-name=disk0,filename=qtest.6kY6px --export
type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-28992-sock.4Kgtk1,node-name=disk0,writable=on,num-queues=1
petmay01 30070  0.0  0.1 1083248 63748 ?   Sl   14:46   0:00
  |   \_ ./qemu-system-i386 -qtest
unix:/tmp/qtest-28992.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-28992.qmp,id=char0 -mon
chardev=char0,mode=control -display none -M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-sock.4Kgtk1
-accel qtest


Backtrace, qos-test:
(gdb) thread apply all bt

Thread 2 (Thread 0x7fd086f1c700 (LWP 28995)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x56448599484b in qemu_futex_wait (val=,
f=)
   at /mnt/nvmedisk/linaro/qemu-for-merges/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x564485c322e8 )
   at ../../util/qemu-thread-posix.c:480
#3  0x56448599dc18 in call_rcu_thread (opaque=opaque@entry=0x0) at
../../util/rcu.c:258
#4  0x564485993966 in qemu_thread_start (args=)
   at ../../util/qemu-thread-posix.c:541
#5  0x7fd088b446db in start_thread (arg=0x7fd086f1c700) at
pthread_create.c:463
#6  0x7fd08886d71f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fd089d9a900 (LWP 28992)):
#0  0x7fd088b4e474 in __libc_read (fd=6,
buf=buf@entry=0x7fff05f024f0, nbytes=nbytes@entry=1024)
   at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x564485947cb2 in read (__nbytes=1024, __buf=0x7fff05f024f0,
__fd=)
   at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2  qtest_client_socket_recv_line (s=0x5644866f38b0) at
../../tests/qtest/libqtest.c:494
#3  0x564485947e61 in qtest_rsp_args (s=s@entry=0x5644866f38b0,
   expected_args=expected_args@entry=1) at ../../tests/qtest/libqtest.c:521
#4  0x56448594846f in qtest_query_target_endianness (s=0x5644866f38b0)
   at ../../tests/qtest/libqtest.c:570
#5  0x564485948ed2 in qtest_init_without_qmp_handshake
(extra_args=)
   at ../../tests/qtest/libqtest.c:332
#6  0x564485949616 in qtest_init (extra_args=) at
../../tests/qtest/libqtest.c:339
#7  0x5644859338cd in qtest_start (
   args=0x5644866f6d00 "-M pc  -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on  -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
../../tests/qtest/libqtest-single.h:29
#8  restart_qemu_or_continue (
   path=0x5644866f6d00 "-M pc  -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on  -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
../../tests/qtest/qos-test.c:105
#9  run_one_test (arg=) at ../../tests/qtest/qos-test.c:178
#10 0x7fd08990c05a in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#12 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x7fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x7fd08990c232 in g_test_run_suite () from
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#19 0x7fd08990c251 in g_test_run () from
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#20 0x564485932359 in main (argc=, argv=, envp=)
   at ../../tests/qtest/qos-test.c:338

Backtrace, qemu-system-i386:Thread 4 (Thread 0x7f965ac7f700 (LWP 30079)):
#0  0x7f9674b6938c in __GI___sigtimedwait (set=,
   set@entry=0x7f965ac7c090, info=info@entry=0x7f965ac7bfd0,
timeout=timeout@entry=0x0)
   at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
#1  0x7f9674f2c54c in __sigwait (set=set@entry=0x7f965ac7c090,
sig=sig@entry=0x7f965ac7c08c)
   at

Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host


On Sun, Jul 11, 2021 at 06:23:41AM -0700, Richard Henderson wrote:

On 7/11/21 5:16 AM, Peter Maydell wrote:

On Sun, 11 Jul 2021 at 13:10, Coiby Xu  wrote:


Hi Peter,

On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:

I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).


I have a newbie question. How do you do an aarch32 chroot on an aarch64
box? At least, this issue seems to be not reproducible on an aarch64 box
directly. I specifically ran the qos-test for 5 consecutive times and
each time the test could finish successfully,


Your aarch64 host CPU needs to support aarch32 at EL0 (some
AArch64 CPUs are pure-64 bit these days). The host kernel needs
to implement the 32-bit compat layer. It probably also needs to be
built for 4K pages (which mostly means "not RedHat"). Then you can
set up the 32-bit chroot however you'd normally set up a chroot
(for Debian you can do this with debootstrap; other distros will vary;
schroot is also a bit nicer than raw chroot IMHO.)


If you do have a kernel built with 64k pages ("RedHat"), but you do 
have a host cpu that supports aarch32 at EL1 and EL0, then you can run 
aarch32 under KVM.


The command-line I use is

../run/bin/qemu-system-aarch64 -m 4096 -smp 8 -nographic \
 -M virt -cpu host,aarch64=off --accel kvm \
 -kernel vmlinuz-4.19.0-16-armmp-lpae \
 -initrd initrd.img-4.19.0-16-armmp-lpae \
 -append 'console=ttyAMA0 root=/dev/vda2' \
 -drive if=none,file=hda.q,format=qcow2,id=hd,discard=on \
 -device virtio-blk-device,drive=hd \
 -netdev tap,id=tap0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper \
 -device virtio-net-device,netdev=tap0

I believe that I had to perform the install under tcg because I 
couldn't find the right magic to boot off the debian cdrom with kvm.


Thanks for the instructions! Since this issue is also reproducible on 
qemu-system-i386 guest x86-64 Linux host according to Peter's new email, 
I'll check it on i386 guest first.





r~


--
Best regards,
Coiby

Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host


Hi Peter,

On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:

I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).


I have a newbie question. How do you do an aarch32 chroot on an aarch64
box? At least, this issue seems to be not reproducible on an aarch64 box
directly. I specifically ran the qos-test for 5 consecutive times and
each time the test could finish successfully, 


$ MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} \
  QTEST_QEMU_BINARY=build/i386-softmmu/qemu-system-i386 
QTEST_QEMU_IMG=build/qemu-img \
  QTEST_QEMU_STORAGE_DAEMON_BINARY=build/storage-daemon/qemu-storage-daemon \
  build/tests/qtest/qos-test



I think from grep that this must be the vhost-user-blk test.

Here's the process tree:

pmaydell 13126  0.0  0.0   8988  6416 ?SJul09   0:01 make
-C build/all-a32 check V=1 GCC_COLORS= -j9
pmaydell 19632  0.0  0.0   4432  2096 ?SJul09   0:00  \_
bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k' &&
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k -m quick < /dev/null |
./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
pmaydell 19634  0.0  0.0  13608  3076 ?Sl   Jul09   0:02
\_ tests/qtest/qos-test --tap -k -m quick
pmaydell 20679  0.0  0.0 109076 16100 ?Sl   Jul09   0:00
|   \_ ./storage-daemon/qemu-storage-daemon --blockdev
driver=file,node-name=disk0,filename=qtest.X7RL2X --export
type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
pmaydell 20681  0.0  0.2 447828 46544 ?Sl   Jul09   0:00
|   \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
/dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
chardev=char0,mode=control -display none -M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
-accel qtest
pmaydell 19635  0.0  0.0  10256  7176 ?SJul09   0:00
\_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test


Backtrace from tests/qtest/qos-test (not as helpful as it could
be since this is an optimized build):

(gdb) thread apply all bt

Thread 2 (Thread 0xf76ff240 (LWP 19636)):
#0  syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1  0x005206de in qemu_futex_wait (val=, f=) at /home/peter.maydell/qemu/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x5816fc ) at
../../util/qemu-thread-posix.c:480
#3  0x005469c2 in call_rcu_thread (opaque=) at
../../util/rcu.c:258
#4  0x0051fbc2 in qemu_thread_start (args=) at
../../util/qemu-thread-posix.c:541
#5  0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
#6  0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xf7a04010 (LWP 19634)):
#0  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1  0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
#2  0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
__fd=) at
/usr/include/arm-linux-gnueabihf/bits/unistd.h:44
#3  qtest_client_socket_recv_line (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:494
#4  0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
expected_args=expected_args@entry=1) at
../../tests/qtest/libqtest.c:521
#5  0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:570
#6  0x004ec94a in qtest_init_without_qmp_handshake
(extra_args=) at ../../tests/qtest/libqtest.c:332
#7  0x004ecd7a in qtest_init (extra_args=) at
../../tests/qtest/libqtest.c:339
#8  0x004ded10 in qtest_start (
   args=0x1a63710 "-M pc  -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on  -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
../../tests/qtest/libqtest-single.h:29
#9  restart_qemu_or_continue (
   path=0x1a63710 "-M pc  -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on  -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at

Re: [PATCH v2 05/12] test: new qTest case to test the vhost-user-blk-server

2020-12-18 Thread Coiby Xu


On Mon, Dec 07, 2020 at 05:20:23PM +, Stefan Hajnoczi wrote:

From: Coiby Xu 

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since the vhost-user-blk export only serves one
client one time, two exports are started by qemu-storage-daemon for the
hotplug test.

Suggested-by: Thomas Huth 
Signed-off-by: Coiby Xu 
Signed-off-by: Stefan Hajnoczi 
---
MAINTAINERS |   2 +
tests/qtest/libqos/vhost-user-blk.h |  48 ++
tests/qtest/libqos/vhost-user-blk.c | 130 +
tests/qtest/vhost-user-blk-test.c   | 788 
tests/qtest/libqos/meson.build  |   1 +
tests/qtest/meson.build |   4 +
6 files changed, 973 insertions(+)
create mode 100644 tests/qtest/libqos/vhost-user-blk.h
create mode 100644 tests/qtest/libqos/vhost-user-blk.c
create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 68bc160f41..d351280d1f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3103,6 +3103,8 @@ F: block/export/vhost-user-blk-server.c
F: block/export/vhost-user-blk-server.h
F: include/qemu/vhost-user-server.h
F: tests/qtest/libqos/vhost-user-blk.c
+F: tests/qtest/libqos/vhost-user-blk.h
+F: tests/qtest/vhost-user-blk-test.c
F: util/vhost-user-server.c

Replication
diff --git a/tests/qtest/libqos/vhost-user-blk.h 
b/tests/qtest/libqos/vhost-user-blk.h
new file mode 100644
index 00..2a03456a45
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.h
@@ -0,0 +1,48 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#ifndef TESTS_LIBQOS_VHOST_USER_BLK_H
+#define TESTS_LIBQOS_VHOST_USER_BLK_H
+
+#include "qgraph.h"
+#include "virtio.h"
+#include "virtio-pci.h"
+
+typedef struct QVhostUserBlk QVhostUserBlk;
+typedef struct QVhostUserBlkPCI QVhostUserBlkPCI;
+typedef struct QVhostUserBlkDevice QVhostUserBlkDevice;
+
+struct QVhostUserBlk {
+QVirtioDevice *vdev;
+};
+
+struct QVhostUserBlkPCI {
+QVirtioPCIDevice pci_vdev;
+QVhostUserBlk blk;
+};
+
+struct QVhostUserBlkDevice {
+QOSGraphObject obj;
+QVhostUserBlk blk;
+};
+
+#endif
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..568c3426ed
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,130 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  Q

Re: [PATCH for-5.2 01/10] test: new qTest case to test the vhost-user-blk-server

2020-12-18 Thread Coiby Xu

On Mon, Dec 07, 2020 at 11:28:38AM +, Stefan Hajnoczi wrote:

On Wed, Nov 25, 2020 at 04:28:20PM +0800, Coiby Xu wrote:

On Wed, Nov 25, 2020 at 04:20:56PM +0800, Coiby Xu wrote:
> On Wed, Nov 11, 2020 at 12:43:22PM +, Stefan Hajnoczi wrote:
> > +static void quit_storage_daemon(void *qmp_test_state)
> > +{
> > +const char quit_str[] = "{ 'execute': 'quit' }";
> > +
> > +/* Before quiting storate-daemon, quit qemu to avoid dubious messages 
*/
> > +qobject_unref(qtest_qmp(global_qtest, quit_str));
> > +
> > +/*
> > + * Give storage-daemon enough time to wake up
> > + * vu_client_trip coroutine so the Coroutine object could
> > + * be cleaned up. Otherwise LeakSanitizer would complain
> > + * about memory leaks.
> > + */
> > +g_usleep(1000);
>
> Your "[PATCH for-5.2 07/10] vhost-user-blk-test: fix races by using fd 
passing"
> prompts to me think if there is a race condition under which 1000 ms
  ^^^
Sorry, I meant 1000 μs.

In the next revision vhost-user-blk-test sends a SIGTERM signal to
qemu-storage-daemon and then calls waitpid(2). This way there is a clean
shutdown without a sleep.

Thank you for the explaining! Do you think checking if
qemu-storage-daemon has exited by detecting if qemu-storage-daemon's
QMP monitor socket has been closed is simpler solution than
waitpid(2) in v2?

Regarding the LeakSanitizer issue you saw, are you still able to
reproduce it with commit f10802d2c9fd8bfd92c70f465da1a5992445157f
("qemu-storage-daemon: add missing cleanup calls") applied? Maybe
qemu-storage-daemon is still missing some cleanup code (e.g. to stop
exports before terminating).

It seems commit f10802d2c9fd8bfd92c70f465da1a5992445157f
("qemu-storage-daemon: add missing cleanup calls") is not related to
the LeakSanitizer issue.

When applying v9 (no "g_usleep(1000)") on
commit eea8f5df4ecc607d64f091b8d916fcc11a697541
("Merge remote-tracking branch 'remotes/awilliam/tags/vfio-update-20200611.0' into 
staging")
which doesn't contain commit f10802d2c9fd8bfd92c70f465da1a5992445157f,
there is no LeakSanitizer issue.

However, if applying v9 on
commit d0ed6a69d399ae193959225cdeaa9382746c91cc ("Update version for v5.1.0 
release")
which contains commit f10802d2c9fd8bfd92c70f465da1a5992445157f, there is
the LeakSanitizer issue.

And if applying this patch set on
commit 23895cbd82be95428e90168b12e925d0d3ca2f06
("Merge remote-tracking branch 'remotes/awilliam/tags/vfio-update-20201123.0' into 
staging"),
the LeakSanitizer issue couldn't be reproduced even without "g_usleep(1000)".

Stefan

--
Best regards,
Coiby

Re: [PATCH for-5.2 05/10] vhost-user-blk-test: close fork child file descriptors

2020-12-18 Thread Coiby Xu

On Thu, Dec 03, 2020 at 01:57:21PM +, Stefan Hajnoczi wrote:

On Tue, Nov 24, 2020 at 08:08:26PM +0800, Coiby Xu wrote:

Hi Stefan,

On Wed, Nov 11, 2020 at 12:43:26PM +, Stefan Hajnoczi wrote:
> Do not leave stdin and stdout open after fork. stdout is the
> tap-driver.pl pipe. If we keep the pipe open then tap-driver.pl will not
> detect that qos-test has terminated and it will hang.

I wonder under which situation this would happen. I couldn't re-produce
this issue locally.

Try adding g_assert_not_reached() to one of the tests so the qos-test
process aborts. Then tap-driver.pl will hang because the pipe hasn't
been closed and "make check" never completes.

Thank you for the explanation! I thought closing fork child file is
for dealing with a subtle race condition. So it's actually for dealing
with the situation when g_assert_* fails.

Do you think g_spawn_async_with_pipes is a better way than fork/exec
since g_spawn_* could help us take care of closing standard file
descriptors?

g_test_message("starting vhost-user backend: %s",
   storage_daemon_command->str);

GPid child_pid;
g_autoptr(GError) error = NULL;
const gchar * const argv[] = { "/bin/sh", "-c", 
storage_daemon_command->str, NULL };
g_spawn_async_with_pipes (NULL, argv, NULL, G_SPAWN_STDOUT_TO_DEV_NULL, 
NULL,
NULL, _pid, NULL, NULL,
NULL, );

if (error != NULL)
{
g_error ("Starting qemu-storage-daemon failed: %s", error->message);
abort();
}
g_string_free(storage_daemon_command, true);

--
Best regards,
Coiby

Re: [PATCH for-5.2 01/10] test: new qTest case to test the vhost-user-blk-server

2020-11-25 Thread Coiby Xu


On Wed, Nov 25, 2020 at 04:20:56PM +0800, Coiby Xu wrote:

On Wed, Nov 11, 2020 at 12:43:22PM +, Stefan Hajnoczi wrote:

From: Coiby Xu 

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of vhost-user-blk-server are started by
qemu-storage-daemon for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Suggested-by: Thomas Huth 
Signed-off-by: Coiby Xu 
Reviewed-by: Stefan Hajnoczi 
Reviewed-by: Marc-André Lureau 
Message-id: 20200918080912.321299-7-coiby...@gmail.com
[Update meson.build to only test when CONFIG_TOOLS has built
qemu-storage-daemon. This prevents CI failures with --disable-tools.
Also bump RAM to 256 MB because that is the minimum RAM granularity on
ppc64 spapr machines.
--Stefan]
Signed-off-by: Stefan Hajnoczi 
---
tests/qtest/libqos/libqtest.h   |  17 +
tests/qtest/libqos/vhost-user-blk.h |  48 ++
tests/qtest/libqos/vhost-user-blk.c | 129 +
tests/qtest/libqtest.c  |  36 +-
tests/qtest/vhost-user-blk-test.c   | 751 
tests/qtest/libqos/meson.build  |   1 +
tests/qtest/meson.build |   2 +
7 files changed, 982 insertions(+), 2 deletions(-)
create mode 100644 tests/qtest/libqos/vhost-user-blk.h
create mode 100644 tests/qtest/libqos/vhost-user-blk.c
create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index 724f65aa94..d6236ea7a0 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -132,6 +132,23 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ...)
void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...)
   GCC_FMT_ATTR(2, 3);

+/**
+ * qtest_socket_client:
+ * @server_socket_path: the socket server's path
+ *
+ * Connect to a socket server.
+ */
+int qtest_socket_client(char *server_socket_path);
+
+/**
+ * qtest_create_state_with_qmp_fd:
+ * @fd: socket fd
+ *
+ * Wrap socket fd in QTestState to make use of qtest_qmp*
+ * functions
+ */
+QTestState *qtest_create_state_with_qmp_fd(int fd);
+
/**
* qtest_vqmp_fds:
* @s: #QTestState instance to operate on.
diff --git a/tests/qtest/libqos/vhost-user-blk.h 
b/tests/qtest/libqos/vhost-user-blk.h
new file mode 100644
index 00..2a03456a45
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.h
@@ -0,0 +1,48 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#ifndef TESTS_LIBQOS_VHOST_USER_BLK_H
+#define TESTS_LIBQOS_VHOST_USER_BLK_H
+
+#include "qgraph.h"
+#include "virtio.h"
+#include "virtio-pci.h"
+
+typedef struct QVhostUserBlk QVhostUserBlk;
+typedef struct QVhostUserBlkPCI QVhostUserBlkPCI;
+typedef struct QVhostUserBlkDevice QVhostUserBlkDevice;
+
+struct QVhostUserBlk {
+QVirtioDevice *vdev;
+};
+
+struct QVhostUserBlkPCI {
+QVirtioPCIDevice pci_vdev;
+QVhostUserBlk blk;
+};
+
+struct QVhostUserBlkDevice {
+QOSGraphObject obj;
+QVhostUserBlk blk;
+};
+
+#endif
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..58c7e1eb69
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,129 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+

Re: [PATCH for-5.2 01/10] test: new qTest case to test the vhost-user-blk-server

2020-11-25 Thread Coiby Xu


On Wed, Nov 11, 2020 at 12:43:22PM +, Stefan Hajnoczi wrote:

From: Coiby Xu 

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of vhost-user-blk-server are started by
qemu-storage-daemon for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Suggested-by: Thomas Huth 
Signed-off-by: Coiby Xu 
Reviewed-by: Stefan Hajnoczi 
Reviewed-by: Marc-André Lureau 
Message-id: 20200918080912.321299-7-coiby...@gmail.com
[Update meson.build to only test when CONFIG_TOOLS has built
qemu-storage-daemon. This prevents CI failures with --disable-tools.
Also bump RAM to 256 MB because that is the minimum RAM granularity on
ppc64 spapr machines.
--Stefan]
Signed-off-by: Stefan Hajnoczi 
---
tests/qtest/libqos/libqtest.h   |  17 +
tests/qtest/libqos/vhost-user-blk.h |  48 ++
tests/qtest/libqos/vhost-user-blk.c | 129 +
tests/qtest/libqtest.c  |  36 +-
tests/qtest/vhost-user-blk-test.c   | 751 
tests/qtest/libqos/meson.build  |   1 +
tests/qtest/meson.build |   2 +
7 files changed, 982 insertions(+), 2 deletions(-)
create mode 100644 tests/qtest/libqos/vhost-user-blk.h
create mode 100644 tests/qtest/libqos/vhost-user-blk.c
create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index 724f65aa94..d6236ea7a0 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -132,6 +132,23 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ...)
void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...)
GCC_FMT_ATTR(2, 3);

+/**
+ * qtest_socket_client:
+ * @server_socket_path: the socket server's path
+ *
+ * Connect to a socket server.
+ */
+int qtest_socket_client(char *server_socket_path);
+
+/**
+ * qtest_create_state_with_qmp_fd:
+ * @fd: socket fd
+ *
+ * Wrap socket fd in QTestState to make use of qtest_qmp*
+ * functions
+ */
+QTestState *qtest_create_state_with_qmp_fd(int fd);
+
/**
 * qtest_vqmp_fds:
 * @s: #QTestState instance to operate on.
diff --git a/tests/qtest/libqos/vhost-user-blk.h 
b/tests/qtest/libqos/vhost-user-blk.h
new file mode 100644
index 00..2a03456a45
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.h
@@ -0,0 +1,48 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#ifndef TESTS_LIBQOS_VHOST_USER_BLK_H
+#define TESTS_LIBQOS_VHOST_USER_BLK_H
+
+#include "qgraph.h"
+#include "virtio.h"
+#include "virtio-pci.h"
+
+typedef struct QVhostUserBlk QVhostUserBlk;
+typedef struct QVhostUserBlkPCI QVhostUserBlkPCI;
+typedef struct QVhostUserBlkDevice QVhostUserBlkDevice;
+
+struct QVhostUserBlk {
+QVirtioDevice *vdev;
+};
+
+struct QVhostUserBlkPCI {
+QVirtioPCIDevice pci_vdev;
+QVhostUserBlk blk;
+};
+
+struct QVhostUserBlkDevice {
+QOSGraphObject obj;
+QVhostUserBlk blk;
+};
+
+#endif
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..58c7e1eb69
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,129 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include &quo

Re: [PATCH for-5.2 05/10] vhost-user-blk-test: close fork child file descriptors

2020-11-24 Thread Coiby Xu


Hi Stefan,

On Wed, Nov 11, 2020 at 12:43:26PM +, Stefan Hajnoczi wrote:

Do not leave stdin and stdout open after fork. stdout is the
tap-driver.pl pipe. If we keep the pipe open then tap-driver.pl will not
detect that qos-test has terminated and it will hang.


I wonder under which situation this would happen. I couldn't re-produce
this issue locally.



Signed-off-by: Stefan Hajnoczi 
---
tests/qtest/vhost-user-blk-test.c | 9 +
1 file changed, 9 insertions(+)

diff --git a/tests/qtest/vhost-user-blk-test.c 
b/tests/qtest/vhost-user-blk-test.c
index f05f14c192..4019a72ac0 100644
--- a/tests/qtest/vhost-user-blk-test.c
+++ b/tests/qtest/vhost-user-blk-test.c
@@ -749,6 +749,15 @@ static char *start_vhost_user_blk(GString *cmd_line, int 
vus_instances,
   storage_daemon_command->str);
pid_t pid = fork();
if (pid == 0) {
+/*
+ * Close standard file descriptors so tap-driver.pl pipe detects when
+ * our parent terminates.
+ */
+close(0);
+close(1);
+open("/dev/null", O_RDONLY);
+open("/dev/null", O_WRONLY);
+
execlp("/bin/sh", "sh", "-c", storage_daemon_command->str, NULL);
exit(1);
}
--
2.28.0



--
Best regards,
Coiby

Re: [PATCH v10 0/7] vhost-user block device backend implementation

2020-10-10 Thread Coiby Xu


On Fri, Oct 09, 2020 at 11:18:43AM +0100, Stefan Hajnoczi wrote:

On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:

v10
 - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
 - Merge improvements and fixes from Stefan
 - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
 - When shutting down the client, wait for the coroutine of processing
   vhost-user messages to be finished [Kevin]
 - Allocate struct req_data on the heap [Kevin]
 - Improve coding of qtest case [Thomas]
 - Fix several memory leaks detected by ASAN

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (7):
  libvhost-user: Allow vu_message_read to be replaced
  libvhost-user: remove watch for kick_fd when de-initialize vu-dev
  util/vhost-user-server: generic vhost user server
  block: move logical block size check function to a common utility
function
  block/export: vhost-user block device backend server
  test: new qTest case to test the vhost-user-blk-server
  MAINTAINERS: Add vhost-user block device backend server maintainer

 MAINTAINERS|   8 +
 block/export/vhost-user-blk-server.c   | 661 ++
 block/export/vhost-user-blk-server.h   |  36 +
 block/meson.build  |   1 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  15 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 hw/core/qdev-properties.c  |  31 +-
 softmmu/vl.c   |   4 +
 tests/qtest/libqos/libqtest.h  |  17 +
 tests/qtest/libqos/meson.build |   1 +
 tests/qtest/libqos/vhost-user-blk.c| 129 
 tests/qtest/libqos/vhost-user-blk.h|  48 ++
 tests/qtest/libqtest.c |  36 +-
 tests/qtest/meson.build|   4 +-
 tests/qtest/vhost-user-blk-test.c  | 751 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/block-helpers.c   |  46 ++
 util/block-helpers.h   |  19 +
 util/meson.build   |   2 +
 util/vhost-user-server.c   | 428 
 util/vhost-user-server.h   |  65 ++
 23 files changed, 2292 insertions(+), 39 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhos

Re: [PATCH v10 6/7] test: new qTest case to test the vhost-user-blk-server

2020-10-10 Thread Coiby Xu


On Wed, Sep 23, 2020 at 01:36:06PM +0100, Stefan Hajnoczi wrote:

On Fri, Sep 18, 2020 at 04:09:11PM +0800, Coiby Xu wrote:

+int qtest_socket_client(char *server_socket_path)
+{
+struct sockaddr_un serv_addr;
+int sock;
+int ret;
+int retries = 0;
+sock = socket(PF_UNIX, SOCK_STREAM, 0);
+g_assert_cmpint(sock, !=, -1);
+serv_addr.sun_family = AF_UNIX;
+snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+ server_socket_path);
+
+for (retries = 0; retries < 3; retries++) {
+ret = connect(sock, (struct sockaddr *)_addr, sizeof(serv_addr));
+if (ret == 0) {
+break;
+}
+g_usleep(G_USEC_PER_SEC);
+}


This is a race condition. On a heavily loaded machine the server might
not be available within 3 seconds and the test will fail randomly.

Solutions:
1. Wait output from the server indicating it is ready (e.g. 'Listening
  on /path/to/foo.sock...') when spawning the server process.
2. Create the listen socket and pass the fd to the server process. This
  way the socket already exists can the client will block in connect
  until the server accepts the connection.
3. Create a socketpair. Pass one side to the server and use the other
  side in the client.

However, I think this is okay for now. After my patch series that
converts the vhost-user-blk server to the new block exports API we can
consider how to pass file descriptors.


+static void quit_storage_daemon(void *qmp_test_state)
+{
+const char quit_str[] = "{ 'execute': 'quit' }";
+
+/* Before quiting storate-daemon, quit qemu to avoid dubious messages */
+qobject_unref(qtest_qmp(global_qtest, quit_str));
+
+/*
+ * Give storage-daemon enough time to wake up
+ * vu_client_trip coroutine so the Coroutine object could
+ * be cleaned up. Otherwise LeakSanitizer would complain
+ * about memory leaks.
+ */
+g_usleep(1000);


Also a race that may cause random failures. This can be addressed after
the block exports API conversion.


Thank you for spotting two race conditions!

--
Best regards,
Coiby

Re: [PATCH v9 0/5] vhost-user block device backend implementation


On Tue, Sep 15, 2020 at 04:35:57PM +0100, Stefan Hajnoczi wrote:

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.


Hi,
Any update on v10?

Please let me know if there's anything I can do to help.

Stefan


Thank you for ping me! v10 has been submitted.


--
Best regards,
Coiby

[PATCH v10 7/7] MAINTAINERS: Add vhost-user block device backend server maintainer

Suggested-by: Stefano Garzarella 
Signed-off-by: Coiby Xu 
---
 MAINTAINERS | 8 
 1 file changed, 8 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3d17cad19a..55ad6abe73 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3039,6 +3039,14 @@ L: qemu-bl...@nongnu.org
 S: Supported
 F: tests/image-fuzzer/
 
+Vhost-user block device backend server
+M: Coiby Xu 
+S: Maintained
+F: block/export/vhost-user-blk-server.c
+F: util/vhost-user-server.c
+F: tests/qtest/vhost-user-blk-test.c
+F: tests/qtest/libqos/vhost-user-blk.c
+
 Replication
 M: Wen Congyang 
 M: Xie Changlong 
-- 
2.28.0

[PATCH v10 5/7] block/export: vhost-user block device backend server

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Suggested-by: Kevin Wolf 
Signed-off-by: Stefan Hajnoczi 
Signed-off-by: Coiby Xu 
---
 block/export/vhost-user-blk-server.c | 661 +++
 block/export/vhost-user-blk-server.h |  36 ++
 block/meson.build|   1 +
 softmmu/vl.c |   4 +
 4 files changed, 702 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..ec78130f09
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,661 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Parts of the code based on nbd/server.c.
+ *
+ * Copyright (c) Coiby Xu .
+ * Copyright (c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#include "util/block-helpers.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64_to_cpu(desc.sector) << 9,
+  le32_to_cpu(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->server = server;
+req->vq = vq;
+req->elem = elem;
+
+if (unlikely(iov_to_buf(out_iov, out_num, 0, >out,
+sizeof(req->out)) != sizeof(req->out))) {
+error_report("virtio-blk request outhdr too short");
+goto err;
+}
+
+iov_discard_front(_iov, _num, sizeof(req->out));
+
+if (in_iov[in_num - 1].iov_len < sizeof(struct virtio_blk_inhdr)) {
+error_report(&quo

[PATCH v10 6/7] test: new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of vhost-user-blk-server are started by
qemu-storage-daemon for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Suggested-by: Thomas Huth 
Signed-off-by: Coiby Xu 
---
 tests/qtest/libqos/libqtest.h   |  17 +
 tests/qtest/libqos/meson.build  |   1 +
 tests/qtest/libqos/vhost-user-blk.c | 129 +
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c  |  36 +-
 tests/qtest/meson.build |   4 +-
 tests/qtest/vhost-user-blk-test.c   | 751 
 7 files changed, 983 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index a6ee1654f2..2c20381cee 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -132,6 +132,23 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ...)
 void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...)
 GCC_FMT_ATTR(2, 3);
 
+/**
+ * qtest_socket_client:
+ * @server_socket_path: the socket server's path
+ *
+ * Connect to a socket server.
+ */
+int qtest_socket_client(char *server_socket_path);
+
+/**
+ * qtest_create_state_with_qmp_fd:
+ * @fd: socket fd
+ *
+ * Wrap socket fd in QTestState to make use of qtest_qmp*
+ * functions
+ */
+QTestState *qtest_create_state_with_qmp_fd(int fd);
+
 /**
  * qtest_vqmp_fds:
  * @s: #QTestState instance to operate on.
diff --git a/tests/qtest/libqos/meson.build b/tests/qtest/libqos/meson.build
index 1cddf5bdaa..1f5c8f1053 100644
--- a/tests/qtest/libqos/meson.build
+++ b/tests/qtest/libqos/meson.build
@@ -32,6 +32,7 @@ libqos_srcs = files('../libqtest.c',
 'virtio-9p.c',
 'virtio-balloon.c',
 'virtio-blk.c',
+'vhost-user-blk.c',
 'virtio-mmio.c',
 'virtio-net.c',
 'virtio-pci.c',
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..58c7e1eb69
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,129 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVhostUserBlkDevice *vhost_user_blk = g_new0(QVhostUserBlkDevice, 1);
+QVhostUserBlk *interface = _user_blk->blk;
+
+interface->vdev = virtio_dev;
+
+vhost_user_blk->obj.get_driver = qvhost_user_blk_device_get_driver;
+
+return _user_blk->obj;
+}
+
+/* virtio-blk-pci */
+static void *qvhost_user_blk_pci_get_driver(void *object, const char 
*interface)
+{
+QVhostUserBlkPCI *v_blk = object;
+if (!g_strcmp0(interface, "pci-device")) {

[PATCH v10 4/7] block: move logical block size check function to a common utility function

Move the constants from hw/core/qdev-properties.c to
util/block-helpers.h so that knowledge of the min/max values is

Signed-off-by: Stefan Hajnoczi 
Signed-off-by: Coiby Xu 
---
 hw/core/qdev-properties.c | 31 +-
 util/block-helpers.c  | 46 +++
 util/block-helpers.h  | 19 
 util/meson.build  |  1 +
 4 files changed, 71 insertions(+), 26 deletions(-)
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h

diff --git a/hw/core/qdev-properties.c b/hw/core/qdev-properties.c
index 098298c78e..e6ffd80b36 100644
--- a/hw/core/qdev-properties.c
+++ b/hw/core/qdev-properties.c
@@ -16,6 +16,7 @@
 #include "qemu/uuid.h"
 #include "qemu/units.h"
 #include "qemu/cutils.h"
+#include "util/block-helpers.h"
 
 void qdev_prop_set_after_realize(DeviceState *dev, const char *name,
   Error **errp)
@@ -851,16 +852,6 @@ const PropertyInfo qdev_prop_size32 = {
 
 /* --- blocksize --- */
 
-/* lower limit is sector size */
-#define MIN_BLOCK_SIZE  512
-#define MIN_BLOCK_SIZE_STR  "512 B"
-/*
- * upper limit is arbitrary, 2 MiB looks sufficient for all sensible uses, and
- * matches qcow2 cluster size limit
- */
-#define MAX_BLOCK_SIZE  (2 * MiB)
-#define MAX_BLOCK_SIZE_STR  "2 MiB"
-
 static void set_blocksize(Object *obj, Visitor *v, const char *name,
   void *opaque, Error **errp)
 {
@@ -868,6 +859,7 @@ static void set_blocksize(Object *obj, Visitor *v, const 
char *name,
 Property *prop = opaque;
 uint32_t *ptr = qdev_get_prop_ptr(dev, prop);
 uint64_t value;
+Error *local_err = NULL;
 
 if (dev->realized) {
 qdev_prop_set_after_realize(dev, name, errp);
@@ -877,24 +869,11 @@ static void set_blocksize(Object *obj, Visitor *v, const 
char *name,
 if (!visit_type_size(v, name, , errp)) {
 return;
 }
-/* value of 0 means "unset" */
-if (value && (value < MIN_BLOCK_SIZE || value > MAX_BLOCK_SIZE)) {
-error_setg(errp,
-   "Property %s.%s doesn't take value %" PRIu64
-   " (minimum: " MIN_BLOCK_SIZE_STR
-   ", maximum: " MAX_BLOCK_SIZE_STR ")",
-   dev->id ? : "", name, value);
-return;
-}
-
-/* We rely on power-of-2 blocksizes for bitmasks */
-if ((value & (value - 1)) != 0) {
-error_setg(errp,
-  "Property %s.%s doesn't take value '%" PRId64 "', it's not a 
power of 2",
-  dev->id ?: "", name, (int64_t)value);
+check_block_size(dev->id ? : "", name, value, _err);
+if (local_err) {
+error_propagate(errp, local_err);
 return;
 }
-
 *ptr = value;
 }
 
diff --git a/util/block-helpers.c b/util/block-helpers.c
new file mode 100644
index 00..c4851432f5
--- /dev/null
+++ b/util/block-helpers.c
@@ -0,0 +1,46 @@
+/*
+ * Block utility functions
+ *
+ * Copyright IBM, Corp. 2011
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "block-helpers.h"
+
+/**
+ * check_block_size:
+ * @id: The unique ID of the object
+ * @name: The name of the property being validated
+ * @value: The block size in bytes
+ * @errp: A pointer to an area to store an error
+ *
+ * This function checks that the block size meets the following conditions:
+ * 1. At least MIN_BLOCK_SIZE
+ * 2. No larger than MAX_BLOCK_SIZE
+ * 3. A power of 2
+ */
+void check_block_size(const char *id, const char *name, int64_t value,
+  Error **errp)
+{
+/* value of 0 means "unset" */
+if (value && (value < MIN_BLOCK_SIZE || value > MAX_BLOCK_SIZE)) {
+error_setg(errp, QERR_PROPERTY_VALUE_OUT_OF_RANGE,
+   id, name, value, MIN_BLOCK_SIZE, MAX_BLOCK_SIZE);
+return;
+}
+
+/* We rely on power-of-2 blocksizes for bitmasks */
+if ((value & (value - 1)) != 0) {
+error_setg(errp,
+   "Property %s.%s doesn't take value '%" PRId64
+   "', it's not a power of 2",
+   id, name, value);
+return;
+}
+}
diff --git a/util/block-helpers.h b/util/block-helpers.h
new file mode 100644
index 00..b53295a529
--- /dev/null
+++ b/util/block-helpers.h
@@ -0,0 +1,19 @@
+#ifndef BLOCK_HELPERS_H
+#define BLOCK_HELPERS_H
+
+#include "qemu/units.h"
+
+/* lower limit is sector size */
+#define MIN_BLOCK_SIZE  INT64_C(512)
+#define MIN_BLOCK_SIZE_STR

[PATCH v10 3/7] util/vhost-user-server: generic vhost user server

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Suggested-by: Kevin Wolf 
Signed-off-by: Stefan Hajnoczi 
Signed-off-by: Coiby Xu 
---
 util/meson.build |   1 +
 util/vhost-user-server.c | 428 +++
 util/vhost-user-server.h |  65 ++
 3 files changed, 494 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/meson.build b/util/meson.build
index e6b207a99e..3921981ccf 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -66,6 +66,7 @@ if have_block
   util_ss.add(files('main-loop.c'))
   util_ss.add(files('nvdimm-utils.c'))
   util_ss.add(files('qemu-coroutine.c', 'qemu-coroutine-lock.c', 
'qemu-coroutine-io.c'))
+  util_ss.add(when: 'CONFIG_LINUX', if_true: files('vhost-user-server.c'))
   util_ss.add(files('qemu-coroutine-sleep.c'))
   util_ss.add(files('qemu-co-shared-resource.c'))
   util_ss.add(files('thread-pool.c', 'qemu-timer.c'))
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..7b50a2b1fd
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,428 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Copyright (c) Coiby Xu .
+ * Copyright (c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+/*
+ * Before closing the client
+ *
+ * 1. Let vu_client_trip stop processing new vhost-user msg
+ *
+ * 2. remove kick_handler
+ *
+ * 3. wait for the kick handler to be finished
+ *
+ * 4. wait for the current vhost-user msg to be finished processing
+ */
+
+QIOChannelSocket *sioc = server->sioc;
+/* When this is set vu_client_trip will stop new processing vhost-user 
message */
+server->sioc = NULL;
+
+VuFdWatch *vu_fd_watch, *next;
+QTAILQ_FOREACH_SAFE(vu_fd_watch, >vu_fd_watches, next, next) {
+aio_set_fd_handler(server->ioc->ctx, vu_fd_watch->fd, true, NULL,
+   NULL, NULL, NULL);
+}
+
+while (!QTAILQ_EMPTY(>vu_fd_watches)) {
+QTAILQ_FOREACH_SAFE(vu_fd_watch, >vu_fd_watches, next, next) {
+if (!vu_fd_watch->processing) {
+QTAILQ_REMOVE(>vu_fd_watches, vu_fd_watch, next);
+g_free(vu_fd_watch);
+}
+}
+}
+
+while (server->processing_msg) {
+if (server->ioc->read_coroutine) {
+server->ioc->read_coroutine = NULL;
+qio_channel_set_aio_fd_handler(server->ioc, server->ioc->ctx, NULL,
+   NULL, server->ioc);
+server->processing_msg = false;
+}
+}
+
+vu_deinit(>vu_dev);
+object_unref(OBJECT(sioc));
+object_unref(OBJECT(server->ioc));
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+/* avoid while loop in close_client */
+server->processing_msg = false;
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary var

[PATCH v10 0/7] vhost-user block device backend implementation

v10
 - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
 - Merge improvements and fixes from Stefan
 - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
 - When shutting down the client, wait for the coroutine of processing
   vhost-user messages to be finished [Kevin]
 - Allocate struct req_data on the heap [Kevin]
 - Improve coding of qtest case [Thomas]
 - Fix several memory leaks detected by ASAN

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (7):
  libvhost-user: Allow vu_message_read to be replaced
  libvhost-user: remove watch for kick_fd when de-initialize vu-dev
  util/vhost-user-server: generic vhost user server
  block: move logical block size check function to a common utility
function
  block/export: vhost-user block device backend server
  test: new qTest case to test the vhost-user-blk-server
  MAINTAINERS: Add vhost-user block device backend server maintainer

 MAINTAINERS|   8 +
 block/export/vhost-user-blk-server.c   | 661 ++
 block/export/vhost-user-blk-server.h   |  36 +
 block/meson.build  |   1 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  15 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 hw/core/qdev-properties.c  |  31 +-
 softmmu/vl.c   |   4 +
 tests/qtest/libqos/libqtest.h  |  17 +
 tests/qtest/libqos/meson.build |   1 +
 tests/qtest/libqos/vhost-user-blk.c| 129 
 tests/qtest/libqos/vhost-user-blk.h|  48 ++
 tests/qtest/libqtest.c |  36 +-
 tests/qtest/meson.build|   4 +-
 tests/qtest/vhost-user-blk-test.c  | 751 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/block-helpers.c   |  46 ++
 util/block-helpers.h   |  19 +
 util/meson.build   |   2 +
 util/vhost-user-server.c   | 428 
 util/vhost-user-server.h   |  65 ++
 23 files changed, 2292 insertions(+), 39 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
 create mode 100644 util/

[PATCH v10 1/7] libvhost-user: Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest. For slave channel, we still use the default vu_message_read.

Reviewed-by: Marc-André Lureau 
Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 14 +++---
 contrib/libvhost-user/libvhost-user.h  | 21 +
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);
 
-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 53f16bdf08..73732b928f 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -267,7 +265,7 @@ have_userfault(void)
 }
 
 static bool
-vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+vu_message_read_default(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
 char control[CMSG_SPACE(VHOST_MEMORY_BASELINE_NREGIONS * sizeof(int))] = 
{};
 struct iovec iov = {
@@ -415,7 +413,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }
 
-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!vu_message_read_default(dev, dev->slave_fd, _reply)) {
 goto out;
 }
 
@@ -898,7 +896,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1860,7 +1858,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;
 
-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }
 
@@ -1958,6 +1956,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1975,6 +1974,7 @@ vu_init(VuDev *dev,
 
 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read_default;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
@@ -2340,7 +2340,7 @@ static void _vu_queue_notify(VuDev *dev, VuVirtq *vq, 
bool sync)
 
 vu_message_write(dev, dev->slave_fd, );
 if (ack) {
-vu_message_read(dev, dev->slave_fd, );
+vu_message_read_default(dev, dev->slave_fd, );
 }
 return;
 }
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index 844c37c648..fe27831395 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -36,6 +36,8 @@
  */
 #define VHOST_USER_MAX_RAM_SLOTS 32
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -221,6 +223,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -389,6 +392,23 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;
 
+/* @read_msg: custom method to read vhost-user message
+

[PATCH v10 2/7] libvhost-user: remove watch for kick_fd when de-initialize vu-dev

When the client is running in gdb and quit command is run in gdb,
QEMU will still dispatch the event which will cause segment fault in
the callback function.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 73732b928f..5fdfa64294 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -1909,6 +1909,7 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
-- 
2.28.0

Re: Should I delete created coroutines manually?

2020-09-17 Thread Coiby Xu


On Tue, Aug 18, 2020 at 04:19:58PM +0100, Stefan Hajnoczi wrote:

On Mon, Aug 17, 2020 at 2:13 PM Coiby Xu  wrote:

I'm working on vhost-user block device backend [1]. When I tested my
patches on the latest version of QEMU (commit
d0ed6a69d399ae193959225cdeaa9382746c91cc) with --enable-sanitizer turned
on, the follow error will be thrown out,

 ==46631==ERROR: LeakSanitizer: detected memory leaks

 Direct leak of 312 byte(s) in 1 object(s) allocated from:
 #0 0x55d0a7813921 in calloc (qemu/qemu-storage-daemon+0x646921)
 #1 0x7f8bd6f7b591 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x4f591)
 #2 0x55d0a815f3fb in qemu_coroutine_create 
qemu/util/qemu-coroutine.c:75:14
 #3 0x55d0a817321e in vu_client_start 
qemu/util/vhost-user-server.c:225:23
 #4 0x55d0a816fcf3 in vu_accept qemu/util/vhost-user-server.c:341:5
 #5 0x55d0a7f1293f in qio_net_listener_channel_func 
qemu/io/net-listener.c:54:9
 #6 0x55d0a7efb0f3 in qio_channel_fd_source_dispatch 
qemu/io/channel-watch.c:84:12
 #7 0x7f8bd6f7e33f in g_main_context_dispatch 
(/usr/lib/libglib-2.0.so.0+0x5233f)

I have to call qemu_coroutine_delete to eliminate this kind of errors.


qemu_coroutine_delete() is an internal API and shouldn't be called directly.

If you are sure that the coroutine terminated (returned from its entry
function) then the leak is probably caused by the following:


Thank you for this clue! Yesterday I realized the execution of coroutine
has been yielded since qio_channel_yield is called. Now I give the socket
server enough time to do the clean-up work after the client socket is
closed and the issue is gone.



util/qemu-coroutine.c has a pool of Coroutine objects that are reused.
It's likely that the "leaked" object is in that pool.

I'm surprised that this issue hasn't been reported before, but
release_pool should be cleaned up when the program terminates.


I still can't figure out why the issue hasn't been reported before. I guess
I have to put it aside.


Stefan


--
Best regards,
Coiby

Should I delete created coroutines manually?


Hi,

I'm working on vhost-user block device backend [1]. When I tested my
patches on the latest version of QEMU (commit
d0ed6a69d399ae193959225cdeaa9382746c91cc) with --enable-sanitizer turned
on, the follow error will be thrown out,

==46631==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 312 byte(s) in 1 object(s) allocated from:
#0 0x55d0a7813921 in calloc (qemu/qemu-storage-daemon+0x646921)
#1 0x7f8bd6f7b591 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x4f591)
#2 0x55d0a815f3fb in qemu_coroutine_create 
qemu/util/qemu-coroutine.c:75:14
#3 0x55d0a817321e in vu_client_start 
qemu/util/vhost-user-server.c:225:23
#4 0x55d0a816fcf3 in vu_accept qemu/util/vhost-user-server.c:341:5
#5 0x55d0a7f1293f in qio_net_listener_channel_func 
qemu/io/net-listener.c:54:9
#6 0x55d0a7efb0f3 in qio_channel_fd_source_dispatch 
qemu/io/channel-watch.c:84:12
#7 0x7f8bd6f7e33f in g_main_context_dispatch 
(/usr/lib/libglib-2.0.so.0+0x5233f)

I have to call qemu_coroutine_delete to eliminate this kind of errors.

I noticed
make docker-test-clang@ubuntu SHOW_ENV=1 J=12 TARGET_LIST=x86_64-softmmu 
EXTRA_CONFIGURE_OPTS=--enable-sanitizers"
won't give the above error while
"make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=12" would.

And for older commits like commit eea8f5df4ecc607d64f091b8d916fcc11a697541,
this kind of error won't be shown either.

Is this kind of error true a false positve because "ASan doesn't fully
support makecontext/swapcontext functions and may produce false positives in some 
cases"?

I don't have sufficient knowledge on QEMU coroutine and ASan to reach a
conclusion. Could anyone enlighten me? Thank you!

[1] https://patchew.org/QEMU/20200614183907.514282-1-coiby...@gmail.com/

--
Best regards,
Coiby

Re: [PATCH 2/6] vhost-user-server: drop unused #include


On Fri, Jun 19, 2020 at 01:00:42PM +0100, Stefan Hajnoczi wrote:

Signed-off-by: Stefan Hajnoczi 
---
util/vhost-user-server.c | 1 -
1 file changed, 1 deletion(-)

diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
index e94a8d8a83..49ada8bc78 100644
--- a/util/vhost-user-server.c
+++ b/util/vhost-user-server.c
@@ -7,7 +7,6 @@
 * later.  See the COPYING file in the top-level directory.
 */
#include "qemu/osdep.h"
-#include 
#include "qemu/main-loop.h"
#include "vhost-user-server.h"

--
2.26.2



All the patches have been applied to v10. I'm curious how do you find
this issue. Is there a tool to detect this issue or simply you are so
familiar with the QEMU code that you can spot it very easily?

--
Best regards,
Coiby

Re: [PATCH v9 4/5] vhost-user block device backend server


On Thu, Jun 18, 2020 at 05:57:40PM +0200, Kevin Wolf wrote:

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   1 +
 block/export/vhost-user-blk-server.c | 669 +++
 block/export/vhost-user-blk-server.h |  35 ++
 softmmu/vl.c |   4 +
 4 files changed, 709 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..0eb7eff470 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o

 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..bbf2ceaa9b
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,669 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#include "util/block-helpers.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64_to_cpu(desc.sector) << 9,
+  le32_to_cpu(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req-&

Re: [PATCH v9 2/5] generic vhost user server


On Thu, Jun 18, 2020 at 03:29:26PM +0200, Kevin Wolf wrote:

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   1 +
 util/vhost-user-server.c | 400 +++
 util/vhost-user-server.h |  61 ++
 3 files changed, 462 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index cc5e37177a..b4d4af06dc 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += qemu-sockets.o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..393beeb6b9
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,400 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+vu_deinit(>vu_dev);
+object_unref(OBJECT(server->sioc));
+object_unref(OBJECT(server->ioc));
+server->sioc_slave = NULL;


Where is sioc_slave closed/freed?


Thank you for pointing out my neglect! When working on v10, I realized
communication on the slave channel can't be done easily in a coroutine.
So I simply dropped the support.


+object_unref(OBJECT(server->ioc_slave));
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);


If connecting another client to the server should work, don't we have to
set at least server->sioc = NULL so that vu_accept() won't error out?


Previously I set `server->sioc = NULL` in the panic_cb, i.e. only when the
client disconnects, because I thought it's different from the case that the
server is shutdown. But this differentiating is not necessary. In v10, I
has moved `server->sioc = NULL` into `close_client`.




+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+server->sioc = NULL;
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+Error **local_err)
+{
+if (server->sioc_slave) {
+if (fd == server->sioc_slave->fd) {
+return server->ioc_slave;
+}
+} else {
+server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+if (!*local_err) {
+server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+return server->ioc_slave;
+}
+}
+
+return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+

Re: [PATCH v9 2/5] generic vhost user server


On Fri, Jun 19, 2020 at 01:13:00PM +0100, Stefan Hajnoczi wrote:

On Mon, Jun 15, 2020 at 02:39:04AM +0800, Coiby Xu wrote:

+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+KickInfo *kick_info = opaque;
+kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);


Where is kick_info->index assigned? It appears to be NULL in all cases.


+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+  vu_watch_cb cb, void *pvt)
+{
+
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+g_assert(vu_dev);
+g_assert(fd >= 0);
+long index = (intptr_t) pvt;


The meaning of the pvt argument is not defined in the library interface.
set_watch() callbacks shouldn't interpret pvt.

You could modify libvhost-user to explicitly pass the virtqueue index
(or -1 if the fd is not associated with a virtqueue), but it's nice to
avoid libvhost-user API changes so that existing libvhost-user
applications don't require modifications.

What I would do here is to change the ->kick_info[] data struct. How
about a linked list of VuFdWatch objects? That way the code can handle
any number of fd watches and doesn't make assumptions about virtqueues.
set_watch() is a generic fd monitoring interface and doesn't need to be
tied to virtqueues.


A linked list of VuFdWatch objects has been adopted in v10. Thank you!

--
Best regards,
Coiby

Re: [PATCH v9 0/5] vhost-user block device backend implementation


On Thu, Jun 18, 2020 at 09:28:44AM +0100, Stefan Hajnoczi wrote:

On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:

On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-re...@patchew.org wrote:
> Patchew URL: 
https://patchew.org/QEMU/20200614183907.514282-1-coiby...@gmail.com/
>
>
>
> Hi,
>
> This series failed the asan build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce 
it
> locally.
>
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> export ARCH=x86_64
> make docker-image-fedora V=1 NETWORK=1
> time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
> === TEST SCRIPT END ===
>
>  CC  stubs/vm-stop.o
>  CC  ui/input-keymap.o
>  CC  qemu-keymap.o
> /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared 
identifier 'VHOST_MEMORY_MAX_NREGIONS'
> VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
> ^
>
> The full log is available at
> 
http://patchew.org/logs/20200614183907.514282-1-coiby...@gmail.com/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7
and this docker test. And I can't see any reason for this error to occur since
VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
which has been included by util/vhost-user-server.h.


Using G_N_ELEMENTS(vmsg->fds) instead of VHOST_MEMORY_MAX_NREGIONS is an
even cleaner fix.

Stefan


Thank you for this cleaner fix!

--
Best regards,
Coiby

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server


On Wed, Jun 24, 2020 at 05:14:22PM +0200, Thomas Huth wrote:

On 14/06/2020 20.39, Coiby Xu wrote:

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---

[...]

diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index 49075b55a1..02cc09f893 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -52,8 +52,7 @@ typedef struct QTestClientTransportOps {
 QTestRecvFn recv_line; /* for receiving qtest command responses */
 } QTestTransportOps;
-struct QTestState
-{
+struct QTestState {
 int fd;
 int qmp_fd;
 pid_t qemu_pid;  /* our child QEMU process */
@@ -608,6 +607,38 @@ QDict *qtest_qmp_receive(QTestState *s)
 return qmp_fd_receive(s->qmp_fd);
 }
+QTestState *qtest_create_state_with_qmp_fd(int fd)
+{
+QTestState *qmp_test_state = g_new0(QTestState, 1);
+qmp_test_state->qmp_fd = fd;
+return qmp_test_state;
+}
+
+int qtest_socket_client(char *server_socket_path)
+{
+struct sockaddr_un serv_addr;
+int sock;
+int ret;
+int retries = 0;
+sock = socket(PF_UNIX, SOCK_STREAM, 0);
+g_assert_cmpint(sock, !=, -1);
+serv_addr.sun_family = AF_UNIX;
+snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+ server_socket_path);
+
+do {


Why not simply:

for (retries = 0; retries < 3; retries++)

?


Thank you for the advice which has been applied to v10.


+ret = connect(sock, (struct sockaddr *)_addr, sizeof(serv_addr));
+if (ret == 0) {
+break;
+}
+retries += 1;
+g_usleep(G_USEC_PER_SEC);
+} while (retries < 3);
+
+g_assert_cmpint(ret, ==, 0);
+return sock;
+}

[...]

diff --git a/tests/qtest/vhost-user-blk-test.c 
b/tests/qtest/vhost-user-blk-test.c
new file mode 100644
index 00..56e3d8f338
--- /dev/null
+++ b/tests/qtest/vhost-user-blk-test.c
@@ -0,0 +1,739 @@
+/*
+ * QTest testcase for VirtIO Block Device
+ *
+ * Copyright (c) 2014 SUSE LINUX Products GmbH
+ * Copyright (c) 2014 Marc Marí
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest-single.h"
+#include "qemu/bswap.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "standard-headers/linux/virtio_pci.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+#include "libqos/libqos-pc.h"
+
+/* TODO actually test the results and get rid of this */
+#define qmp_discard_response(...) qobject_unref(qmp(__VA_ARGS__))


Please no more qmp_discard_response() in new code!


I planned to check if the response has the dict key "return" or "QMP".
But when sometimes the SHUTDOWN event is returned which doesn't has
the key "QMP". I'm not sure if QMI will change in the fure. So I use
qobject_unref instead which I don't think will affect the correctness
of the tests.


+#define TEST_IMAGE_SIZE (64 * 1024 * 1024)
+#define QVIRTIO_BLK_TIMEOUT_US  (30 * 1000 * 1000)
+#define PCI_SLOT_HP 0x06
+
+typedef struct QVirtioBlkReq {
+uint32_t type;
+uint32_t ioprio;
+uint64_t sector;
+char *data;
+uint8_t status;
+} QVirtioBlkReq;
+
+
+#ifdef HOST_WORDS_BIGENDIAN
+static const bool host_is_big_endian = true;
+#else
+static const bool host_is_big_endian; /* false */
+#endif
+
+static inline void virtio_blk_fix_request(QVirtioDevice *d, QVirtioBlkReq *req)
+{
+if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+req->type = bswap32(req->type);
+req->ioprio = bswap32(req->ioprio);
+req->sector = bswap64(req->sector);
+}
+}
+
+


One empty line should be enough ;-)


+static inline void virtio_blk_fix_dwz_hdr(QVirtioDevice *d,
+struct virtio_blk_discard_write_zeroes *dwz_hdr)
+{
+if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+dwz_hdr->sector = bswap64(dwz_hdr->sector);
+dwz_hdr->num_sectors = bswap32(dwz_hdr->num_sectors);
+dwz_hdr->flags = bswap32(dwz_hdr->flags);
+}
+}
+
+static uint64_t virtio_blk_request(QGuestAllocator *alloc, QVirtioDevice *d,
+   QVirtioBlkReq *req, uint64_t data_size)
+{
+uint64_t addr;
+uint8_t status = 0xFF;
+
+switch (req->type) {
+case VIRTIO_BLK_T_IN:
+case VIRTIO_BLK_T_OUT:
+g_assert_cmpuint(data_size % 512, ==, 0);
+

Re: [PATCH v9 0/5] vhost-user block device backend implementation

2020-06-25 Thread Coiby Xu


On Fri, Jun 19, 2020 at 01:07:46PM +0100, Stefan Hajnoczi wrote:

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.


I have replied with patches that you can consider squashing into your
series. I was testing this patch series and decided it was easier to
send code than to go back and write review comments since I was already
on a git branch.

My patches can be combined into your original patches using "git rebase
-i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing
patches so that the git log records that I have confirmed that I have
permission to contribute this code.

If you have questions about any of the patches, please let me know.


Besides your Signed-off-by: line, shouldn't I also add copyright info to
the affected files as follows?

 * Copyright (c) 2020 Coiby Xu 
 *
 * Copyright (c) 2020 Red Hat, Inc., Stefan Hajnoczi 
 *


--
Best regards,
Coiby

Re: [PATCH v9 0/5] vhost-user block device backend implementation


On Fri, Jun 19, 2020 at 01:07:46PM +0100, Stefan Hajnoczi wrote:

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.


I have replied with patches that you can consider squashing into your
series. I was testing this patch series and decided it was easier to
send code than to go back and write review comments since I was already
on a git branch.

My patches can be combined into your original patches using "git rebase
-i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing
patches so that the git log records that I have confirmed that I have
permission to contribute this code.


I was thinking on how to incorporate your work while reading the emails.
You just provide the instruction! Thank you!


--
Best regards,
Coiby

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server


On Thu, Jun 18, 2020 at 04:17:51PM +0100, Stefan Hajnoczi wrote:

On Mon, Jun 15, 2020 at 02:39:07AM +0800, Coiby Xu wrote:

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c  |  35 +-
 tests/qtest/libqtest.h  |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c


This test case fails for me:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read msg header. Read -1 instead of 12. Original 
request 11.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Input/output error (5)

Does "make -j4 check" pass for you?


Actually it's a success since it won't fail CI. The reason for the
occurrence of these dubious messages is after finishing the tests,
vhost-user-blk-server is stopped before qemu-system-x86_64 is destroyed.
I'll see if I can find a way to kill qemu-system-x86_64 first.

--
Best regards,
Coiby

Re: [PATCH v9 0/5] vhost-user block device backend implementation

On Thu, Jun 18, 2020 at 09:27:48AM +0100, Stefan Hajnoczi wrote:

On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:

On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-re...@patchew.org wrote:
> Patchew URL: 
https://patchew.org/QEMU/20200614183907.514282-1-coiby...@gmail.com/
>
>
>
> Hi,
>
> This series failed the asan build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce 
it
> locally.
>
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> export ARCH=x86_64
> make docker-image-fedora V=1 NETWORK=1
> time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
> === TEST SCRIPT END ===
>
>  CC  stubs/vm-stop.o
>  CC  ui/input-keymap.o
>  CC  qemu-keymap.o
> /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared 
identifier 'VHOST_MEMORY_MAX_NREGIONS'
> VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
> ^
>
> The full log is available at
> 
http://patchew.org/logs/20200614183907.514282-1-coiby...@gmail.com/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7
and this docker test. And I can't see any reason for this error to occur since
VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
which has been included by util/vhost-user-server.h.

Please see the recent change in commit
b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in
libvhost-user").

The error can be solved by replacing VHOST_MEMORY_MAX_NREGIONS with
VHOST_MEMORY_BASELINE_NREGIONS in util/vhost-user-server.c.

Thank you for the clarification! I did run "git pull" when checking this error.
It seems there is a delay when pulling updates from git://git.qemu.org/qemu.git.

--
Best regards,
Coiby

Re: [PATCH v9 1/5] Allow vu_message_read to be replaced

On Thu, Jun 18, 2020 at 12:43:47PM +0200, Kevin Wolf wrote:

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Signed-off-by: Coiby Xu 

_vu_queue_notify() still has a direct call of vu_message_read() instead
of using the pointer. Is this intentional?

This is a mistake. Thank you for reminding me!

Renaming the function would make sure that such semantic merge conflicts
don't stay unnoticed.

Thank you for this tip! Do you suggest renaming the function only for
the purpose of testing or should I adopt a name when submitting the
patch? For the latter case, I will change it to vu_message_read_default
then.

@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }

 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }

This hunk looks unrelated.

In v4, I made the comment to explain why it's needed. But libvhost-user
is supposed to be independent from QEMU, so Stefan suggested to remove it,

> @@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
>  }
>
>  if (vq->kick_fd != -1) {
> +/* remove watch for kick_fd
> + * When client process is running in gdb and
> + * quit command is run in gdb, QEMU will still dispatch the event
> + * which will cause segment fault in the callback function
> + */

Code and comments in libvhost-user should not refer to QEMU specifics.
Removing the watch is a good idea regardless of the application or event
loop implementation.  No comment is needed here.
+/* remove watch for kick_fd
+ * When client process is running in gdb and
+ * quit command is run in gdb, QEMU will still dispatch the event
+ * which will cause segment fault in the callback function
+ */
+dev->remove_watch(dev, vq->kick_fd);

--
Best regards,
Coiby

Re: [PATCH v8 0/4] vhost-user block device backend implementation

2020-06-16 Thread Coiby Xu


On Mon, Jun 15, 2020 at 10:46:10AM +0200, Stefano Garzarella wrote:

On Mon, Jun 15, 2020 at 02:46:40AM +0800, Coiby Xu wrote:

Hi Stefano Garzarella,

On Thu, Jun 11, 2020 at 02:37:03PM +0200, Stefano Garzarella wrote:
> Hi Coiby Xu,
>
> On Fri, Jun 05, 2020 at 07:35:34AM +0800, Coiby Xu wrote:
> > v8
> >  - re-try connecting to socket server to fix asan error
> >  - fix license naming issue
> >
> > v7
> >  - fix docker-test-debug@fedora errors by freeing malloced memory
> >
> > v6
> >  - add missing license header and include guard
> >  - vhost-user server only serve one client one time
> >  - fix a bug in custom vu_message_read
> >  - using qemu-storage-daemon to start vhost-user-blk-server
> >  - a bug fix to pass docker-test-clang@ubuntu
> >
> > v5:
> >  * re-use vu_kick_cb in libvhost-user
> >  * keeping processing VhostUserMsg in the same coroutine until there is
> >detachment/attachment of AIOContext
> >  * Spawn separate coroutine for each VuVirtqElement
> >  * Other changes including relocating vhost-user-blk-server.c, coding
> >style etc.
> >
> > v4:
> >  * add object properties in class_init
> >  * relocate vhost-user-blk-test
> >  * other changes including using SocketAddress, coding style, etc.
> >
> > v3:
> >  * separate generic vhost-user-server code from vhost-user-blk-server
> >code
> >  * re-write vu_message_read and kick hander function as coroutines to
> >directly call blk_co_preadv, blk_co_pwritev, etc.
> >  * add aio_context notifier functions to support multi-threading model
> >  * other fixes regarding coding style, warning report, etc.
> >
> > v2:
> >  * Only enable this feature for Linux because eventfd is a Linux-specific
> >feature
> >
> >
> > This patch series is an implementation of vhost-user block device
> > backend server, thanks to Stefan and Kevin's guidance.
> >
> > Vhost-user block device backend server is a UserCreatable object and can be
> > started using object_add,
> >
> >  (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
> >  (qemu) object_del ID
> >
> > or appending the "-object" option when starting QEMU,
> >
> >   $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
> >
> > Then vhost-user client can connect to the server backend.
> > For example, QEMU could act as a client,
> >
> >   $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1
> >
> > And guest OS could access this vhost-user block device after mounting it.
> >
> > Coiby Xu (4):
> >   Allow vu_message_read to be replaced
> >   generic vhost user server
> >   vhost-user block device backend server
> >   new qTest case to test the vhost-user-blk-server
> >
> >  block/Makefile.objs|   1 +
> >  block/export/vhost-user-blk-server.c   | 716 
> >  block/export/vhost-user-blk-server.h   |  34 +
> >  contrib/libvhost-user/libvhost-user-glib.c |   2 +-
> >  contrib/libvhost-user/libvhost-user.c  |  11 +-
> >  contrib/libvhost-user/libvhost-user.h  |  21 +
> >  softmmu/vl.c   |   4 +
> >  tests/Makefile.include |   3 +-
> >  tests/qtest/Makefile.include   |   2 +
> >  tests/qtest/libqos/vhost-user-blk.c| 130 
> >  tests/qtest/libqos/vhost-user-blk.h|  44 ++
> >  tests/qtest/libqtest.c |  54 +-
> >  tests/qtest/libqtest.h |  38 ++
> >  tests/qtest/vhost-user-blk-test.c  | 737 +
> >  tests/vhost-user-bridge.c  |   2 +
> >  tools/virtiofsd/fuse_virtio.c  |   4 +-
> >  util/Makefile.objs |   1 +
> >  util/vhost-user-server.c   | 406 
> >  util/vhost-user-server.h   |  59 ++
> >  19 files changed, 2229 insertions(+), 40 deletions(-)
> >  create mode 100644 block/export/vhost-user-blk-server.c
> >  create mode 100644 block/export/vhost-user-blk-server.h
> >  create mode 100644 tests/qtest/libqos/vhost-user-blk.c
> >  create mode 100644 tests/qtest/libqos/vhost-user-blk.h
> >  create mode 100644 tests/qtest/vhost-user-blk-test.c
> >  create mode 100644 util/vhost-user-server.c
> >  create mode 100644 util/vhost-user-server.h
> >
>
> Should we add an entry in the MAINTAINERS file for some of the new files?
> (e.g. util/vhost-user-server.*)

Yes, please. Thank you!


I think the best thing should be to edit MAINTAINERS in this series,
since you're adding new files, but I don't know who will maintain them ;-)


Thank you for the explanation! I thought the MAINTAINERS file is supposed
to be treated in a special way:)

--
Best regards,
Coiby

Re: [PATCH v9 0/5] vhost-user block device backend implementation

2020-06-16 Thread Coiby Xu


On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-re...@patchew.org wrote:

Patchew URL: 
https://patchew.org/QEMU/20200614183907.514282-1-coiby...@gmail.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

 CC  stubs/vm-stop.o
 CC  ui/input-keymap.o
 CC  qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared 
identifier 'VHOST_MEMORY_MAX_NREGIONS'
VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
^

The full log is available at
http://patchew.org/logs/20200614183907.514282-1-coiby...@gmail.com/testing.asan/?type=message.


I couldn't re-produce this error locally for both docker-test-quick@centos7
and this docker test. And I can't see any reason for this error to occur since
VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
which has been included by util/vhost-user-server.h.

--
Best regards,
Coiby

Re: [PATCH v8 3/4] vhost-user block device backend server


On Thu, Jun 11, 2020 at 04:24:52PM +0100, Stefan Hajnoczi wrote:

On Fri, Jun 05, 2020 at 07:35:37AM +0800, Coiby Xu wrote:

+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);


elem was allocated with enough space for VuBlockReq. Can this allocation
be eliminated?

 typedef struct VuBlockReq {
- VuVirtqElement *elem;
+ VuVirtqElement elem;
 int64_t sector_num;
 size_t size;
 struct virtio_blk_inhdr *in;
 struct virtio_blk_outhdr out;
 VuServer *server;
 struct VuVirtq *vq;
 } VuBlockReq;


Thank you for review this patch. Other issues for this patch have been
addressed in v9 except for this one. I'm not sure what you mean. I can't
find a way that doesn't require to allocate a VuBlockReq struct.


--
Best regards,
Coiby

Re: [PATCH v8 0/4] vhost-user block device backend implementation


Hi Stefano Garzarella,

On Thu, Jun 11, 2020 at 02:37:03PM +0200, Stefano Garzarella wrote:

Hi Coiby Xu,

On Fri, Jun 05, 2020 at 07:35:34AM +0800, Coiby Xu wrote:

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
  Allow vu_message_read to be replaced
  generic vhost user server
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs|   1 +
 block/export/vhost-user-blk-server.c   | 716 
 block/export/vhost-user-blk-server.h   |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  11 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 130 
 tests/qtest/libqos/vhost-user-blk.h|  44 ++
 tests/qtest/libqtest.c |  54 +-
 tests/qtest/libqtest.h |  38 ++
 tests/qtest/vhost-user-blk-test.c  | 737 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   1 +
 util/vhost-user-server.c   | 406 
 util/vhost-user-server.h   |  59 ++
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h



Should we add an entry in the MAINTAINERS file for some of the new files?
(e.g. util/vhost-user-server.*)


Yes, please. Thank you!

--
Best regards,
Coiby

Re: [PATCH v8 2/4] generic vhost user server


On Thu, Jun 11, 2020 at 02:14:49PM +0100, Stefan Hajnoczi wrote:

On Fri, Jun 05, 2020 at 07:35:36AM +0800, Coiby Xu wrote:

+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+ioc = server->ioc;
+} else {
+/* Slave communication will also use this function to read msg */
+ioc = slave_io_channel(server, conn_fd, _err);
+}
+
+if (!ioc) {
+error_report_err(local_err);
+goto fail;
+}
+
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_err(local_err);
+return false;
+}
+}
+read_bytes += rc;
+if (nfds_t > 0) {
+fds = g_renew(int, fds, nfds + nfds_t);
+memcpy(fds + nfds, fds_t, nfds_t *sizeof(int));
+nfds += nfds_t;
+if (nfds > VHOST_MEMORY_MAX_NREGIONS) {
+error_report("A maximum of %d fds are allowed, "
+ "however got %lu fds now",
+ VHOST_MEMORY_MAX_NREGIONS, nfds);
+goto fail;
+}
+g_free(fds_t);


I'm not sure why the temporary fds[] array is necessary. Copying the fds
directly into vmsg->fds would be simpler:

 if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
 error_report("A maximum of %d fds are allowed, "
  "however got %lu fds now",
  VHOST_MEMORY_MAX_NREGIONS, nfds);
 goto fail;
 }
 memcpy(vmsg->fds + nfds, fds_t, nfds_t * sizeof(vds->fds[0]));
 nfds += nfds_t;

Did I misunderstand how this works?


No, the temporary fds[] array is not necessary. Thank for the
simplication!


+}
+if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+break;
+}
+iov.iov_base = (char *)vmsg + read_bytes;
+iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
+} while (true);
+
+vmsg->fd_num = nfds;
+if (nfds > 0) {
+memcpy(vmsg->fds, fds, nfds * sizeof(int));
+}
+g_free(fds);
+/* qio_channel_readv_full will make socket fds blocking, unblock them */
+vmsg_unblock_fds(vmsg);
+if (vmsg->size > sizeof(vmsg->payload)) {
+error_report("Error: too big message request: %d, "
+ "size: vmsg->size: %u, "
+ "while sizeof(vmsg->payload) = %zu",
+ vmsg->request, vmsg->size, sizeof(vmsg->payload));
+goto fail;
+}
+
+struct iovec iov_payload = {
+.iov_base = (char *)>payload,
+.iov_len = vmsg->size,
+};
+if (vmsg->size) {
+rc = qio_channel_readv_all_eof(ioc, _payload, 1, _err);
+if (rc == -1) {
+error_report_err(local_err);
+goto fail;
+}
+}
+
+return true;
+
+fail:
+vmsg_close_fds(vmsg);
+
+return false;
+}
+
+
+static void vu_client_start(VuServer *server);
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+VuServer *server = opaque;
+
+while (!server->aio_context_changed && server->sioc) {
+vu_dispatch(>vu_dev);
+}
+
+if (server->aio_context_changed && server->sioc) {
+server->aio_context_changed = false;
+vu_client_start(server);
+}
+}
+
+static void vu_client_start(VuServer *server)
+{
+server->co_trip = qemu_coroutine_create(vu_client_trip, server);
+aio_co_enter(server->ctx, server->co_trip);
+}
+
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handl

[PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c  |  35 +-
 tests/qtest/libqtest.h  |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index c2397de8ed..303235b40f 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -638,7 +638,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index 9e5a51d033..b6f081cb26 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..3de9c59194
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,130 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+

[PATCH v9 1/5] Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 11 ++-
 contrib/libvhost-user/libvhost-user.h  | 21 +
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);
 
-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 3bca996c62..0c7368baa2 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }
 
-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!dev->read_msg(dev, dev->slave_fd, _reply)) {
 goto out;
 }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;
 
-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index f30394fab6..d756da8548 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;
 
+/* @read_msg: custom method to read vhost-user message
+ *
+ * Read data from vhost_user socket fd and fill up
+ * the passed VhostUserMsg *vmsg struct.
+ *
+ * If reading fails, it should close the received set of file
+ * descriptors as socket message's auxiliary data.
+ *
+ * For the details, please refer to vu_message_read in libvhost-user.c
+ * which will be used by default if not custom method is provided when
+ * calling vu_init
+ *
+ * Returns: true if vhost-user message successfully received,
+

[PATCH v9 4/5] vhost-user block device backend server

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   1 +
 block/export/vhost-user-blk-server.c | 669 +++
 block/export/vhost-user-blk-server.h |  35 ++
 softmmu/vl.c |   4 +
 4 files changed, 709 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..0eb7eff470 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..bbf2ceaa9b
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,669 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#include "util/block-helpers.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64_to_cpu(desc.sector) << 9,
+  le32_to_cpu(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->server = server;
+req->vq = vq;
+req->elem = elem;
+
+if (unlikely(io

[PATCH v9 0/5] vhost-user block device backend implementation

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.


Coiby Xu (5):
  Allow vu_message_read to be replaced
  generic vhost user server
  move logical block size check function to a common utility function
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs|   1 +
 block/export/vhost-user-blk-server.c   | 669 +++
 block/export/vhost-user-blk-server.h   |  35 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  11 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 hw/core/qdev-properties.c  |  18 +-
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 130 
 tests/qtest/libqos/vhost-user-blk.h|  48 ++
 tests/qtest/libqtest.c |  35 +-
 tests/qtest/libqtest.h |  17 +
 tests/qtest/vhost-user-blk-test.c  | 739 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   2 +
 util/block-helpers.c   |  46 ++
 util/block-helpers.h   |   7 +
 util/vhost-user-server.c   | 400 +++
 util/vhost-user-server.h   |  61 ++
 22 files changed, 2231 insertions(+), 26 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.27.0

[PATCH v9 2/5] generic vhost user server

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   1 +
 util/vhost-user-server.c | 400 +++
 util/vhost-user-server.h |  61 ++
 3 files changed, 462 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index cc5e37177a..b4d4af06dc 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += qemu-sockets.o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..393beeb6b9
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,400 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+vu_deinit(>vu_dev);
+object_unref(OBJECT(server->sioc));
+object_unref(OBJECT(server->ioc));
+server->sioc_slave = NULL;
+object_unref(OBJECT(server->ioc_slave));
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+server->sioc = NULL;
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+Error **local_err)
+{
+if (server->sioc_slave) {
+if (fd == server->sioc_slave->fd) {
+return server->ioc_slave;
+}
+} else {
+server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+if (!*local_err) {
+server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+return server->ioc_slave;
+}
+}
+
+return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+ioc = server->ioc;
+} else {
+/* Slave communication will also use this function to read msg */
+ioc = slave_io_channel(server, conn_fd, _err);
+}
+
+if (!ioc) {
+error_report_err(local_err);
+goto fail;
+}
+
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_err(local_err);
+

[PATCH v9 3/5] move logical block size check function to a common utility function

Move logical block size check function in 
hw/core/qdev-properties.c:set_blocksize() to util/block-helpers.c

Signed-off-by: Coiby Xu 
---
 hw/core/qdev-properties.c | 18 +++
 util/Makefile.objs|  1 +
 util/block-helpers.c  | 46 +++
 util/block-helpers.h  |  7 ++
 4 files changed, 57 insertions(+), 15 deletions(-)
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h

diff --git a/hw/core/qdev-properties.c b/hw/core/qdev-properties.c
index cc924815da..a4a6aa5204 100644
--- a/hw/core/qdev-properties.c
+++ b/hw/core/qdev-properties.c
@@ -14,6 +14,7 @@
 #include "qapi/visitor.h"
 #include "chardev/char.h"
 #include "qemu/uuid.h"
+#include "util/block-helpers.h"
 
 void qdev_prop_set_after_realize(DeviceState *dev, const char *name,
   Error **errp)
@@ -736,8 +737,6 @@ static void set_blocksize(Object *obj, Visitor *v, const 
char *name,
 Property *prop = opaque;
 uint16_t value, *ptr = qdev_get_prop_ptr(dev, prop);
 Error *local_err = NULL;
-const int64_t min = 512;
-const int64_t max = 32768;
 
 if (dev->realized) {
 qdev_prop_set_after_realize(dev, name, errp);
@@ -749,21 +748,10 @@ static void set_blocksize(Object *obj, Visitor *v, const 
char *name,
 error_propagate(errp, local_err);
 return;
 }
-/* value of 0 means "unset" */
-if (value && (value < min || value > max)) {
-error_setg(errp, QERR_PROPERTY_VALUE_OUT_OF_RANGE,
-   dev->id ? : "", name, (int64_t)value, min, max);
+check_logical_block_size(dev->id ? : "", name, value, errp);
+if (errp) {
 return;
 }
-
-/* We rely on power-of-2 blocksizes for bitmasks */
-if ((value & (value - 1)) != 0) {
-error_setg(errp,
-  "Property %s.%s doesn't take value '%" PRId64 "', it's not a 
power of 2",
-  dev->id ?: "", name, (int64_t)value);
-return;
-}
-
 *ptr = value;
 }
 
diff --git a/util/Makefile.objs b/util/Makefile.objs
index b4d4af06dc..fa5380ddab 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-y += block-helpers.o
 util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
diff --git a/util/block-helpers.c b/util/block-helpers.c
new file mode 100644
index 00..d31309cc0e
--- /dev/null
+++ b/util/block-helpers.c
@@ -0,0 +1,46 @@
+/*
+ * Block utility functions
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "block-helpers.h"
+
+/*
+ * Logical block size input validation
+ *
+ * The size should meet the following conditions:
+ * 1. min=512
+ * 2. max=32768
+ * 3. a power of 2
+ *
+ *  Moved from hw/core/qdev-properties.c:set_blocksize()
+ */
+void check_logical_block_size(const char *id, const char *name, uint16_t value,
+ Error **errp)
+{
+const int64_t min = 512;
+const int64_t max = 32768;
+
+/* value of 0 means "unset" */
+if (value && (value < min || value > max)) {
+error_setg(errp, QERR_PROPERTY_VALUE_OUT_OF_RANGE,
+   id, name, (int64_t)value, min, max);
+return;
+}
+
+/* We rely on power-of-2 blocksizes for bitmasks */
+if ((value & (value - 1)) != 0) {
+error_setg(errp,
+   "Property %s.%s doesn't take value '%" PRId64
+   "', it's not a power of 2",
+   id, name, (int64_t)value);
+return;
+}
+}
diff --git a/util/block-helpers.h b/util/block-helpers.h
new file mode 100644
index 00..f06be282a1
--- /dev/null
+++ b/util/block-helpers.h
@@ -0,0 +1,7 @@
+#ifndef BLOCK_HELPERS_H
+#define BLOCK_HELPERS_H
+
+void check_logical_block_size(const char *id, const char *name, uint16_t value,
+ Error **errp);
+
+#endif /* BLOCK_HELPERS_H */
-- 
2.27.0

Re: [PATCH v8 0/4] vhost-user block device backend implementation

2020-06-12 Thread Coiby Xu


On Thu, Jun 11, 2020 at 04:27:44PM +0100, Stefan Hajnoczi wrote:

On Fri, Jun 05, 2020 at 07:35:34AM +0800, Coiby Xu wrote:

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue


Great, thanks for posting these patches!

I have posted feedback. I'd like to merge this soon. If you are busy I
can send you patches that address the comments I've made, please let me
know.


Thank you for reviewing my work! I'll post v9 to address all the comments this
weekend, does you think it's soon enough?

Best regards,
Coiby

Re: [PATCH v2 08/20] qemu-storage-daemon: Add --object option

2020-06-09 Thread Coiby Xu


On Mon, Feb 24, 2020 at 03:29:56PM +0100, Kevin Wolf wrote:

Add a command line option to create user-creatable QOM objects.

Signed-off-by: Kevin Wolf 
---
qemu-storage-daemon.c | 47 +++
Makefile.objs |  2 +-
qom/Makefile.objs |  1 +
3 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index c30caaf59e..0cd8144c81 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -33,15 +33,19 @@
#include "qapi/error.h"
#include "qapi/qapi-visit-block-core.h"
#include "qapi/qapi-commands-block-core.h"
+#include "qapi/qmp/qdict.h"
#include "qapi/qobject-input-visitor.h"

#include "qemu-common.h"
#include "qemu-version.h"
#include "qemu/config-file.h"
#include "qemu/error-report.h"
+#include "qemu/help_option.h"
#include "qemu/log.h"
#include "qemu/main-loop.h"
#include "qemu/module.h"
+#include "qemu/option.h"
+#include "qom/object_interfaces.h"

#include "trace/control.h"

@@ -63,12 +67,31 @@ static void help(void)
" [,driver specific parameters...]\n"
" configure a block backend\n"
"\n"
+"  --object help  list object types that can be added\n"
+"  --object ,help   list properties for the given object type\n"
+"  --object [,=...]\n"
+" create a new object of type , setting\n"
+" properties in the order they are specified. Note\n"
+" that the 'id' property must be set.\n"
+" See the qemu(1) man page for documentation of the\n"
+" objects that can be added.\n"
+"\n"
QEMU_HELP_BOTTOM "\n",
error_get_progname());
}

enum {
OPTION_BLOCKDEV = 256,
+OPTION_OBJECT,
+};
+
+static QemuOptsList qemu_object_opts = {
+.name = "object",
+.implied_opt_name = "qom-type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+.desc = {
+{ }
+},
};


I'm not sure whether qemu-storage-daemon's QMP monitor is supposed to support
`object-del` or not. If it's yes, we should add 
`qemu_add_opts(_object_opts);`,
otherwise, qemu-storage-daemon will exit with the following error,


   Unexpected error in find_list() at util/qemu-config.c:24:

The steps to reproduce this error,

  1. start qemu-storage-daemon

./qemu-storage-daemon --object iothread,id=iothread0 --chardev 
socket,id=qmp,path=abc,server,nowait --monitor chardev=qmp

  2. send object-del command

echo '{ "execute": "qmp_capabilities" }'\
'{ 'object-del', 'arguments': {'id': 'iothread0'} }'\
'{ "execute": "quit" } ' | socat - UNIX-CONNECT:abc

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

2020-06-05 Thread Coiby Xu


On Fri, Jun 05, 2020 at 11:25:26AM +0200, Thomas Huth wrote:

On 05/06/2020 08.22, Coiby Xu wrote:

On Fri, Jun 05, 2020 at 07:01:33AM +0200, Thomas Huth wrote:

diff --git a/tests/qtest/libqos/vhost-user-blk.h
b/tests/qtest/libqos/vhost-user-blk.h
new file mode 100644
index 00..ef4ef09cca
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.h
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito

+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.


... but you've missed the header here :-(


Thank you for reminding me of this issue!


diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index 49075b55a1..a7b7c96206 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -31,40 +31,9 @@
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qstring.h"

-#define MAX_IRQ 256
 #define SOCKET_TIMEOUT 50
 #define SOCKET_MAX_FDS 16

-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-    QTestSendFn send;  /* for sending qtest commands */
-
-    /*
- * use external_send to send qtest command strings through
functions which
- * do not accept a QTestState as the first parameter.
- */
-    ExternalSendFn  external_send;
-
-    QTestRecvFn recv_line; /* for receiving qtest command
responses */
-} QTestTransportOps;
-
-struct QTestState
-{
-    int fd;
-    int qmp_fd;
-    pid_t qemu_pid;  /* our child QEMU process */
-    int wstatus;
-    int expected_status;
-    bool big_endian;
-    bool irq_level[MAX_IRQ];
-    GString *rx;
-    QTestTransportOps ops;
-};


Why do you have to move struct QTestState and friends to the header
instead? I'd prefer if we could keep it here if possible?


tests/qtest/vhost-user-blk-test.c needs to talk to qemu-storage-daemon's
QMP. Thus I g_new0 a QTestState struct to make use of related functions
like qtest_qmp and this requires the QTestState struct definition.


Hm, ok, could that maybe be solved by introducing a wrapper function to
libqtest.c instead? Something like qtest_create_state_with_qmp_fd() or so?
Moving a define with a generic name like MAX_IRQ to a header really does
not sound like a good idea to me, so if that idea with the wrapper
function does not work out, could you please at least rename MAX_IRQ to
QTEST_MAX_IRQ or something similar?

I didn't realize the QTestState struct is supposed to be hidden from the user 
and
not directly accessible. To typedef a struct in a header file and define
the struct in the c file is a new trick for me:)

This idea of creating a wrapper function qtest_create_state_with_qmp_fd
works as expected. Thank you!

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

2020-06-05 Thread Coiby Xu


On Fri, Jun 05, 2020 at 07:01:33AM +0200, Thomas Huth wrote:

diff --git a/tests/qtest/libqos/vhost-user-blk.h 
b/tests/qtest/libqos/vhost-user-blk.h
new file mode 100644
index 00..ef4ef09cca
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.h
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.


... but you've missed the header here :-(


Thank you for reminding me of this issue!


diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index 49075b55a1..a7b7c96206 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -31,40 +31,9 @@
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qstring.h"

-#define MAX_IRQ 256
 #define SOCKET_TIMEOUT 50
 #define SOCKET_MAX_FDS 16

-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-QTestSendFn send;  /* for sending qtest commands */
-
-/*
- * use external_send to send qtest command strings through functions which
- * do not accept a QTestState as the first parameter.
- */
-ExternalSendFn  external_send;
-
-QTestRecvFn recv_line; /* for receiving qtest command responses */
-} QTestTransportOps;
-
-struct QTestState
-{
-int fd;
-int qmp_fd;
-pid_t qemu_pid;  /* our child QEMU process */
-int wstatus;
-int expected_status;
-bool big_endian;
-bool irq_level[MAX_IRQ];
-GString *rx;
-QTestTransportOps ops;
-};


Why do you have to move struct QTestState and friends to the header
instead? I'd prefer if we could keep it here if possible?


tests/qtest/vhost-user-blk-test.c needs to talk to qemu-storage-daemon's
QMP. Thus I g_new0 a QTestState struct to make use of related functions
like qtest_qmp and this requires the QTestState struct definition.

[PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/libqtest.c  |  54 +-
 tests/qtest/libqtest.h  |  38 ++
 tests/qtest/vhost-user-blk-test.c   | 737 
 7 files changed, 976 insertions(+), 32 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e3d6370df..d8578346b0 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -636,7 +636,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index 9e5a51d033..b6f081cb26 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..3de9c59194
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,130 @@
+/*
+ * libqos driver framework
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu 
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+

[PATCH v8 3/4] vhost-user block device backend server

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   1 +
 block/export/vhost-user-blk-server.c | 716 +++
 block/export/vhost-user-blk-server.h |  34 ++
 softmmu/vl.c |   4 +
 4 files changed, 755 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..0eb7eff470 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..a9dec0625f
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,716 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->server = ser

[PATCH v8 2/4] generic vhost user server

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   1 +
 util/vhost-user-server.c | 406 +++
 util/vhost-user-server.h |  59 ++
 3 files changed, 466 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index fe339c2636..f54a6c80ec 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -40,6 +40,7 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..8fafd97bdc
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,406 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+vu_deinit(>vu_dev);
+server->sioc = NULL;
+object_unref(OBJECT(server->ioc));
+
+server->sioc_slave = NULL;
+object_unref(OBJECT(server->ioc_slave));
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+Error **local_err)
+{
+if (server->sioc_slave) {
+if (fd == server->sioc_slave->fd) {
+return server->ioc_slave;
+}
+} else {
+server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+if (!*local_err) {
+server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+return server->ioc_slave;
+}
+}
+
+return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+ioc = server->ioc;
+} else {
+/* Slave communication will also use this function to read msg */
+ioc = slave_io_channel(server, conn_fd, _err);
+}
+
+if (!ioc) {
+error_report_err(local_err);
+goto fail;
+}
+
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_er

[PATCH v8 1/4] Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 11 ++-
 contrib/libvhost-user/libvhost-user.h  | 21 +
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);
 
-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 3bca996c62..0c7368baa2 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }
 
-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!dev->read_msg(dev, dev->slave_fd, _reply)) {
 goto out;
 }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;
 
-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index f30394fab6..d756da8548 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;
 
+/* @read_msg: custom method to read vhost-user message
+ *
+ * Read data from vhost_user socket fd and fill up
+ * the passed VhostUserMsg *vmsg struct.
+ *
+ * If reading fails, it should close the received set of file
+ * descriptors as socket message's auxiliary data.
+ *
+ * For the details, please refer to vu_message_read in libvhost-user.c
+ * which will be used by default if not custom method is provided when
+ * calling vu_init
+ *
+ * Returns: true if vhost-user message successfully received,
+

[PATCH v8 0/4] vhost-user block device backend implementation

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
  Allow vu_message_read to be replaced
  generic vhost user server
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs|   1 +
 block/export/vhost-user-blk-server.c   | 716 
 block/export/vhost-user-blk-server.h   |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  11 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 130 
 tests/qtest/libqos/vhost-user-blk.h|  44 ++
 tests/qtest/libqtest.c |  54 +-
 tests/qtest/libqtest.h |  38 ++
 tests/qtest/vhost-user-blk-test.c  | 737 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   1 +
 util/vhost-user-server.c   | 406 
 util/vhost-user-server.h   |  59 ++
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.26.2

Re: [PATCH v6 4/4] new qTest case to test the vhost-user-blk-server


On Thu, Jun 04, 2020 at 03:49:13PM +0200, Thomas Huth wrote:

On 04/06/2020 15.42, Thomas Huth wrote:

+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 


Don't you want to add a remark here for you, too?


+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.


Could you please change "version 2" into "verion 2.1" ? There never was
a "version 2" of the LGPL, only version 2.1 (or 3.x).


I meant, there never was a version 2 of the "Lesser GPL". Version 2.0 of
the license was still called "Library GPL" instead.

Thomas


Thanks for reviewing this work! Just missed this email while submitting
v7. I will fix them in v8.

[PATCH v7 2/4] generic vhost user server

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   1 +
 util/vhost-user-server.c | 406 +++
 util/vhost-user-server.h |  59 ++
 3 files changed, 466 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index fe339c2636..f54a6c80ec 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -40,6 +40,7 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..8fafd97bdc
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,406 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+vu_deinit(>vu_dev);
+server->sioc = NULL;
+object_unref(OBJECT(server->ioc));
+
+server->sioc_slave = NULL;
+object_unref(OBJECT(server->ioc_slave));
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+Error **local_err)
+{
+if (server->sioc_slave) {
+if (fd == server->sioc_slave->fd) {
+return server->ioc_slave;
+}
+} else {
+server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+if (!*local_err) {
+server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+return server->ioc_slave;
+}
+}
+
+return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+ioc = server->ioc;
+} else {
+/* Slave communication will also use this function to read msg */
+ioc = slave_io_channel(server, conn_fd, _err);
+}
+
+if (!ioc) {
+error_report_err(local_err);
+goto fail;
+}
+
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_er

[PATCH v7 4/4] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 126 +
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/libqtest.c  |  44 +-
 tests/qtest/libqtest.h  |  38 ++
 tests/qtest/vhost-user-blk-test.c   | 743 
 7 files changed, 968 insertions(+), 32 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e3d6370df..d8578346b0 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -636,7 +636,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index 9e5a51d033..b6f081cb26 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..ec46b7ddb4
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,126 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVho

[PATCH v7 3/4] vhost-user block device backend server

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   1 +
 block/export/vhost-user-blk-server.c | 716 +++
 block/export/vhost-user-blk-server.h |  34 ++
 softmmu/vl.c |   4 +
 4 files changed, 755 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..0eb7eff470 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..a9dec0625f
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,716 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->server = ser

[PATCH v7 0/4] vhost-user block device backend implementation

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (5):
  allow vu_message_read to be replaced
  generic vhost user server
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs|   1 +
 block/export/vhost-user-blk-server.c   | 716 
 block/export/vhost-user-blk-server.h   |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  11 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 126 
 tests/qtest/libqos/vhost-user-blk.h|  44 ++
 tests/qtest/libqtest.c |  44 +-
 tests/qtest/libqtest.h |  38 ++
 tests/qtest/vhost-user-blk-test.c  | 743 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   1 +
 util/vhost-user-server.c   | 406 +++
 util/vhost-user-server.h   |  59 ++
 19 files changed, 2221 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.26.2

[PATCH v7 1/4] Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 11 ++-
 contrib/libvhost-user/libvhost-user.h  | 21 +
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);
 
-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 3bca996c62..0c7368baa2 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }
 
-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!dev->read_msg(dev, dev->slave_fd, _reply)) {
 goto out;
 }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;
 
-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index f30394fab6..d756da8548 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;
 
+/* @read_msg: custom method to read vhost-user message
+ *
+ * Read data from vhost_user socket fd and fill up
+ * the passed VhostUserMsg *vmsg struct.
+ *
+ * If reading fails, it should close the received set of file
+ * descriptors as socket message's auxiliary data.
+ *
+ * For the details, please refer to vu_message_read in libvhost-user.c
+ * which will be used by default if not custom method is provided when
+ * calling vu_init
+ *
+ * Returns: true if vhost-user message successfully received,
+

[PATCH v6 3/4] vhost-user block device backend server

By making use of libvhost-user, block device drive can be shared to
the connected vhost-user client. Only one client can connect to the
server one time.

Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   1 +
 block/export/vhost-user-blk-server.c | 715 +++
 block/export/vhost-user-blk-server.h |  34 ++
 softmmu/vl.c |   4 +
 4 files changed, 754 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..0eb7eff470 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..630aeb4ac4
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,715 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuServer *server;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >server->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuServer *server;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuServer *server = data->server;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->server = ser

[PATCH v6 2/4] generic vhost user server

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   1 +
 util/vhost-user-server.c | 404 +++
 util/vhost-user-server.h |  59 ++
 3 files changed, 464 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index fe339c2636..f54a6c80ec 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -40,6 +40,7 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..414d0d4e47
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,404 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+  gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+vu_deinit(>vu_dev);
+server->sioc = NULL;
+object_unref(OBJECT(server->ioc));
+
+server->sioc_slave = NULL;
+object_unref(OBJECT(server->ioc_slave));
+/*
+ * Set the callback function for network listener so another
+ * vhost-user client can connect to this server
+ */
+qio_net_listener_set_client_func(server->listener,
+ vu_accept,
+ server,
+ NULL);
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (server->sioc) {
+close_client(server);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(server);
+}
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+Error **local_err)
+{
+if (server->sioc_slave) {
+if (fd == server->sioc_slave->fd) {
+return server->ioc_slave;
+}
+} else {
+server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+if (!*local_err) {
+server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+return server->ioc_slave;
+}
+}
+
+return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+Error *local_err = NULL;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+QIOChannel *ioc = NULL;
+
+if (conn_fd == server->sioc->fd) {
+ioc = server->ioc;
+} else {
+/* Slave communication will also use this function to read msg */
+ioc = slave_io_channel(server, conn_fd, _err);
+}
+
+if (!ioc) {
+error_report_err(local_err);
+goto fail;
+}
+
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_er

[PATCH v6 1/4] Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 11 ++-
 contrib/libvhost-user/libvhost-user.h  | 21 +
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);
 
-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 3bca996c62..0c7368baa2 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }
 
-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!dev->read_msg(dev, dev->slave_fd, _reply)) {
 goto out;
 }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;
 
-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index f30394fab6..d756da8548 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;
 
+/* @read_msg: custom method to read vhost-user message
+ *
+ * Read data from vhost_user socket fd and fill up
+ * the passed VhostUserMsg *vmsg struct.
+ *
+ * If reading fails, it should close the received set of file
+ * descriptors as socket message's auxiliary data.
+ *
+ * For the details, please refer to vu_message_read in libvhost-user.c
+ * which will be used by default if not custom method is provided when
+ * calling vu_init
+ *
+ * Returns: true if vhost-user message successfully received,
+

[PATCH v6 4/4] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize. Since vhost-user server can only server one
client one time, two instances of qemu-storage-daemon are launched
for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will
send "quit" command to qemu-storage-daemon's QMP monitor. So a function
is added to libqtest.c to establish socket connection with socket
server.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 126 +
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/libqtest.c  |  44 +-
 tests/qtest/libqtest.h  |  38 ++
 tests/qtest/vhost-user-blk-test.c   | 741 
 7 files changed, 966 insertions(+), 32 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 03a74b60f6..f7136b2fc6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -636,7 +636,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index 9e5a51d033..b6f081cb26 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..ec46b7ddb4
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,126 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVho

[PATCH v6 0/4] vhost-user block device backend implementation

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - a bug fix in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

Coiby Xu (4):
  Allow vu_message_read to be replaced
  generic vhost user server
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs|   1 +
 block/export/vhost-user-blk-server.c   | 715 
 block/export/vhost-user-blk-server.h   |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  11 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 126 
 tests/qtest/libqos/vhost-user-blk.h|  44 ++
 tests/qtest/libqtest.c |  44 +-
 tests/qtest/libqtest.h |  38 ++
 tests/qtest/vhost-user-blk-test.c  | 741 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   1 +
 util/vhost-user-server.c   | 404 +++
 util/vhost-user-server.h   |  59 ++
 19 files changed, 2216 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.26.2

Re: [PATCH v2 08/20] qemu-storage-daemon: Add --object option

2020-04-15 Thread Coiby Xu


Hi Kevin,

I tried to start vhost-user block device backend server with the
following command,
  $ qemu-storage-daemon --blockdev 
driver=file,node-name=disk,filename=file.img,read-only=off \
  --object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vu_blk.socket,node-name=disk,writable=off

but the following error occurred,

  qemu-storage-daemon: Invalid parameter type for 'writable', expected: boolean

I notice qmp_object_add is used. Should we use user_creatable_add_opts
instead?

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index dd128978cc..9d21aaaf54 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -277,8 +277,6 @@ static void process_options(int argc, char *argv[])
 {
 QemuOpts *opts;
 const char *type;
-QDict *args;
-QObject *ret_data = NULL;

 /* FIXME The keyval parser rejects 'help' arguments, so we must
  * unconditionall try QemuOpts first. */
@@ -288,12 +286,11 @@ static void process_options(int argc, char *argv[])
 if (type && user_creatable_print_help(type, opts)) {
 exit(EXIT_SUCCESS);
 }
+
+
+user_creatable_add_opts(opts, _fatal);
 qemu_opts_del(opts);

-args = keyval_parse(optarg, "qom-type", _fatal);
-qmp_object_add(args, _data, _fatal);
-qobject_unref(args);
-qobject_unref(ret_data);
 break;
 }
 default:

On Mon, Feb 24, 2020 at 03:29:56PM +0100, Kevin Wolf wrote:

Add a command line option to create user-creatable QOM objects.

Signed-off-by: Kevin Wolf 
---
qemu-storage-daemon.c | 47 +++
Makefile.objs |  2 +-
qom/Makefile.objs |  1 +
3 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index c30caaf59e..0cd8144c81 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -33,15 +33,19 @@
#include "qapi/error.h"
#include "qapi/qapi-visit-block-core.h"
#include "qapi/qapi-commands-block-core.h"
+#include "qapi/qmp/qdict.h"
#include "qapi/qobject-input-visitor.h"

#include "qemu-common.h"
#include "qemu-version.h"
#include "qemu/config-file.h"
#include "qemu/error-report.h"
+#include "qemu/help_option.h"
#include "qemu/log.h"
#include "qemu/main-loop.h"
#include "qemu/module.h"
+#include "qemu/option.h"
+#include "qom/object_interfaces.h"

#include "trace/control.h"

@@ -63,12 +67,31 @@ static void help(void)
" [,driver specific parameters...]\n"
" configure a block backend\n"
"\n"
+"  --object help  list object types that can be added\n"
+"  --object ,help   list properties for the given object type\n"
+"  --object [,=...]\n"
+" create a new object of type , setting\n"
+" properties in the order they are specified. Note\n"
+" that the 'id' property must be set.\n"
+" See the qemu(1) man page for documentation of the\n"
+" objects that can be added.\n"
+"\n"
QEMU_HELP_BOTTOM "\n",
error_get_progname());
}

enum {
OPTION_BLOCKDEV = 256,
+OPTION_OBJECT,
+};
+
+static QemuOptsList qemu_object_opts = {
+.name = "object",
+.implied_opt_name = "qom-type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+.desc = {
+{ }
+},
};

static void process_options(int argc, char *argv[])
@@ -78,6 +101,7 @@ static void process_options(int argc, char *argv[])
static const struct option long_options[] = {
{"blockdev", required_argument, NULL, OPTION_BLOCKDEV},
{"help", no_argument, NULL, 'h'},
+{"object", required_argument, NULL, OPTION_OBJECT},
{"trace", required_argument, NULL, 'T'},
{"version", no_argument, NULL, 'V'},
{0, 0, 0, 0}
@@ -121,6 +145,29 @@ static void process_options(int argc, char *argv[])
qapi_free_BlockdevOptions(options);
break;
}
+case OPTION_OBJECT:
+{
+QemuOpts *opts;
+const char *type;
+QDict *args;
+QObject *ret_data = NULL;
+
+/* FIXME The keyval parser rejects 'help' arguments, so we must
+ * unconditionall try QemuOpts first. */
+opts = qemu_opts_parse(_object_opts,
+   optarg, true, _fatal);
+type = qemu_opt_get(opts, "qom-type");
+if (type && user_creatable_print_help(type, opts)) {
+exit(EXIT_SUCCESS);
+}
+qemu_opts_del(opts);
+
+args = keyval_parse(optarg, "qom-type", _fatal);
+qmp_object_add(args, _data, _fatal);
+

Replace GSource with AioContext for chardev

2020-04-09 Thread Coiby Xu

Hi,

I'm now implementing vhost-user block device backend
https://patchew.org/QEMU/20200309100342.14921-1-coiby...@gmail.com/
and want to use chardev to help manage vhost-user client connections
and read socket message. However there are two issues that need to be
addressed.

Firstly, chardev isn't suitable for the case when exported drive is
run in an IOThread because for mow chardev use GSource to dispatch
socket fd events. So I have to specify which IOThread the exported
drive is using when launching vhost-user block device backend,
for example, the following syntax will be used,

-drive file=file.img,id=disk -device virtio-blk,drive=disk,iothread=iothread0
\
-object vhost-user-blk-server,node-name=disk,chardev=mon1,iothread=iothread0
\
-object iothread,id=iothread0 \
-chardev socket,id=mon1,path=/tmp/vhost-user-blk_vhost.socket,server,nowait

then iothread_get_g_main_context(IOThread *iothread) has to be called
to run the gcontext in IOThread. If we use AioContext to dispatch socket
fd events, we needn't to specify IOThread twice. Besides aio_poll is faster
than g_main_loop_run.

Secondly, socket chardev's async read handler (set through
qemu_chr_fe_set_handlers) doesn't take the case of socket short read
into consideration. I plan to add one which will make use qio_channel_yield.

According to
[1] Improving the QEMU Event Loop - Linux Foundation Events
http://events17.linuxfoundation.org/sites/events/files/slides/Improving%20the%20QEMU%20Event%20Loop%20-%203.pdf

"Convert chardev GSource to aio or an equivalent source" (p.30) should have
been finished. I'm curious why the plan didn't continue. If it's desirable,
I'm going to finish the leftover work to resolve the aforementioned two issues.

Any suggestion will be appreciated.
Thank you!

[PATCH v5 2/5] generic vhost user server

Sharing QEMU devices via vhost-user protocol

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   3 +
 util/vhost-user-server.c | 360 +++
 util/vhost-user-server.h |  57 ++
 3 files changed, 420 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index 6b38b67cf1..d207b5f981 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -37,6 +37,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..7da96bc815
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,360 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+
+static void close_client(VuClientInfo *client)
+{
+vu_deinit(>vu_dev);
+client->sioc = NULL;
+object_unref(OBJECT(client->ioc));
+client->closed = true;
+
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+VuClientInfo *client = container_of(vu_dev, VuClientInfo, vu_dev);
+VuServer *server = client->server;
+
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+if (!client->closed) {
+close_client(client);
+QTAILQ_REMOVE(>clients, client, next);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(client);
+}
+}
+
+
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+/*
+ * Store fds/nfds returned from qio_channel_readv_full into
+ * temporary variables.
+ *
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuClientInfo *client = container_of(vu_dev, VuClientInfo, vu_dev);
+QIOChannel *ioc = client->ioc;
+
+Error *local_err = NULL;
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, _err);
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report_err(local_err);
+return false;
+}
+}
+read_bytes += rc;
+fds = g_renew(int, fds_t, nfds + nfds_t);
+memcpy(fds + nfds, fds_t, nfds_t);
+nfds += nfds_t;
+if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+break;
+}
+} while (true);
+assert(nfds <= VHOST_MEMORY_MAX_NREGIONS);
+vmsg->fd_num = nfds;
+memcpy(vmsg->fds, fds, nfds * sizeof(int));
+g_free(fds);
+/* qio_channel_readv_full will make socket fds blocking, unblock them */
+vmsg_unblock_fds(vmsg);
+if (vmsg->size > sizeof(vmsg->payload)) {
+error_report("Error: too big message request: %d, "
+ "size: vmsg->size: %u, "
+ "while sizeof(vmsg->payload) = %zu",
+ vmsg->request, vmsg->size, sizeof(vmsg->payload));
+goto fail;
+}
+
+struct iovec iov_payload = {
+.iov_base = (char *)>payload,
+.iov_len = vmsg->size,
+};
+if (vmsg->size) {
+rc = qio_channel_readv_all_eof(ioc, _payload, 1, _err);
+if (rc == -1) {
+error_report_err(local_err);
+goto fail;
+}
+}
+
+return

[PATCH v5 5/5] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 126 +
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/vhost-user-blk-test.c   | 694 
 5 files changed, 868 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index edcbd475aa..0fccc7573e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -631,7 +631,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_VU_BINARY=./qemu-vu$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index 383b0ab217..868ff7ab1e 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -185,6 +185,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -228,6 +229,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..ec46b7ddb4
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,126 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVhostUserBlkDevice *vhost_user_blk = g_new0(QVhostUserBlkDevice, 1);
+QVhostUserBlk *interface = _user_blk->blk;
+
+interface->vdev = virtio_dev;
+
+vhost_user_blk->obj.get_driver = qvhost_user_blk_device_get_driver;
+
+return _user_blk->obj;
+}
+
+/* virtio-blk-pci */
+static void *qvhost_user_blk_pci_get_driver(void *object, const char 
*interface)
+{
+QVhostUserBlkPCI *v_blk = object;
+if (!g_strcmp0(interface, "pci-device")) {

[PATCH v5 1/5] allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the
QIOChannel functions. Thus reading vhost-user message won't stall the
guest.

Remove watch for kick fd in vu_deinit. When QEMU as vhost-user client
process is restarted in gdb, QEMU as vhost-user-server will still dispatch
the event. When vu_kick_cb is trying to access vq->kick_fd, a segmentation
fault will occur since vq has become a NULL pointer.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c  | 11 ++-
 contrib/libvhost-user/libvhost-user.h  |  6 ++
 tests/vhost-user-bridge.c  |  2 ++
 tools/virtiofsd/fuse_virtio.c  |  4 ++--
 5 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user-glib.c 
b/contrib/libvhost-user/libvhost-user-glib.c
index 53f1ca4cdd..0df2ec9271 100644
--- a/contrib/libvhost-user/libvhost-user-glib.c
+++ b/contrib/libvhost-user/libvhost-user-glib.c
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
 g_assert(dev);
 g_assert(iface);

-if (!vu_init(>parent, max_queues, socket, panic, set_watch,
+if (!vu_init(>parent, max_queues, socket, panic, NULL, set_watch,
  remove_watch, iface)) {
 return false;
 }
diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index 3bca996c62..0c7368baa2 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1

-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg 
*vmsg)
 goto out;
 }

-if (!vu_message_read(dev, dev->slave_fd, _reply)) {
+if (!dev->read_msg(dev, dev->slave_fd, _reply)) {
 goto out;
 }

@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg 
*vmsg)
 /* Wait for QEMU to confirm that it's registered the handler for the
  * faults.
  */
-if (!vu_message_read(dev, dev->sock, vmsg) ||
+if (!dev->read_msg(dev, dev->sock, vmsg) ||
 vmsg->size != sizeof(vmsg->payload.u64) ||
 vmsg->payload.u64 != 0) {
 vu_panic(dev, "failed to receive valid ack for postcopy 
set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
 int reply_requested;
 bool need_reply, success = false;

-if (!vu_message_read(dev, dev->sock, )) {
+if (!dev->read_msg(dev, dev->sock, )) {
 goto end;
 }

@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
 }

 if (vq->kick_fd != -1) {
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
 uint16_t max_queues,
 int socket,
 vu_panic_cb panic,
+vu_read_msg_cb read_msg,
 vu_set_watch_cb set_watch,
 vu_remove_watch_cb remove_watch,
 const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,

 dev->sock = socket;
 dev->panic = panic;
+dev->read_msg = read_msg ? read_msg : vu_message_read;
 dev->set_watch = set_watch;
 dev->remove_watch = remove_watch;
 dev->iface = iface;
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index 6fc8000e99..7e3f0c8598 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@

 #define VHOST_MEMORY_MAX_NREGIONS 8

+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,8 @@ struct VuDev {
 bool broken;
 uint16_t max_queues;

+/* allowing vu_read_msg_cb to be replaced */
+vu_read_msg_cb read_msg;
 /* @set_watch: add or update the given fd to the watch set,
  * call cb when condition is met */
 vu_set_watch_cb set_watch;
@@ -416,6 +421,7 @@ bool vu_init(VuDev *dev,
  uint16_t max_queues,

[PATCH v5 3/5] vhost-user block device backend server

By making use of libvhost, multiple block device drives can be exported
and each drive can serve multiple clients simultaneously.
Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 block/Makefile.objs  |   3 +
 block/export/vhost-user-blk-server.c | 727 +++
 block/export/vhost-user-blk-server.h |  21 +
 softmmu/vl.c |   4 +
 4 files changed, 755 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3bcb35c81d..5ffc5bb2ff 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,9 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o

 block-obj-y += nbd.o
+ifdef CONFIG_LINUX
+block-obj-y += export/vhost-user-blk-server.o 
../contrib/libvhost-user/libvhost-user.o
+endif
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
diff --git a/block/export/vhost-user-blk-server.c 
b/block/export/vhost-user-blk-server.c
new file mode 100644
index 00..f0e2bc6b7a
--- /dev/null
+++ b/block/export/vhost-user-blk-server.c
@@ -0,0 +1,727 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuClientInfo *client;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >client->vu_dev;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_client(VuClientInfo *client)
+{
+return container_of(client->server->ptr_in_device, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+struct req_data {
+VuClientInfo *client;
+VuVirtq *vq;
+VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+struct req_data *data = opaque;
+VuClientInfo *client = data->client;
+VuVirtq *vq = data->vq;
+VuVirtqElement *elem = data->elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(client);
+BlockBackend *backend = vdev_blk->backend;
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+unsigned in_num = elem->in_num;
+unsigned out_num = elem->out_num;
+/* refer to hw/block/virtio_blk.c */
+if (elem->out_num < 1 || elem->in_num < 1) {
+error_report("virtio-blk request missing headers");
+free(elem);
+return;
+}
+
+req = g_new0(VuBlockReq, 1);
+req->c

[PATCH v5 4/5] a standone-alone tool to directly share disk image file via vhost-user protocol

vhost-user-blk could have played as vhost-user backend but it only supports raw
file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES
operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real
block device).

Kevin's qemu-storage-daemon will be used to replace this
tool after his patches are merged into the mainline QEMU.

Signed-off-by: Coiby Xu 
---
 Makefile  |   4 +
 configure |   3 +
 qemu-vu.c | 252 ++
 3 files changed, 259 insertions(+)
 create mode 100644 qemu-vu.c

diff --git a/Makefile b/Makefile
index 9d4b224126..6da19d 100644
--- a/Makefile
+++ b/Makefile
@@ -585,6 +585,10 @@ qemu-img.o: qemu-img-cmds.h
 
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+
+ifdef CONFIG_LINUX
+qemu-vu$(EXESUF): qemu-vu.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+endif
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
diff --git a/configure b/configure
index fab6281eb7..53dd9fb88f 100755
--- a/configure
+++ b/configure
@@ -6318,6 +6318,9 @@ if test "$want_tools" = "yes" ; then
   if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
 tools="qemu-nbd\$(EXESUF) $tools"
   fi
+  if [ "$linux" = "yes" ] ; then
+tools="qemu-vu\$(EXESUF) $tools"
+  fi
   if [ "$ivshmem" = "yes" ]; then
 tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
   fi
diff --git a/qemu-vu.c b/qemu-vu.c
new file mode 100644
index 00..3d2e4df5eb
--- /dev/null
+++ b/qemu-vu.c
@@ -0,0 +1,252 @@
+/*
+ *  Copyright (C) 2020  Coiby Xu 
+ *
+ *  standone-alone vhost-user-blk device server backend
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; under version 2 of the License.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include 
+#include 
+#include "block/export/vhost-user-blk-server.h"
+#include "block/block_int.h"
+#include "io/net-listener.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qstring.h"
+#include "qemu/config-file.h"
+#include "qemu/cutils.h"
+#include "qemu/main-loop.h"
+#include "qemu/module.h"
+#include "qemu/option.h"
+#include "qemu-common.h"
+#include "qemu-version.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#define QEMU_VU_OPT_CACHE 256
+#define QEMU_VU_OPT_AIO   257
+#define QEMU_VU_OBJ_ID   "vu_disk"
+static QemuOptsList qemu_object_opts = {
+.name = "object",
+.implied_opt_name = "qom-type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+.desc = {
+{ }
+},
+};
+static char *srcpath;
+
+static void usage(const char *name)
+{
+(printf) (
+"Usage: %s [OPTIONS] FILE\n"
+"  or:  %s -L [OPTIONS]\n"
+"QEMU Vhost-user Server Utility\n"
+"\n"
+"  -h, --helpdisplay this help and exit\n"
+"  -V, --version output version information and exit\n"
+"\n"
+"Connection properties:\n"
+"  -k, --socket=PATH path to the unix socket\n"
+"\n"
+"General purpose options:\n"
+"  -e, -- exit-panic When the panic callback is called, the program\n"
+"will exit. Useful for make check-qtest.\n"
+"\n"
+"Block device options:\n"
+"  -f, --format=FORMAT   set image format (raw, qcow2, ...)\n"
+"  -r, --read-only   export read-only\n"
+"  -n, --nocache disable host cache\n"
+"  --cache=MODE  set cache mode (none, writeback, ...)\n"
+"  --aio=MODEset AIO mode (native or threads)\n"
+"\n"
+QEMU_HELP_BOTTOM "\n"
+, name, name);
+}
+
+s

[PATCH v5 0/5] vhost-user block device backend implementation

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feauture for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mouting it.

Coiby Xu (5):
  allow vu_message_read to be replaced
  generic vhost user server
  vhost-user block device backend server
  a standone-alone tool to directly share disk image file via vhost-user
protocol
  new qTest case to test the vhost-user-blk-server

 Makefile   |   4 +
 block/Makefile.objs|   3 +
 block/export/vhost-user-blk-server.c   | 727 +
 block/export/vhost-user-blk-server.h   |  21 +
 configure  |   3 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  10 +-
 contrib/libvhost-user/libvhost-user.h  |   6 +
 qemu-vu.c  | 252 +++
 softmmu/vl.c   |   4 +
 tests/Makefile.include |   3 +-
 tests/qtest/Makefile.include   |   2 +
 tests/qtest/libqos/vhost-user-blk.c| 126 
 tests/qtest/libqos/vhost-user-blk.h|  44 ++
 tests/qtest/vhost-user-blk-test.c  | 694 
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/Makefile.objs |   3 +
 util/vhost-user-server.c   | 360 ++
 util/vhost-user-server.h   |  57 ++
 20 files changed, 2318 insertions(+), 9 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 qemu-vu.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.25.1

Re: [PATCH v4 2/5] generic vhost user server

2020-02-27 Thread Coiby Xu

> > +static coroutine_fn void vu_client_next_trip(VuClient *client);
> > +
> > +static coroutine_fn void vu_client_trip(void *opaque)
> > +{
> > +VuClient *client = opaque;
> > +
> > +vu_dispatch(>parent);
> > +client->co_trip = NULL;
> > +if (!client->closed) {
> > +vu_client_next_trip(client);
> > +}
> > +}

> > The last part is very untypical coroutine code: It says that we want to
spawn a new coroutine with vu_client_trip() as its entry point, and then
terminates the current one.

> > Why don't we just put the whole thing in a while (!client->closed) loop
and stay in the same coroutine instead of terminating the old one and
starting a new one all the time?

> > +static coroutine_fn void vu_client_next_trip(VuClient *client)
> > +{
> > +if (!client->co_trip) {
> > +client->co_trip = qemu_coroutine_create(vu_client_trip, client);
> > +aio_co_schedule(client->ioc->ctx, client->co_trip);
> > +}
> > +}
> > +
> > +static void vu_client_start(VuClient *client)
> > +{
> > +client->co_trip = qemu_coroutine_create(vu_client_trip, client);
> > +aio_co_enter(client->ioc->ctx, client->co_trip);
> > +}

> This is essentially a duplicate of vu_client_next_trip(). The only
place where it is called (vu_accept()) knows that client->co_trip is
already NULL, so it could just call vu_client_next_trip().

> Or in fact, if vu_client_trip() gets turned into a loop, it's
> vu_client_next_trip() that becomes unnecessary.

This part of code is an imitation of nbd_client_trip in nbd/server.c.
I think the reason to repeatedly create/start/terminate vu_client_trip
is to support BlockBackendAioNotifier. In v5, I will keep running the
spawned coroutine in a loop until being informed of the change of
AioContext of the block device backend, i.e. vu_client_trip will only
be restarted when the block device backend is attached to a different
AiOContext.

> > +if (rc != sizeof(eventfd_t)) {
> > +if (errno == EAGAIN) {
> > +qio_channel_yield(data->ioc, G_IO_IN);
> > +} else if (errno != EINTR) {
> > +data->co = NULL;
> > +return;
> > +}
> > +} else {
> > +vq->handler(dev, index);
> > +}
> > +data->co = NULL;
> > +vu_kick_cb_next(client, data);

> This can be a loop, too, instead of terminating the coroutine and
starting a new one for the same function.

In v5, I plan to use aio_set_fd_handler to set a read hander which is
a wrapper for vu_kick_cb to deal with kick events since eventfd
doesn't have the short read issue like socket. Thus vu_kick_cb in
libvhost-user can be re-used. My only concern is if this could lead to
worse performance in comparison to keep reading from eventfd until
getting EAGAIN errno.

On Tue, Feb 25, 2020 at 11:44 PM Kevin Wolf  wrote:
>
> Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:
> > Sharing QEMU devices via vhost-user protocol
> >
> > Signed-off-by: Coiby Xu 
> > ---
> >  util/Makefile.objs   |   3 +
> >  util/vhost-user-server.c | 427 +++
> >  util/vhost-user-server.h |  56 +
> >  3 files changed, 486 insertions(+)
> >  create mode 100644 util/vhost-user-server.c
> >  create mode 100644 util/vhost-user-server.h
> >
> > diff --git a/util/vhost-user-server.h b/util/vhost-user-server.h
> > new file mode 100644
> > index 00..ff6d3145cd
> > --- /dev/null
> > +++ b/util/vhost-user-server.h
> > @@ -0,0 +1,56 @@
> > +#include "io/channel-socket.h"
> > +#include "io/channel-file.h"
> > +#include "io/net-listener.h"
> > +#include "contrib/libvhost-user/libvhost-user.h"
> > +#include "standard-headers/linux/virtio_blk.h"
> > +#include "qemu/error-report.h"
> > +
> > +typedef struct VuClient VuClient;
>
> I find the terminology a bit confusing here: VuClient is really the
> connection to a single client, but it's part of the server. The name
> gives the impression as if this were client-side code. (This is
> something that already tends to confuse me in the NBD code.)
>
> I'm not sure what a better name could be, though. Maybe
> VuServerConnevtion or VuExportClient or VuExportConnection?
>
> > +typedef struct VuServer {
> > +QIONetListener *listener;
> > +AioContext *ctx;
> > +QTAILQ_HEAD(, VuClient) clients;
> > +void (*device_panic_notifier)(struct VuClient *client) ;
> > +int max_queues;
> > +const VuDevIface *vu_iface;

Re: [PATCH v4 0/5] vhost-user block device backend implementation

2020-02-27 Thread Coiby Xu

> > we still need customized vu_message_read because libvhost-user assumes
> > we will always get a full-size VhostUserMsg and hasn't taken care of
> > this short read case. I will improve libvhost-user's vu_message_read
> > by making it keep reading from socket util getting enough bytes. I
> > assume short read is a rare case thus introduced performance penalty
> > would be negligible.

> In any case, please make sure that we use the QIOChannel functions
> called from a coroutine in QEMU so that it will never block, but the
> coroutine can just yield while it's waiting for more bytes.

But if I am not wrong, libvhost-user is supposed to be indepdent from
the main QEMU code. So it can't use the QIOChannel functions if we
simply modify exiting vu_message_read to address the short read issue.
In v3 & v4, I extended libvhost-user to allow vu_message_read to be
replaced by one which will depend on the main QEMU code. I'm not sure
which way is better.

On Thu, Feb 27, 2020 at 6:02 PM Kevin Wolf  wrote:
>
> Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:
> > Thank you for reminding me of this socket short read issue! It seems
> > we still need customized vu_message_read because libvhost-user assumes
> > we will always get a full-size VhostUserMsg and hasn't taken care of
> > this short read case. I will improve libvhost-user's vu_message_read
> > by making it keep reading from socket util getting enough bytes. I
> > assume short read is a rare case thus introduced performance penalty
> > would be negligible.
>
> In any case, please make sure that we use the QIOChannel functions
> called from a coroutine in QEMU so that it will never block, but the
> coroutine can just yield while it's waiting for more bytes.
>
> Kevin
>
> > On Thu, Feb 27, 2020 at 3:41 PM Stefan Hajnoczi  wrote:
> > >
> > > On Wed, Feb 26, 2020 at 11:18:41PM +0800, Coiby Xu wrote:
> > > > Hi Stefan,
> > > >
> > > > Thank you for reviewing my code!
> > > >
> > > > I tried to reach you on IRC. But somehow either you missed my message
> > > > or I missed your reply. So I will reply by email instead.
> > > >
> > > > If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event,
> > > > i.e. use vu_dispatch as the read handler, then we can re-use
> > > > vu_message_read. And "removing the blocking recv from libvhost-user"
> > > > isn't necessary because "the operation of poll() and ppoll() is not
> > > > affected by the O_NONBLOCK flag" despite that we use
> > > > qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler
> > > > to make recv non-blocking.
> > >
> > > I'm not sure I understand.  poll() just says whether the file descriptor
> > > is readable.  It does not say whether enough bytes are readable :).  So
> > > our callback will be invoked if there is 1 byte ready, but when we try
> > > to read 20 bytes either it will block (without O_NONBLOCK) or return
> > > only 1 byte (with O_NONBLOCK).  Neither case is okay, so I expect that
> > > code changes will be necessary.
> > >
> > > But please go ahead and send the next revision and I'll take a look.
> > >
> > > Stefan
> >
> >
> >
> > --
> > Best regards,
> > Coiby
> >
>


-- 
Best regards,
Coiby

Re: [PATCH v4 0/5] vhost-user block device backend implementation

2020-02-27 Thread Coiby Xu

Thank you for reminding me of this socket short read issue! It seems
we still need customized vu_message_read because libvhost-user assumes
we will always get a full-size VhostUserMsg and hasn't taken care of
this short read case. I will improve libvhost-user's vu_message_read
by making it keep reading from socket util getting enough bytes. I
assume short read is a rare case thus introduced performance penalty
would be negligible.


On Thu, Feb 27, 2020 at 3:41 PM Stefan Hajnoczi  wrote:
>
> On Wed, Feb 26, 2020 at 11:18:41PM +0800, Coiby Xu wrote:
> > Hi Stefan,
> >
> > Thank you for reviewing my code!
> >
> > I tried to reach you on IRC. But somehow either you missed my message
> > or I missed your reply. So I will reply by email instead.
> >
> > If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event,
> > i.e. use vu_dispatch as the read handler, then we can re-use
> > vu_message_read. And "removing the blocking recv from libvhost-user"
> > isn't necessary because "the operation of poll() and ppoll() is not
> > affected by the O_NONBLOCK flag" despite that we use
> > qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler
> > to make recv non-blocking.
>
> I'm not sure I understand.  poll() just says whether the file descriptor
> is readable.  It does not say whether enough bytes are readable :).  So
> our callback will be invoked if there is 1 byte ready, but when we try
> to read 20 bytes either it will block (without O_NONBLOCK) or return
> only 1 byte (with O_NONBLOCK).  Neither case is okay, so I expect that
> code changes will be necessary.
>
> But please go ahead and send the next revision and I'll take a look.
>
> Stefan



--
Best regards,
Coiby

Re: [PATCH v4 0/5] vhost-user block device backend implementation

2020-02-26 Thread Coiby Xu

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message
or I missed your reply. So I will reply by email instead.

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event,
i.e. use vu_dispatch as the read handler, then we can re-use
vu_message_read. And "removing the blocking recv from libvhost-user"
isn't necessary because "the operation of poll() and ppoll() is not
affected by the O_NONBLOCK flag" despite that we use
qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler
to make recv non-blocking.

Previously I needed to run customized vu_kick_cb as coroutines to call
blk_co_readv/blk_co_writev directly. After taking Kevin's feedback on
v4 into consideration, now I use aio_set_fd_handler to set a read
handler for kick_fd and  this read handler will then call vu_kick_cb.

On Thu, Feb 20, 2020 at 1:58 AM Stefan Hajnoczi  wrote:
>
> On Tue, Feb 18, 2020 at 01:07:06PM +0800, Coiby Xu wrote:
> > v4:
> >  * add object properties in class_init
> >  * relocate vhost-user-blk-test
> >  * other changes including using SocketAddress, coding style, etc.
>
> Thanks!  I think the vhost-user server code can be simplified if
> libvhost-user uses the event loop for asynchronous socket I/O.  Then
> it's no longer necessary to duplicate vu_message_read() and
> vu_kick_cb().  I've replied to Patch 1 and we can discuss on IRC if you
> want to chat about it.
>
> I've also CCed Marc-André to see what he thinks about removing the
> blocking recv from libvhost-user and instead using the event loop (just
> like for eventfds).
>
> Stefan

-- 
Best regards,
Coiby

Re: [PATCH v2 1/5] vhost-user block device backend

2020-02-20 Thread Coiby Xu

> > +vmsg->fd_num = 0;
> > +for (cmsg = CMSG_FIRSTHDR();
> > + cmsg != NULL;
> > + cmsg = CMSG_NXTHDR(, cmsg))
> > +{
> > +if (cmsg->cmsg_level == SOL_SOCKET && cmsg->cmsg_type ==
SCM_RIGHTS) {
> > +fd_size = cmsg->cmsg_len - CMSG_LEN(0);
> > +vmsg->fd_num = fd_size / sizeof(int);
> > +memcpy(vmsg->fds, CMSG_DATA(cmsg), fd_size);
> > +break;
> > +}
> > +}

> I think the fd passing part becomes easier when you use the proper
> qio_channel_readv_full() function. Its implementation is also a bit more
> careful than yours. For example, you forgot checking fd_size against
> VHOST_MEMORY_MAX_NREGIONS, allowing a buffer overflow in the memcpy(),
> and you don't adjust fd flags for the new file descriptors.

Oh, I used qio_channel_readv_full in v3 But I still forgot checking
fd_size against  VHOST_MEMORY_MAX_NREGIONS. I'll fix this buffer overflow
issue in v5.

On Thu, Jan 16, 2020 at 9:56 PM Kevin Wolf  wrote:

> Hi,
>
> I'm only doing a quick first review pointing out the more obvious
> things while I familiarise myself with your code. I intend to review it
> in more detail later (either in a second pass for this series, or when
> you post v3).
>
> Am 14.01.2020 um 15:06 hat Coiby Xu geschrieben:
> > By making use of libvhost, multiple block device drives can be exported
> and each drive can serve multiple clients simultaneously. Since
> vhost-user-server needs a block drive to be created first, delay the
> creation of this object.
> >
> > Signed-off-by: Coiby Xu 
>
> Please wrap the commit message at 72 characters.
>
> >  blockdev-vu.c  | 1008 
> >  include/block/vhost-user.h |   46 ++
> >  vl.c   |4 +
> >  3 files changed, 1058 insertions(+)
> >  create mode 100644 blockdev-vu.c
> >  create mode 100644 include/block/vhost-user.h
>
> This adds a single, relatively big source file. I see at least two
> parts: The generic vhost-user infrastructure with connection handling
> etc. and the implementation of the specific vhost-user-blk device.
> Separating these into two files is probably a good idea.
>
> I would also suggest to put the files in a new subdirectory
> block/export/ and call them vhost-user.c/vhost-user-blk.c. The new
> header file can be in the same directory as it shouldn't be used by
> anyone else.
>
> > diff --git a/blockdev-vu.c b/blockdev-vu.c
> > new file mode 100644
> > index 00..45f0bb43a7
> > --- /dev/null
> > +++ b/blockdev-vu.c
> > @@ -0,0 +1,1008 @@
>
> The LICENSE file clarifies that files without a license header are
> GPLv2+, so it's not strictly a problem, but I think it is good style to
> include a license header that explicitly tells so.
>
> > +#include "qemu/osdep.h"
> > +#include "block/vhost-user.h"
> > +#include "qapi/error.h"
> > +#include "qapi/qapi-types-sockets.h"
> > +#include "qapi/qapi-commands-block.h"
> > +
> > +#include "sysemu/block-backend.h"
> > +#include "qemu/main-loop.h"
> > +
> > +#include "qemu/units.h"
> > +
> > +#include "block/block.h"
> > +
> > +#include "qom/object_interfaces.h"
> > +
> > +#include 
> > +
> > +#include "hw/qdev-properties.h"
>
> Does the order of includes and the empty lines between them signify
> anything? If not, I suggest just sorting them alphabetically (and maybe
> using empty lines between different subdirectories if you like this
> better than a single large block).
>
> According to CODING_STYLE.rst, system headers like  come
> before all QEMU headers (except qemu/osdep.h, which always must come
> first).
>
> > +enum {
> > +VHOST_USER_BLK_MAX_QUEUES = 8,
> > +};
> > +
> > +struct virtio_blk_inhdr {
> > +unsigned char status;
> > +};
> > +
> > +
> > +static QTAILQ_HEAD(, VubDev) vub_devs =
> QTAILQ_HEAD_INITIALIZER(vub_devs);
> > +
> > +
> > +typedef struct VubReq {
> > +VuVirtqElement *elem;
>
> Maybe worth a comment that this was allocated with plain malloc(), so
> you must use free() rather than g_free() (which would be the default in
> QEMU)?
>
> > +int64_t sector_num;
> > +size_t size;
> > +struct virtio_blk_inhdr *in;
> > +struct virtio_blk_outhdr out;
> > +VuClient *client;
> > +struct VuVirtq *vq;
> > +} VubReq;
>
> I'm no

Re: [PATCH v2 1/5] vhost-user block device backend

2020-02-19 Thread Coiby Xu

Hi Kevin,

Thank you for reviewing my work in a rather detailed way.

> >  blockdev-vu.c  | 1008 
> >  include/block/vhost-user.h |   46 ++
> >  vl.c   |4 +
> >  3 files changed, 1058 insertions(+)
> >  create mode 100644 blockdev-vu.c
> >  create mode 100644 include/block/vhost-user.h

> This adds a single, relatively big source file. I see at least two
parts: The generic vhost-user infrastructure with connection handling
etc. and the implementation of the specific vhost-user-blk device.
Separating these into two files is probably a good idea.

> I would also suggest to put the files in a new subdirectory
block/export/ and call them vhost-user.c/vhost-user-blk.c. The new
header file can be in the same directory as it shouldn't be used by
anyone else.

I've split blockdev-vu.c in two separate files but in a different subdirectory
 - backends/vhost-user-blk-server.c
 - util/vhost-user-server.c

> > +static QTAILQ_HEAD(, VubDev) vub_devs = QTAILQ_HEAD_INITIALIZER(vub_devs);
> > +
> > +
> > +typedef struct VubReq {
> > +VuVirtqElement *elem;

> Maybe worth a comment that this was allocated with plain malloc(), so
> you must use free() rather than g_free() (which would be the default in
> QEMU)?

Although VuVirtqElement is created using malloc, VubReq is created using g_new0.


I missed several suggestions in v3 but in v4 all suggestions have been
applied. Thank you!
On Thu, Jan 16, 2020 at 9:56 PM Kevin Wolf  wrote:
>
> Hi,
>
> I'm only doing a quick first review pointing out the more obvious
> things while I familiarise myself with your code. I intend to review it
> in more detail later (either in a second pass for this series, or when
> you post v3).
>
> Am 14.01.2020 um 15:06 hat Coiby Xu geschrieben:
> > By making use of libvhost, multiple block device drives can be exported and 
> > each drive can serve multiple clients simultaneously. Since 
> > vhost-user-server needs a block drive to be created first, delay the 
> > creation of this object.
> >
> > Signed-off-by: Coiby Xu 
>
> Please wrap the commit message at 72 characters.
>
> >  blockdev-vu.c  | 1008 
> >  include/block/vhost-user.h |   46 ++
> >  vl.c   |4 +
> >  3 files changed, 1058 insertions(+)
> >  create mode 100644 blockdev-vu.c
> >  create mode 100644 include/block/vhost-user.h
>
> This adds a single, relatively big source file. I see at least two
> parts: The generic vhost-user infrastructure with connection handling
> etc. and the implementation of the specific vhost-user-blk device.
> Separating these into two files is probably a good idea.
>
> I would also suggest to put the files in a new subdirectory
> block/export/ and call them vhost-user.c/vhost-user-blk.c. The new
> header file can be in the same directory as it shouldn't be used by
> anyone else.
>
> > diff --git a/blockdev-vu.c b/blockdev-vu.c
> > new file mode 100644
> > index 00..45f0bb43a7
> > --- /dev/null
> > +++ b/blockdev-vu.c
> > @@ -0,0 +1,1008 @@
>
> The LICENSE file clarifies that files without a license header are
> GPLv2+, so it's not strictly a problem, but I think it is good style to
> include a license header that explicitly tells so.
>
> > +#include "qemu/osdep.h"
> > +#include "block/vhost-user.h"
> > +#include "qapi/error.h"
> > +#include "qapi/qapi-types-sockets.h"
> > +#include "qapi/qapi-commands-block.h"
> > +
> > +#include "sysemu/block-backend.h"
> > +#include "qemu/main-loop.h"
> > +
> > +#include "qemu/units.h"
> > +
> > +#include "block/block.h"
> > +
> > +#include "qom/object_interfaces.h"
> > +
> > +#include 
> > +
> > +#include "hw/qdev-properties.h"
>
> Does the order of includes and the empty lines between them signify
> anything? If not, I suggest just sorting them alphabetically (and maybe
> using empty lines between different subdirectories if you like this
> better than a single large block).
>
> According to CODING_STYLE.rst, system headers like  come
> before all QEMU headers (except qemu/osdep.h, which always must come
> first).
>
> > +enum {
> > +VHOST_USER_BLK_MAX_QUEUES = 8,
> > +};
> > +
> > +struct virtio_blk_inhdr {
> > +unsigned char status;
> > +};
> > +
> > +
> > +static QTAILQ_HEAD(, VubDev) vub_devs = QTAILQ_HEAD_INITIALIZER(vub_devs);
> > +
> > +
> > +typedef struct

[PATCH v4 5/5] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize.

Signed-off-by: Coiby Xu 
---
 tests/Makefile.include  |   3 +-
 tests/qtest/Makefile.include|   2 +
 tests/qtest/libqos/vhost-user-blk.c | 126 +
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/vhost-user-blk-test.c   | 694 
 5 files changed, 868 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 2f1cafed72..4b8637b5d4 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -629,7 +629,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all 
$(check-qtest-y)
$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) 
$(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
- QTEST_QEMU_IMG=qemu-img$(EXESUF))
+ QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+ QTEST_QEMU_VU_BINARY=./qemu-vu$(EXESUF))
 
 check-unit: $(check-unit-y)
$(call do_test_human, $^)
diff --git a/tests/qtest/Makefile.include b/tests/qtest/Makefile.include
index eb0f23b108..f587fe9f4d 100644
--- a/tests/qtest/Makefile.include
+++ b/tests/qtest/Makefile.include
@@ -182,6 +182,7 @@ qos-test-obj-y += tests/qtest/libqos/virtio.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 qos-test-obj-y += tests/qtest/libqos/virtio-balloon.o
 qos-test-obj-y += tests/qtest/libqos/virtio-blk.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 qos-test-obj-y += tests/qtest/libqos/virtio-mmio.o
 qos-test-obj-y += tests/qtest/libqos/virtio-net.o
 qos-test-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -224,6 +225,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += 
tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
diff --git a/tests/qtest/libqos/vhost-user-blk.c 
b/tests/qtest/libqos/vhost-user-blk.c
new file mode 100644
index 00..ec46b7ddb4
--- /dev/null
+++ b/tests/qtest/libqos/vhost-user-blk.c
@@ -0,0 +1,126 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVhostUserBlkDevice *vhost_user_blk = g_new0(QVhostUserBlkDevice, 1);
+QVhostUserBlk *interface = _user_blk->blk;
+
+interface->vdev = virtio_dev;
+
+vhost_user_blk->obj.get_driver = qvhost_user_blk_device_get_driver;
+
+return _user_blk->obj;
+}
+
+/* virtio-blk-pci */
+static void *qvhost_user_blk_pci_get_driver(void *object, const char 
*interface)
+{
+QVhostUserBlkPCI *v_blk = object;
+if (!g_strcmp0(interface, "p

[PATCH v4 3/5] vhost-user block device backend server

By making use of libvhost, multiple block device drives can be exported
and each drive can serve multiple clients simultaneously.
Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 Makefile.target  |   1 +
 backends/Makefile.objs   |   2 +
 backends/vhost-user-blk-server.c | 718 +++
 backends/vhost-user-blk-server.h |  21 +
 vl.c |   4 +
 5 files changed, 746 insertions(+)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h

diff --git a/Makefile.target b/Makefile.target
index 6e61f607b1..8c6c01eb3a 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -159,6 +159,7 @@ obj-y += monitor/
 obj-y += qapi/
 obj-y += memory.o
 obj-y += memory_mapping.o
+obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o
 obj-y += migration/ram.o
 LIBS := $(libs_softmmu) $(LIBS)
 
diff --git a/backends/Makefile.objs b/backends/Makefile.objs
index 28a847cd57..4e7be731e0 100644
--- a/backends/Makefile.objs
+++ b/backends/Makefile.objs
@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o
 common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o
 endif
 
+common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o
+
 common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o
 
 common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
diff --git a/backends/vhost-user-blk-server.c b/backends/vhost-user-blk-server.c
new file mode 100644
index 00..1bf7f7b544
--- /dev/null
+++ b/backends/vhost-user-blk-server.c
@@ -0,0 +1,718 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuClient *client;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >client->parent;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_client(VuClient *client)
+{
+return container_of(client->server->ptr_in_device, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -EINVAL;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+static int coroutine_fn vu_block_virtio_process_req(VuClient *client,
+VuVirtq *vq)
+{
+VuDev *vu_dev = >parent;
+VuVirtqElement *elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(client);
+BlockBackend *backend = vdev_blk->backend;
+elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
+sizeof(VuBlockReq));
+if (!elem) {
+return -1;
+}
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+

[PATCH v4 2/5] generic vhost user server

Sharing QEMU devices via vhost-user protocol

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   3 +
 util/vhost-user-server.c | 427 +++
 util/vhost-user-server.h |  56 +
 3 files changed, 486 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index 11262aafaf..5e450e501c 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -36,6 +36,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..70ff6d6701
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,427 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+
+static void close_client(VuClient *client)
+{
+vu_deinit(>parent);
+client->sioc = NULL;
+object_unref(OBJECT(client->ioc));
+client->closed = true;
+
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+VuClient *client = container_of(vu_dev, VuClient, parent);
+VuServer *server = client->server;
+if (!client->closed) {
+close_client(client);
+QTAILQ_REMOVE(>clients, client, next);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(client);
+}
+}
+
+
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+/*
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuClient *client = container_of(vu_dev, VuClient, parent);
+QIOChannel *ioc = client->ioc;
+
+Error *erp;
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, );
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report("Error while recvmsg: %s", strerror(errno));
+return false;
+}
+}
+read_bytes += rc;
+fds = g_renew(int, fds_t, nfds + nfds_t);
+memcpy(fds + nfds, fds_t, nfds_t);
+nfds += nfds_t;
+if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+break;
+}
+} while (true);
+
+vmsg->fd_num = nfds;
+memcpy(vmsg->fds, fds, nfds * sizeof(int));
+g_free(fds);
+/* qio_channel_readv_full will make socket fds blocking, unblock them */
+vmsg_unblock_fds(vmsg);
+if (vmsg->size > sizeof(vmsg->payload)) {
+error_report("Error: too big message request: %d, "
+ "size: vmsg->size: %u, "
+ "while sizeof(vmsg->payload) = %zu",
+ vmsg->request, vmsg->size, sizeof(vmsg->payload));
+goto fail;
+}
+
+struct iovec iov_payload = {
+.iov_base = (char *)>payload,
+.iov_len = vmsg->size,
+};
+if (vmsg->size) {
+rc = qio_channel_readv_all_eof(ioc, _payload, 1, );
+if (rc == -1) {
+error_report("Error while reading: %s", strerror(errno));
+goto fail;
+}
+}
+
+return true;
+
+fail:
+vmsg_close_fds(vmsg);
+
+return false;
+}
+
+
+static coroutine_fn void vu_clie

[PATCH v4 4/5] a standone-alone tool to directly share disk image file via vhost-user protocol

vhost-user-blk could have played as vhost-user backend but it only supports raw
file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES
operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real
block device).

In the future Kevin's qemu-storage-daemon will be used to replace this
tool.

Signed-off-by: Coiby Xu 
---
 Makefile  |   4 +
 configure |   3 +
 qemu-vu.c | 252 ++
 3 files changed, 259 insertions(+)
 create mode 100644 qemu-vu.c

diff --git a/Makefile b/Makefile
index b5a7377cb1..74fb109675 100644
--- a/Makefile
+++ b/Makefile
@@ -572,6 +572,10 @@ qemu-img.o: qemu-img-cmds.h
 
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+
+ifdef CONFIG_LINUX
+qemu-vu$(EXESUF): qemu-vu.o backends/vhost-user-blk-server.o $(authz-obj-y) 
$(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) 
libvhost-user.a
+endif
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
diff --git a/configure b/configure
index 6f5d850949..2b846cecf0 100755
--- a/configure
+++ b/configure
@@ -6239,6 +6239,9 @@ if test "$want_tools" = "yes" ; then
   if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
 tools="qemu-nbd\$(EXESUF) $tools"
   fi
+  if [ "$linux" = "yes" ] ; then
+tools="qemu-vu\$(EXESUF) $tools"
+  fi
   if [ "$ivshmem" = "yes" ]; then
 tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
   fi
diff --git a/qemu-vu.c b/qemu-vu.c
new file mode 100644
index 00..dd1032b205
--- /dev/null
+++ b/qemu-vu.c
@@ -0,0 +1,252 @@
+/*
+ *  Copyright (C) 2020  Coiby Xu 
+ *
+ *  standone-alone vhost-user-blk device server backend
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; under version 2 of the License.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include 
+#include 
+#include "backends/vhost-user-blk-server.h"
+#include "block/block_int.h"
+#include "io/net-listener.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qstring.h"
+#include "qemu/config-file.h"
+#include "qemu/cutils.h"
+#include "qemu/main-loop.h"
+#include "qemu/module.h"
+#include "qemu/option.h"
+#include "qemu-common.h"
+#include "qemu-version.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#define QEMU_VU_OPT_CACHE 256
+#define QEMU_VU_OPT_AIO   257
+#define QEMU_VU_OBJ_ID   "vu_disk"
+static QemuOptsList qemu_object_opts = {
+.name = "object",
+.implied_opt_name = "qom-type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+.desc = {
+{ }
+},
+};
+static char *srcpath;
+
+static void usage(const char *name)
+{
+(printf) (
+"Usage: %s [OPTIONS] FILE\n"
+"  or:  %s -L [OPTIONS]\n"
+"QEMU Vhost-user Server Utility\n"
+"\n"
+"  -h, --helpdisplay this help and exit\n"
+"  -V, --version output version information and exit\n"
+"\n"
+"Connection properties:\n"
+"  -k, --socket=PATH path to the unix socket\n"
+"\n"
+"General purpose options:\n"
+"  -e, -- exit-panic When the panic callback is called, the program\n"
+"will exit. Useful for make check-qtest.\n"
+"\n"
+"Block device options:\n"
+"  -f, --format=FORMAT   set image format (raw, qcow2, ...)\n"
+"  -r, --read-only   export read-only\n"
+"  -n, --nocache disable host cache\n"
+"  --cache=MODE  set cache mode (none, writeback, ...)\n"
+"  --aio=MODEset AIO mode (native or threads)\n"
+"\n"
+QEMU_HELP_BOTTOM "\n"
+, name, na

[PATCH v4 1/5] extend libvhost to support IOThread and coroutine

Previously libvhost dispatch events in its own GMainContext. Now vhost-user
client's kick event can be dispatched in block device drive's AioContext
thus IOThread is supported. And also allow vu_message_read and
vu_kick_cb to be replaced so QEMU can run them as coroutines.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user.c | 54 ---
 contrib/libvhost-user/libvhost-user.h | 38 ++-
 2 files changed, 85 insertions(+), 7 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index b89bf18501..f95664bb22 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -260,7 +258,7 @@ have_userfault(void)
 }
 
 static bool
-vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+vu_message_read_(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
 char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] = { };
 struct iovec iov = {
@@ -328,6 +326,17 @@ fail:
 return false;
 }
 
+static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+{
+vu_read_msg_cb read_msg;
+if (dev->co_iface) {
+read_msg = dev->co_iface->read_msg;
+} else {
+read_msg = vu_message_read_;
+}
+return read_msg(dev, conn_fd, vmsg);
+}
+
 static bool
 vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
 }
 
 if (dev->vq[index].kick_fd != -1 && dev->vq[index].handler) {
+if (dev->set_watch_packed_data) {
+dev->set_watch_packed_data(dev, dev->vq[index].kick_fd, 
VU_WATCH_IN,
+   dev->co_iface->kick_callback,
+   (void *)(long)index);
+} else {
 dev->set_watch(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
vu_kick_cb, (void *)(long)index);
-
+}
 DPRINT("Waiting for kicks on fd: %d for vq: %d\n",
dev->vq[index].kick_fd, index);
 }
@@ -1097,8 +,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq,
 vq->handler = handler;
 if (vq->kick_fd >= 0) {
 if (handler) {
+if (dev->set_watch_packed_data) {
+dev->set_watch_packed_data(dev, vq->kick_fd, VU_WATCH_IN,
+   dev->co_iface->kick_callback,
+   (void *)(long)qidx);
+} else {
 dev->set_watch(dev, vq->kick_fd, VU_WATCH_IN,
vu_kick_cb, (void *)(long)qidx);
+}
 } else {
 dev->remove_watch(dev, vq->kick_fd);
 }
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+/* remove watch for kick_fd
+ * When client process is running in gdb and
+ * quit command is run in gdb, QEMU will still dispatch the event
+ * which will cause segment fault in the callback function
+ */
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,
 
 assert(max_queues > 0);
 assert(socket >= 0);
-assert(set_watch);
+/* assert(set_watch); */
 assert(remove_watch);
 assert(iface);
 assert(panic);
@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,
 return true;
 }
 
+bool
+vu_init_packed_data(VuDev *dev,
+uint16_t max_queues,
+int socket,
+vu_panic_cb panic,
+vu_set_watch_cb_packed_data set_watch_packed_data,
+vu_remove_watch_cb remove_watch,
+const VuDevIface *iface,
+const CoIface *co_iface)
+{
+if (vu_init(dev, max_queues, socket, panic, NULL, remove_watch, iface)) {
+dev->set_watch_packed_data = set_watch_packed_data;
+dev->co_iface = co_iface;
+return true;
+}
+return false;
+}
+
 VuVirtq *
 vu_get_queue(VuDev *dev, int qidx)
 {
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index 5cb7708559..6aadeaa0f2 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *

[PATCH v4 0/5] vhost-user block device backend implementation

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feauture for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mouting it.

Coiby Xu (5):
  extend libvhost to support IOThread and coroutine
  generic vhost user server
  vhost-user block device backend server
  a standone-alone tool to directly share disk image file via vhost-user
protocol
  new qTest case to test the vhost-user-blk-server

 Makefile  |   4 +
 Makefile.target   |   1 +
 backends/Makefile.objs|   2 +
 backends/vhost-user-blk-server.c  | 718 ++
 backends/vhost-user-blk-server.h  |  21 +
 configure |   3 +
 contrib/libvhost-user/libvhost-user.c |  54 +-
 contrib/libvhost-user/libvhost-user.h |  38 +-
 qemu-vu.c | 252 +
 tests/Makefile.include|   3 +-
 tests/qtest/Makefile.include  |   2 +
 tests/qtest/libqos/vhost-user-blk.c   | 126 +
 tests/qtest/libqos/vhost-user-blk.h   |  44 ++
 tests/qtest/vhost-user-blk-test.c | 694 +
 util/Makefile.objs|   3 +
 util/vhost-user-server.c  | 427 +++
 util/vhost-user-server.h  |  56 ++
 vl.c  |   4 +
 18 files changed, 2444 insertions(+), 8 deletions(-)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h
 create mode 100644 qemu-vu.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.25.0

Re: [PATCH v2 1/5] vhost-user block device backend

2020-02-13 Thread Coiby Xu

Thank you for reviewing this patch! I'm already posted v3 based on
your feedback.

> > +#include "hw/qdev-properties.h"
> > +enum {
> > +VHOST_USER_BLK_MAX_QUEUES = 8,
> > +};

> The number of queues is hardcoded to 1 so this constant can be removed
for now.

> > +
> > +static QTAILQ_HEAD(, VubDev) vub_devs = QTAILQ_HEAD_INITIALIZER(vub_devs);

> I'm not sure if this list will be necessary.  See my comments about
> replacing the "name" property with Object's built-in "id" property.

There is no need to start vhost-user-blk-server with the same exported
drive since vhost-user-blk-server can serve multiple clients
simutanously. And we shoudn't started two vhost-user-blk-servers with
the same unix socket path. So we need this list to avoid dupliate
servers. And as pointed out by Kevin, "name" property actually means
node-name which is used in v3.

> > +#include "hw/qdev-properties.h"
> > +enum {
> > +VHOST_USER_BLK_MAX_QUEUES = 8,
> > +};

> The number of queues is hardcoded to 1 so this constant can be removed
for now.

I've set VHOST_USER_BLK_MAX_QUEUES = 1 in v3 to avoid magic number.

> > +config->seg_max = 128 - 2;

> This is okay for now but should be changed in the future.

> hw/block/virtio-blk.c was recently modified to calculate seg_max based
> on the virtqueue size (which is where the hardcoded 128 originally came
> from).  Applications that use large buffer sizes (128+ KB) benefit from
> larger virtqueue sizes and seg_max.

I've looked at the implementation of "-device
vhost-user-blk-pci,queue-size=512" regarding seg_max and found out
vhost-user-blk-server doesn' have the chance to caculate seg_max based
on the virtqueue size and report it back to QEMU as vhost-user client
in time.

QEMU as vhost-user client will create virtqueues with the size of
queue-size parameter. Later it will get the configureation including
seg_max from vhost-user-blk-server by sending VHOST_USER_SET_CONFIG
message and this seg_max vallue will be sent to guest OS.  Guest OS
will set the real size of virtqueue which will be sent to
vhost-user-blk-server through VHOST_USER_SET_VRING_NUM message. After
that QEMU as vhost-user client will never send VHOST_USER_SET_CONFIG
again.

There could be three ways to address this issue,
1. Add seg_max_adjust and queue-size object property for
vhost-user-blk device in a way similar to virtio-blk device. And QEMU
as vhost-user client will ignore seg_max parameter from
vhost-user-blk-server. It will caculate seg_max based on queue-size
and report it to guest OS.
2. Add seg_max_adjust and queue-size object property for
vhost-user-blk server and let  vhost-user-blk server calculate seg_max
based on queue-size
3. Let QEMU as vhost-user client tell vhost-user-blk-server its queue
size by sending VHOST_USER_SET_VRING_NUM message first. When
vhost-user-blk-server receive the VHOST_USER_SET_CONFIG message, it
will calculate seg_max and report it back to QEMU as vhost-user
client.

Which way do you is the best?

> > +void vub_accept(QIONetListener *listener, QIOChannelSocket *sioc,
> > +gpointer opaque)
> > +{
> > +VuClient *client;
> > +VubDev *vub_device = opaque;
> > +client = g_new0(VuClient, 1);

> Is it helpful to have a separate VuClient struct even though only a
> single vhost-user client can be connected at a time?  It may be simpler
> to keep connection state directly in VubDev.

Currently, I don't use chardev as an object property of
vhost-user-blk-server. So actually multiple clients can be connected
simutaneously.

All the other suggestions have been adopted in v3. Thank you for your advice!

On Thu, Jan 16, 2020 at 9:51 PM Stefan Hajnoczi  wrote:
>
> On Tue, Jan 14, 2020 at 10:06:16PM +0800, Coiby Xu wrote:
> > By making use of libvhost, multiple block device drives can be exported and 
> > each drive can serve multiple clients simultaneously. Since 
> > vhost-user-server needs a block drive to be created first, delay the 
> > creation of this object.
> >
> > Signed-off-by: Coiby Xu 
> > ---
> >  blockdev-vu.c  | 1008 
>
> This file contains both vhost-user-blk code and generic vhost-user
> protocol code.  Please split them into two files:
> 1. backends/vhost-user-blk-server.c
> 2. util/vhost-user-server.c
>
> (As I read and understand the code better I may have better suggestions
> about where to put these source files and how to name them.)
>
> >  include/block/vhost-user.h |   46 ++
> >  vl.c   |4 +
> >  3 files changed, 1058 insertions(+)
> >  create mode 100644 blockdev-vu.c
> >  create mode 100644 include/block/vhost-user.h
> >
> > diff --git

Re: [PATCH v3 4/5] a standone-alone tool to directly share disk image file via vhost-user protocol

I forgot to put backends/vhost-user-blk-server.o as dependency for
qemu-vu target in Makefile,

 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y)
$(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y)
$(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+
+ifdef CONFIG_LINUX
+qemu-vu$(EXESUF): qemu-vu.o backends/vhost-user-blk-server.o
$(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y)
$(COMMON_LDADDS) libvhost-user.a
+endif
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y)
$(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)

I also noticed in the latest version of QEMU, `make check-qtest`
somehow doesn't run qos-test. If you need to run vhost-user-blk
testsuite, please execute the following command after applying the
above fix and the 5th patch,

  MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
QTEST_QEMU_IMG=./qemu-img QTEST_QEMU_VU_BINARY=./qemu-vu
tests/qos-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl
--test-name="qos-test"


On Wed, Feb 12, 2020 at 5:52 PM Coiby Xu  wrote:
>
> vhost-user-blk could have played as vhost-user backend but it only supports 
> raw
> file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES
> operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real
> block device).
>
> In the future Kevin's qemu-storage-daemon will be used to replace this
> tool.
>
> Signed-off-by: Coiby Xu 
> ---
>  Makefile  |   4 +
>  configure |   3 +
>  qemu-vu.c | 252 ++
>  3 files changed, 259 insertions(+)
>  create mode 100644 qemu-vu.c
>
> diff --git a/Makefile b/Makefile
> index f0e1a2fc1d..0bfd2f1ddd 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -572,6 +572,10 @@ qemu-img.o: qemu-img-cmds.h
>
>  qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
> $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
> $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
> +
> +ifdef CONFIG_LINUX
> +qemu-vu$(EXESUF): qemu-vu.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
> $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) libvhost-user.a
> +endif
>  qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
> $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>
>  qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
> diff --git a/configure b/configure
> index 115dc38085..e87c9a5587 100755
> --- a/configure
> +++ b/configure
> @@ -6217,6 +6217,9 @@ if test "$want_tools" = "yes" ; then
>if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
>  tools="qemu-nbd\$(EXESUF) $tools"
>fi
> +  if [ "$linux" = "yes" ] ; then
> +tools="qemu-vu\$(EXESUF) $tools"
> +  fi
>if [ "$ivshmem" = "yes" ]; then
>  tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
>fi
> diff --git a/qemu-vu.c b/qemu-vu.c
> new file mode 100644
> index 00..dd1032b205
> --- /dev/null
> +++ b/qemu-vu.c
> @@ -0,0 +1,252 @@
> +/*
> + *  Copyright (C) 2020  Coiby Xu 
> + *
> + *  standone-alone vhost-user-blk device server backend
> + *
> + *  This program is free software; you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation; under version 2 of the License.
> + *
> + *  This program is distributed in the hope that it will be useful,
> + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + *  GNU General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License
> + *  along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include "qemu/osdep.h"
> +#include 
> +#include 
> +#include "backends/vhost-user-blk-server.h"
> +#include "block/block_int.h"
> +#include "io/net-listener.h"
> +#include "qapi/error.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qapi/qmp/qstring.h"
> +#include "qemu/config-file.h"
> +#include "qemu/cutils.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/module.h"
> +#include "qemu/option.h"
> +#include "qemu-common.h"
> +#include "qemu-version.h"
> +#incl

Re: [PATCH v2 3/5] a standone-alone tool to directly share disk image file via vhost-user protocol

> > > Yes, I think at least for the moment it should work fine this way.
> > > Eventually, I'd like to integrate it with --export (and associated QMP
> > > commands, which are still to be created), too. Maybe at that point we
> > > want to make the QOM object not user creatable any more.
> >
> > Does it mean TYPE_USER_CREATABLE interface in QOM will become
> > deprecated in the future? I'm curious what are the reasons for making
> > QOM object no user creatable? Because we may still need to start
> > vhost-user block device backend through HMP or QMP instead of stating
> > it as a standalone-alone daemon.

> Not in general, but if we have something like a block-export-add QMP
> command, the QOM interface would be redundant. We could still leave it
> there and have both a low-level and a high-level interface, but whether
> we would want to is something we still have to decide.

I see. So QOM interface will still be used as a low-level interface.
In the draft version, vhost-user-blk-server is started using specific
command vhost_user_server_start/vhost_user_server_stop which proivide
interfaces easier for implementing block-export-add QMP command. But
in later versions, only object_add/object_del syntax is supported to
start/stop vhost-user-blk-server. I'll keep an eye on how the storage
daemon develops and adapt my code accordingly.


On Sun, Feb 2, 2020 at 5:33 PM Kevin Wolf  wrote:
>
> Am 31.01.2020 um 17:42 hat Coiby Xu geschrieben:
> > > Yes, I think at least for the moment it should work fine this way.
> > > Eventually, I'd like to integrate it with --export (and associated QMP
> > > commands, which are still to be created), too. Maybe at that point we
> > > want to make the QOM object not user creatable any more.
> >
> > Does it mean TYPE_USER_CREATABLE interface in QOM will become
> > deprecated in the future? I'm curious what are the reasons for making
> > QOM object no user creatable? Because we may still need to start
> > vhost-user block device backend through HMP or QMP instead of stating
> > it as a standalone-alone daemon.
>
> Not in general, but if we have something like a block-export-add QMP
> command, the QOM interface would be redundant. We could still leave it
> there and have both a low-level and a high-level interface, but whether
> we would want to is something we still have to decide.
>
> > > As for test cases, do you think it would be hard to just modify the
> > > tests to send an explicit 'quit' command to the daemon?
> >
> > Accroding to 
> > https://patchew.org/QEMU/20191017130204.16131-1-kw...@redhat.com/20191017130204.16131-10-kw...@redhat.com/,
> >
> > > +static bool exit_requested = false;
> > > +
> > > +void qemu_system_killed(int signal, pid_t pid)
> > > +{
> > > +exit_requested = true;
> > > +}
> >
> > if exit_requested = true, qemu-storage-daemon will exit the main loop
> > and then quit. So is calling qemu_system_killed by what you means "to
> > send an explicit 'quit' command to the daemon"?
>
> qemu_system_killed() is call in the signal handlers for, amongst others,
> SIGTERM and SIGINT. This is one way to stop the storage daemon (for
> manual use, sending SIGINT with Ctrl-C is probably the easiest way).
>
> What I actually meant is the 'quit' QMP command which will cause
> qmp_quit() to be run, which contains the same code. But if sending a
> signal is more convenient, that's just as good.
>
> Kevin
>
> > On Fri, Jan 17, 2020 at 6:12 PM Kevin Wolf  wrote:
> > >
> > > Am 17.01.2020 um 09:12 hat Coiby Xu geschrieben:
> > > > Excellent! I will add an option (or object property) for
> > > > vhost-user-blk server oject which will tell the daemon process to exit
> > > > when the client disconnects, thus "make check-qtest" will not get held
> > > > by this daemon process. After that since Kevin's qemu-storage-daemon
> > > > support "-object" option
> > > > (https://patchew.org/QEMU/20191017130204.16131-1-kw...@redhat.com/20191017130204.16131-3-kw...@redhat.com/)
> > > > and vhost-user-server is a user-creatable QOM object, it will work out
> > > > of the box.
> > >
> > > Yes, I think at least for the moment it should work fine this way.
> > > Eventually, I'd like to integrate it with --export (and associated QMP
> > > commands, which are still to be created), too. Maybe at that point we
> > > want to make the QOM object not user creatable any more.
> > >
> > > Would it make sense to prefix the object type name with "x-"

[PATCH v3 5/5] new qTest case to test the vhost-user-blk-server

This test case has the same tests as tests/virtio-blk-test.c except for
tests have block_resize.

Signed-off-by: Coiby Xu 
---
 tests/libqos/vhost-user-blk.c | 126 ++
 tests/libqos/vhost-user-blk.h |  44 +++
 tests/vhost-user-blk-test.c   | 694 ++
 3 files changed, 864 insertions(+)
 create mode 100644 tests/libqos/vhost-user-blk.c
 create mode 100644 tests/libqos/vhost-user-blk.h
 create mode 100644 tests/vhost-user-blk-test.c

diff --git a/tests/libqos/vhost-user-blk.c b/tests/libqos/vhost-user-blk.c
new file mode 100644
index 00..ec46b7ddb4
--- /dev/null
+++ b/tests/libqos/vhost-user-blk.c
@@ -0,0 +1,126 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito 
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+
+#define PCI_SLOT0x04
+#define PCI_FN  0x00
+
+/* virtio-blk-device */
+static void *qvhost_user_blk_get_driver(QVhostUserBlk *v_blk,
+const char *interface)
+{
+if (!g_strcmp0(interface, "vhost-user-blk")) {
+return v_blk;
+}
+if (!g_strcmp0(interface, "virtio")) {
+return v_blk->vdev;
+}
+
+fprintf(stderr, "%s not present in vhost-user-blk-device\n", interface);
+g_assert_not_reached();
+}
+
+static void *qvhost_user_blk_device_get_driver(void *object,
+   const char *interface)
+{
+QVhostUserBlkDevice *v_blk = object;
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_device_create(void *virtio_dev,
+  QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVhostUserBlkDevice *vhost_user_blk = g_new0(QVhostUserBlkDevice, 1);
+QVhostUserBlk *interface = _user_blk->blk;
+
+interface->vdev = virtio_dev;
+
+vhost_user_blk->obj.get_driver = qvhost_user_blk_device_get_driver;
+
+return _user_blk->obj;
+}
+
+/* virtio-blk-pci */
+static void *qvhost_user_blk_pci_get_driver(void *object, const char 
*interface)
+{
+QVhostUserBlkPCI *v_blk = object;
+if (!g_strcmp0(interface, "pci-device")) {
+return v_blk->pci_vdev.pdev;
+}
+return qvhost_user_blk_get_driver(_blk->blk, interface);
+}
+
+static void *vhost_user_blk_pci_create(void *pci_bus, QGuestAllocator *t_alloc,
+  void *addr)
+{
+QVhostUserBlkPCI *vhost_user_blk = g_new0(QVhostUserBlkPCI, 1);
+QVhostUserBlk *interface = _user_blk->blk;
+QOSGraphObject *obj = _user_blk->pci_vdev.obj;
+
+virtio_pci_init(_user_blk->pci_vdev, pci_bus, addr);
+interface->vdev = _user_blk->pci_vdev.vdev;
+
+g_assert_cmphex(interface->vdev->device_type, ==, VIRTIO_ID_BLOCK);
+
+obj->get_driver = qvhost_user_blk_pci_get_driver;
+
+return obj;
+}
+
+static void vhost_user_blk_register_nodes(void)
+{
+/*
+ * FIXME: every test using these two nodes needs to setup a
+ * -drive,id=drive0 otherwise QEMU is not going to start.
+ * Therefore, we do not include "produces" edge for virtio
+ * and pci-device yet.
+ */
+
+char *arg = g_strdup_printf("id=drv0,chardev=char1,addr=%x.%x",
+PCI_SLOT, PCI_FN);
+
+QPCIAddress addr = {
+.devfn = QPCI_DEVFN(PCI_SLOT, PCI_FN),
+};
+
+QOSGraphEdgeOptions opts = { };
+
+/* virtio-blk-device */
+/** opts.extra_device_opts = "drive=drive0"; */
+qos_node_create_driver("vhost-user-blk-device", 
vhost_user_blk_device_create);
+qos_node_consumes("vhost-user-blk-device", "virtio-bus", );
+qos_node_produces("vhost-user-blk-device", "vhost-user-blk");
+
+/* virtio-blk-pci */
+opts.extra_device_opts = arg;
+add_qpci_address(, );
+qos_node_create_driver("vhost-user-blk-pci", vhost_user_blk_pci_create);
+qos_node_consumes("vhost-user-blk-pci", "pci-bus", );
+qos_node_produces("vhost-use

[PATCH v3 4/5] a standone-alone tool to directly share disk image file via vhost-user protocol

vhost-user-blk could have played as vhost-user backend but it only supports raw
file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES
operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real
block device).

In the future Kevin's qemu-storage-daemon will be used to replace this
tool.

Signed-off-by: Coiby Xu 
---
 Makefile  |   4 +
 configure |   3 +
 qemu-vu.c | 252 ++
 3 files changed, 259 insertions(+)
 create mode 100644 qemu-vu.c

diff --git a/Makefile b/Makefile
index f0e1a2fc1d..0bfd2f1ddd 100644
--- a/Makefile
+++ b/Makefile
@@ -572,6 +572,10 @@ qemu-img.o: qemu-img-cmds.h

 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+
+ifdef CONFIG_LINUX
+qemu-vu$(EXESUF): qemu-vu.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) libvhost-user.a
+endif
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) 
$(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)

 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
diff --git a/configure b/configure
index 115dc38085..e87c9a5587 100755
--- a/configure
+++ b/configure
@@ -6217,6 +6217,9 @@ if test "$want_tools" = "yes" ; then
   if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
 tools="qemu-nbd\$(EXESUF) $tools"
   fi
+  if [ "$linux" = "yes" ] ; then
+tools="qemu-vu\$(EXESUF) $tools"
+  fi
   if [ "$ivshmem" = "yes" ]; then
 tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
   fi
diff --git a/qemu-vu.c b/qemu-vu.c
new file mode 100644
index 00..dd1032b205
--- /dev/null
+++ b/qemu-vu.c
@@ -0,0 +1,252 @@
+/*
+ *  Copyright (C) 2020  Coiby Xu 
+ *
+ *  standone-alone vhost-user-blk device server backend
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; under version 2 of the License.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include 
+#include 
+#include "backends/vhost-user-blk-server.h"
+#include "block/block_int.h"
+#include "io/net-listener.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qstring.h"
+#include "qemu/config-file.h"
+#include "qemu/cutils.h"
+#include "qemu/main-loop.h"
+#include "qemu/module.h"
+#include "qemu/option.h"
+#include "qemu-common.h"
+#include "qemu-version.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#define QEMU_VU_OPT_CACHE 256
+#define QEMU_VU_OPT_AIO   257
+#define QEMU_VU_OBJ_ID   "vu_disk"
+static QemuOptsList qemu_object_opts = {
+.name = "object",
+.implied_opt_name = "qom-type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+.desc = {
+{ }
+},
+};
+static char *srcpath;
+
+static void usage(const char *name)
+{
+(printf) (
+"Usage: %s [OPTIONS] FILE\n"
+"  or:  %s -L [OPTIONS]\n"
+"QEMU Vhost-user Server Utility\n"
+"\n"
+"  -h, --helpdisplay this help and exit\n"
+"  -V, --version output version information and exit\n"
+"\n"
+"Connection properties:\n"
+"  -k, --socket=PATH path to the unix socket\n"
+"\n"
+"General purpose options:\n"
+"  -e, -- exit-panic When the panic callback is called, the program\n"
+"will exit. Useful for make check-qtest.\n"
+"\n"
+"Block device options:\n"
+"  -f, --format=FORMAT   set image format (raw, qcow2, ...)\n"
+"  -r, --read-only   export read-only\n"
+"  -n, --nocache disable host cache\n"
+"  --cache=MODE  set cache mode (none, writeback, ...)\n"
+"  --aio=MODEset AIO mode (native or threads)\n"
+"\n"
+QEMU_HELP_BOTTOM "\n"
+, name, name);
+}
+
+static void version(const ch

[PATCH v3 3/5] vhost-user block device backend server

By making use of libvhost, multiple block device drives can be exported
and each drive can serve multiple clients simultaneously.
Since vhost-user-server needs a block drive to be created first, delay
the creation of this object.

Signed-off-by: Coiby Xu 
---
 Makefile.target  |   1 +
 backends/Makefile.objs   |   2 +
 backends/vhost-user-blk-server.c | 716 +++
 backends/vhost-user-blk-server.h |  21 +
 vl.c |   4 +
 5 files changed, 744 insertions(+)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h

diff --git a/Makefile.target b/Makefile.target
index 6e61f607b1..8c6c01eb3a 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -159,6 +159,7 @@ obj-y += monitor/
 obj-y += qapi/
 obj-y += memory.o
 obj-y += memory_mapping.o
+obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o
 obj-y += migration/ram.o
 LIBS := $(libs_softmmu) $(LIBS)
 
diff --git a/backends/Makefile.objs b/backends/Makefile.objs
index 28a847cd57..4e7be731e0 100644
--- a/backends/Makefile.objs
+++ b/backends/Makefile.objs
@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o
 common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o
 endif
 
+common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o
+
 common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o
 
 common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
diff --git a/backends/vhost-user-blk-server.c b/backends/vhost-user-blk-server.c
new file mode 100644
index 00..7293ad87be
--- /dev/null
+++ b/backends/vhost-user-blk-server.c
@@ -0,0 +1,716 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+ QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+VuVirtqElement *elem;
+int64_t sector_num;
+size_t size;
+struct virtio_blk_inhdr *in;
+struct virtio_blk_outhdr out;
+VuClient *client;
+struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+VuDev *vu_dev = >client->parent;
+
+/* IO size with 1 extra status byte */
+vu_queue_push(vu_dev, req->vq, req->elem,
+  req->size + 1);
+vu_queue_notify(vu_dev, req->vq);
+
+if (req->elem) {
+free(req->elem);
+}
+
+g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_client(VuClient *client)
+{
+return container_of(client->server->ptr_in_device, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+  uint32_t iovcnt, uint32_t type)
+{
+struct virtio_blk_discard_write_zeroes desc;
+ssize_t size = iov_to_buf(iov, iovcnt, 0, , sizeof(desc));
+if (unlikely(size != sizeof(desc))) {
+error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+return -1;
+}
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+uint64_t range[2] = { le64toh(desc.sector) << 9,
+  le32toh(desc.num_sectors) << 9 };
+if (type == VIRTIO_BLK_T_DISCARD) {
+if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+return 0;
+}
+} else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+if (blk_co_pwrite_zeroes(vdev_blk->backend,
+ range[0], range[1], 0) == 0) {
+return 0;
+}
+}
+
+return -1;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+BlockBackend *backend = vdev_blk->backend;
+blk_co_flush(backend);
+}
+
+
+static int coroutine_fn vu_block_virtio_process_req(VuClient *client,
+VuVirtq *vq)
+{
+VuDev *vu_dev = >parent;
+VuVirtqElement *elem;
+uint32_t type;
+VuBlockReq *req;
+
+VuBlockDev *vdev_blk = get_vu_block_device_by_client(client);
+BlockBackend *backend = vdev_blk->backend;
+elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
+sizeof(VuBlockReq));
+if (!elem) {
+return -1;
+}
+
+struct iovec *in_iov = elem->in_sg;
+struct iovec *out_iov = elem->out_sg;
+

[PATCH v3 1/5] extend libvhost to support IOThread and coroutine

Previously libvhost dispatch events in its own GMainContext. Now vhost-user
client's kick event can be dispatched in block device drive's AioContext
thus IOThread is supported. And also allow vu_message_read and
vu_kick_cb to be replaced so QEMU can run them as coroutines.

Signed-off-by: Coiby Xu 
---
 contrib/libvhost-user/libvhost-user.c | 54 ---
 contrib/libvhost-user/libvhost-user.h | 38 ++-
 2 files changed, 85 insertions(+), 7 deletions(-)

diff --git a/contrib/libvhost-user/libvhost-user.c 
b/contrib/libvhost-user/libvhost-user.c
index b89bf18501..f95664bb22 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -260,7 +258,7 @@ have_userfault(void)
 }
 
 static bool
-vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+vu_message_read_(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
 char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] = { };
 struct iovec iov = {
@@ -328,6 +326,17 @@ fail:
 return false;
 }
 
+static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+{
+vu_read_msg_cb read_msg;
+if (dev->co_iface) {
+read_msg = dev->co_iface->read_msg;
+} else {
+read_msg = vu_message_read_;
+}
+return read_msg(dev, conn_fd, vmsg);
+}
+
 static bool
 vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
 }
 
 if (dev->vq[index].kick_fd != -1 && dev->vq[index].handler) {
+if (dev->set_watch_packed_data) {
+dev->set_watch_packed_data(dev, dev->vq[index].kick_fd, 
VU_WATCH_IN,
+   dev->co_iface->kick_callback,
+   (void *)(long)index);
+} else {
 dev->set_watch(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
vu_kick_cb, (void *)(long)index);
-
+}
 DPRINT("Waiting for kicks on fd: %d for vq: %d\n",
dev->vq[index].kick_fd, index);
 }
@@ -1097,8 +,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq,
 vq->handler = handler;
 if (vq->kick_fd >= 0) {
 if (handler) {
+if (dev->set_watch_packed_data) {
+dev->set_watch_packed_data(dev, vq->kick_fd, VU_WATCH_IN,
+   dev->co_iface->kick_callback,
+   (void *)(long)qidx);
+} else {
 dev->set_watch(dev, vq->kick_fd, VU_WATCH_IN,
vu_kick_cb, (void *)(long)qidx);
+}
 } else {
 dev->remove_watch(dev, vq->kick_fd);
 }
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
 }
 
 if (vq->kick_fd != -1) {
+/* remove watch for kick_fd
+ * When client process is running in gdb and
+ * quit command is run in gdb, QEMU will still dispatch the event
+ * which will cause segment fault in the callback function
+ */
+dev->remove_watch(dev, vq->kick_fd);
 close(vq->kick_fd);
 vq->kick_fd = -1;
 }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,
 
 assert(max_queues > 0);
 assert(socket >= 0);
-assert(set_watch);
+/* assert(set_watch); */
 assert(remove_watch);
 assert(iface);
 assert(panic);
@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,
 return true;
 }
 
+bool
+vu_init_packed_data(VuDev *dev,
+uint16_t max_queues,
+int socket,
+vu_panic_cb panic,
+vu_set_watch_cb_packed_data set_watch_packed_data,
+vu_remove_watch_cb remove_watch,
+const VuDevIface *iface,
+const CoIface *co_iface)
+{
+if (vu_init(dev, max_queues, socket, panic, NULL, remove_watch, iface)) {
+dev->set_watch_packed_data = set_watch_packed_data;
+dev->co_iface = co_iface;
+return true;
+}
+return false;
+}
+
 VuVirtq *
 vu_get_queue(VuDev *dev, int qidx)
 {
diff --git a/contrib/libvhost-user/libvhost-user.h 
b/contrib/libvhost-user/libvhost-user.h
index 5cb7708559..6aadeaa0f2 100644
--- a/contrib/libvhost-user/libvhost-user.h
+++ b/contrib/libvhost-user/libvhost-user.h
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
 VHOST_SET_CONFIG_TYPE_MASTER = 0,
 VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *

[PATCH v3 2/5] generic vhost user server

Sharing QEMU devices via vhost-user protocol

Signed-off-by: Coiby Xu 
---
 util/Makefile.objs   |   3 +
 util/vhost-user-server.c | 429 +++
 util/vhost-user-server.h |  56 +
 3 files changed, 489 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

diff --git a/util/Makefile.objs b/util/Makefile.objs
index 11262aafaf..5e450e501c 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -36,6 +36,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
diff --git a/util/vhost-user-server.c b/util/vhost-user-server.c
new file mode 100644
index 00..0766b414c3
--- /dev/null
+++ b/util/vhost-user-server.c
@@ -0,0 +1,429 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include 
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+close(vmsg->fds[i]);
+}
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+int i;
+for (i = 0; i < vmsg->fd_num; i++) {
+qemu_set_nonblock(vmsg->fds[i]);
+}
+}
+
+
+static void close_client(VuClient *client)
+{
+vu_deinit(>parent);
+client->sioc = NULL;
+object_unref(OBJECT(client->ioc));
+client->closed = true;
+
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+if (buf) {
+error_report("vu_panic: %s", buf);
+}
+
+VuClient *client = container_of(vu_dev, VuClient, parent);
+VuServer *server = client->server;
+if (!client->closed) {
+close_client(client);
+QTAILQ_REMOVE(>clients, client, next);
+}
+
+if (server->device_panic_notifier) {
+server->device_panic_notifier(client);
+}
+}
+
+
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+struct iovec iov = {
+.iov_base = (char *)vmsg,
+.iov_len = VHOST_USER_HDR_SIZE,
+};
+int rc, read_bytes = 0;
+/*
+ * VhostUserMsg is a packed structure, gcc will complain about passing
+ * pointer to a packed structure member if we pass _num
+ * and  directly when calling qio_channel_readv_full,
+ * thus two temporary variables nfds and fds are used here.
+ */
+size_t nfds = 0, nfds_t = 0;
+int *fds = NULL, *fds_t = NULL;
+VuClient *client = container_of(vu_dev, VuClient, parent);
+QIOChannel *ioc = client->ioc;
+
+Error *erp;
+assert(qemu_in_coroutine());
+do {
+/*
+ * qio_channel_readv_full may have short reads, keeping calling it
+ * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+ */
+rc = qio_channel_readv_full(ioc, , 1, _t, _t, );
+if (rc < 0) {
+if (rc == QIO_CHANNEL_ERR_BLOCK) {
+qio_channel_yield(ioc, G_IO_IN);
+continue;
+} else {
+error_report("Error while recvmsg: %s", strerror(errno));
+return false;
+}
+}
+read_bytes += rc;
+fds = g_renew(int, fds_t, nfds + nfds_t);
+memcpy(fds + nfds, fds_t, nfds_t);
+nfds += nfds_t;
+if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+break;
+}
+} while (true);
+
+vmsg->fd_num = nfds;
+memcpy(vmsg->fds, fds, nfds * sizeof(int));
+g_free(fds);
+/* qio_channel_readv_full will make socket fds blocking, unblock them */
+vmsg_unblock_fds(vmsg);
+if (vmsg->size > sizeof(vmsg->payload)) {
+error_report("Error: too big message request: %d, "
+ "size: vmsg->size: %u, "
+ "while sizeof(vmsg->payload) = %zu",
+ vmsg->request, vmsg->size, sizeof(vmsg->payload));
+goto fail;
+}
+
+struct iovec iov_payload = {
+.iov_base = (char *)>payload,
+.iov_len = vmsg->size,
+};
+if (vmsg->size) {
+rc = qio_channel_readv_all_eof(ioc, _payload, 1, );
+if (rc == -1) {
+error_report("Error while reading: %s", strerror(errno));
+goto fail;
+}
+}
+
+return true;
+
+fail:
+vmsg_close_fds(vmsg);
+
+return false;
+}
+
+
+static coroutine_fn void vu_clie

[PATCH v3 0/5] vhost-user block device backend implementation