[Qemu-devel] [virtio-net] migration
Hi, What is the workflow for virtio-net devices when migrating the VM it belongs to? >From what I understand so far the stop() primitives in qemu are called for >virtio_net and vhost_net which makes the virtqueue reset such that it seems >the queues are completely empty. How does the guest know that the jobs he sent on the virtqueue were discarded so that he can retry or release resources that were used for managing those jobs/packets? Catalin Vasile
Re: [Qemu-devel] virtio-net and vhost-net init, virtio-scsi and vhost-scsi init
Yes it is. Thanks From: Stefan Hajnoczi <stefa...@gmail.com> Sent: Thursday, May 5, 2016 9:00:33 PM To: Catalin Vasile Cc: qemu-devel@nongnu.org Subject: Re: [Qemu-devel] virtio-net and vhost-net init, virtio-scsi and vhost-scsi init On Thu, May 05, 2016 at 11:05:27AM +, Catalin Vasile wrote: > When the virtio-net and virtio-scsi drivers have done the probe() primitive > they set the DRIVER_OK flag. > > If the vhost kernel backend is used, the set_status() primitive in qemu will > be triggered with DRIVER_OK status and it will trigger vhost_XXX_start(). > > How does the net and scsi solutions ensure that the vhost_XXX_start() > primitive has finished before sending jobs to through the virtqueues? Because > if vhost_XXX_start() has not finished, jobs might get to the qemu dummy > virtqueue handles, instead of getting to the vhost server in the host kernel. The trick is to explicitly signal the ioeventfd. See QEMU's virtio_pci_set_host_notifier_internal(): r = event_notifier_init(notifier, 1); The '1' means the notifier is initialized in a readable state so whoever is monitoring the file descriptor will be able to read right away. Whether vhost receives the kick from the guest or not, it will definitely notice that the ioeventfd can be read. And maybe the virtqueue has no new available buffers. That's okay too, then this was just a spurious wakeup. Is this what you were asking about? Stefan
[Qemu-devel] virtio-net and vhost-net init, virtio-scsi and vhost-scsi init
When the virtio-net and virtio-scsi drivers have done the probe() primitive they set the DRIVER_OK flag. If the vhost kernel backend is used, the set_status() primitive in qemu will be triggered with DRIVER_OK status and it will trigger vhost_XXX_start(). How does the net and scsi solutions ensure that the vhost_XXX_start() primitive has finished before sending jobs to through the virtqueues? Because if vhost_XXX_start() has not finished, jobs might get to the qemu dummy virtqueue handles, instead of getting to the vhost server in the host kernel.
Re: [Qemu-devel] [virtio guest] vring_need_event() from virtqueue_kick_prepare()
Do you mean vhost_net - old kernel, qemu - latest, guest - latest? On Thu, Jul 16, 2015 at 7:33 PM, Stefan Hajnoczi stefa...@gmail.com wrote: On Thu, Jul 16, 2015 at 1:54 PM, Catalin Vasile catalinvasil...@gmail.com wrote: Both. The compiled kernel was common for both. Does vhost_net work with the old kernel + new QEMU combo? Stefan
Re: [Qemu-devel] [virtio guest] vring_need_event() from virtqueue_kick_prepare()
Both. The compiled kernel was common for both. On Thu, Jul 16, 2015 at 3:52 PM, Stefan Hajnoczi stefa...@gmail.com wrote: On Thu, Jul 09, 2015 at 01:45:50PM +0300, Catalin Vasile wrote: I have managed to deal with it. The thing is I was using one of the latest versions of qemu, but an old Linux Kernel version of 3.12. Old guest kernel or old host kernel? New QEMU should support old guest kernels. Maybe you have found a bug?
Re: [Qemu-devel] [virtio guest] vring_need_event() from virtqueue_kick_prepare()
I have managed to deal with it.
The thing is I was using one of the latest versions of qemu, but an
old Linux Kernel version of 3.12.
On Thu, Jul 9, 2015 at 1:43 PM, Stefan Hajnoczi stefa...@gmail.com wrote:
On Tue, Jul 07, 2015 at 02:43:06PM +0300, Catalin Vasile wrote:
My vhost module respects the format vhost-net uses:
/* code summary */
mutex_lock(vq-mutex);
vhost_disable_notify();
for (;;) {
head = vhost_get_vq_desc();
if (head == vq-num) {
if (unlikely(vhost_enable_notify())) {
vhost_disable_notify();
continue;
}
break;
}
vhost_add_used_and_signal();
}
mutex_unlock(vq-mutex);
/* /code */
I have made a lot of printk() calls and the first job gets processed
completely, and gets through all those calls:
1. it goes into a first loop and processes the first job (get
descriptor, work with the descriptor, add used and signal).
2. On the second loop it hits head == vq-num, and goes back to
listening to notifications (successfully, it does not get into the
fallback).
Now in the guest:
1. sends first job and the paramers used to call vring_need_event() are:
vring_avail_event=0, new=1, old=0 (which makes the function evaluate to 0
1)
2. the queue is kicked and vhost does its job.
3. the guest driver reaches the end of the first job, and lets the
following job take its course, only this time vring_need_event()
receives the following parameters:
vring_avail_event=0, new=2, old=1 (which makes the function evaluate to 1
1)
This means you need to look at the vhost code because vring_avail_event
shouldn't be 0.
Why wasn't vhost_update_avail_event() called?
Maybe the feature bit negotiation for your device is broken and vhost
thinks VIRTIO_RING_F_EVENT_IDX is not set.
Or does the used ring have the VRING_USED_F_NO_NOTIFY flag?
Re: [Qemu-devel] [virtio guest] vring_need_event() from virtqueue_kick_prepare()
My vhost module respects the format vhost-net uses:
/* code summary */
mutex_lock(vq-mutex);
vhost_disable_notify();
for (;;) {
head = vhost_get_vq_desc();
if (head == vq-num) {
if (unlikely(vhost_enable_notify())) {
vhost_disable_notify();
continue;
}
break;
}
vhost_add_used_and_signal();
}
mutex_unlock(vq-mutex);
/* /code */
I have made a lot of printk() calls and the first job gets processed
completely, and gets through all those calls:
1. it goes into a first loop and processes the first job (get
descriptor, work with the descriptor, add used and signal).
2. On the second loop it hits head == vq-num, and goes back to
listening to notifications (successfully, it does not get into the
fallback).
Now in the guest:
1. sends first job and the paramers used to call vring_need_event() are:
vring_avail_event=0, new=1, old=0 (which makes the function evaluate to 0 1)
2. the queue is kicked and vhost does its job.
3. the guest driver reaches the end of the first job, and lets the
following job take its course, only this time vring_need_event()
receives the following parameters:
vring_avail_event=0, new=2, old=1 (which makes the function evaluate to 1 1)
so a kick is not actually sent because vring_need_event() returns
false. From what I see as the definition for vring_need_event(), it
does not actually look at flags.
if (vq-event) { evaluates to true in both cases, so it always
verifies those indexes (it does not go on the branch which verifies
flags).
I am also pretty sure the jobs are serialized in the guest driver, and
do not cross each other's path. One of the reasons is that every
function that sends a job must hold a mutex that protects the
virtqueue.
The guest driver blocks awaiting an interrupt for the job being
finished, but vhost does not get woken up to process the job in the
first places, because a notification is not actually triggered because
of what I have explained above.
On Tue, Jul 7, 2015 at 1:17 PM, Stefan Hajnoczi stefa...@gmail.com wrote:
On Mon, Jul 06, 2015 at 06:13:29PM +0300, Catalin Vasile wrote:
What is the logic behind vring_need_event() when used with
virtqueue_kick_prepare()?
What does the keyword just refer to from the following context:
/* The following is used with USED_EVENT_IDX and AVAIL_EVENT_IDX */
/* Assuming a given event_idx value from the other size, if
* we have just incremented index from old to new_idx,
* should we trigger an event? */
?
just means since the last time the host/guest-visible index field was
changed. After avail or used rings have been processed, the index field
for that ring is published to the host/guest. At that point a check is
made whether the other side needs to be kicked.
I am sending 2 jobs, one after another, and the second one just does
not want to kick, although the first one finished completely and the
backend went back to interrupt mode, all because vring_need_event()
returns false.
Maybe the vhost driver called vhost_disable_notify() and hasn't
re-enabled notify yet?
This could happen if the guest adds buffers to the virtqueue while the
host is processing the virtqueue. Take a look at the vhost_net code for
how to correctly disable and re-enable notify without race conditions on
the host.
The idea behind disabling notify is to eliminate unnecessary
vmexits/notifications since the host is already processing the virtqueue
and will see new buffers. It's like a polling vs interrupt mode.
If the vhost driver on the host doesn't implement it correctly, then the
device could stop responding to the avail ring.
[Qemu-devel] [virtio guest] vring_need_event() from virtqueue_kick_prepare()
What is the logic behind vring_need_event() when used with virtqueue_kick_prepare()? What does the keyword just refer to from the following context: /* The following is used with USED_EVENT_IDX and AVAIL_EVENT_IDX */ /* Assuming a given event_idx value from the other size, if * we have just incremented index from old to new_idx, * should we trigger an event? */ ? I am sending 2 jobs, one after another, and the second one just does not want to kick, although the first one finished completely and the backend went back to interrupt mode, all because vring_need_event() returns false.
[Qemu-devel] [vhost] virtio (guest) interrupt kernel modules
Is there something I need to install (kernel module or something like that) so that the virtio guest receives signals from vhost? The communication from and to qemu works (I can send a message from guest to qemu, and I can send a message from qemu to guest), and I can send a message from guest to vhost, but I don't get any signals when I call vhost_add_used_and_signal()).
Re: [Qemu-devel] [vhost] virtio (guest) interrupt kernel modules
It seems that the problem is from the code itself. If I call vhost_dev_start() in set_status() (in qemu), when the status changes to VIRTIO_CONFIG_S_DRIVER_OK, only the communication path from guest to vhost works. In another scenario I send a message from guest to qemu, and the vq handler in qemu calls vhost_dev_start(). From here right on the communication works both ways between guest and vhost (I can also receive signals from vhost to guest). I used the first scenario because I took vhost-scsi as an example. The thing is I don't understand what I'm doing different. On Thu, Jun 11, 2015 at 5:26 PM, Catalin Vasile catalinvasil...@gmail.com wrote: Is there something I need to install (kernel module or something like that) so that the virtio guest receives signals from vhost? The communication from and to qemu works (I can send a message from guest to qemu, and I can send a message from qemu to guest), and I can send a message from guest to vhost, but I don't get any signals when I call vhost_add_used_and_signal()).
Re: [Qemu-devel] [virtio] when is VIRTIO_CONFIG_S_DRIVER_OK set?
The problem was that my device started having activity before the backend was finished to be set up. On Wed, Jun 10, 2015 at 12:26 PM, Catalin Vasile catalinvasil...@gmail.com wrote: When is VIRTIO_CONFIG_S_DRIVER_OK set? I've seen in vhost-scsi from qemu, that it sets it's own set_status() member function for a virtio object, which checks for VIRTIO_CONFIG_S_DRIVER_OK (from a status variable received as a parameter) so it knows if it has to call vhost_scsi_start(), vhost_scsi_stop(), or don't do anything if the status already corresponds to what was received. I've tried to implement the same logic in my virtio device, but it always triggers set_status() with a status of 0 (no flags set).
[Qemu-devel] [virtio] when is VIRTIO_CONFIG_S_DRIVER_OK set?
When is VIRTIO_CONFIG_S_DRIVER_OK set? I've seen in vhost-scsi from qemu, that it sets it's own set_status() member function for a virtio object, which checks for VIRTIO_CONFIG_S_DRIVER_OK (from a status variable received as a parameter) so it knows if it has to call vhost_scsi_start(), vhost_scsi_stop(), or don't do anything if the status already corresponds to what was received. I've tried to implement the same logic in my virtio device, but it always triggers set_status() with a status of 0 (no flags set).
[Qemu-devel] [virtio] virtqueue_add_sgs
If I send to the backend out buffers using multiple sgs, what should I expect when the user sends me back the result? What should I expect to get when I do a virtqueue_get_buf()? Do I need to make multiple calls to virtqueue_get_buf() in order to get all my out buffers? Or it returns with the first buffer and then I somehow have to guess the other buffers from the data received in the first buffer?
[Qemu-devel] [virtio-net] guest driver, vq sync
From what I see, there are no sync primitives used on virtqueues in the guest driver. How does it make sure start_xmit() (for example) is not called multiple times, so that the vq internal control data doesn't get destroyed?
[Qemu-devel] eventfd_signal()
Is eventfd blocking? By that I mean: If I signal the guest for something, will that thread remain blocked till the guest clears the interrupt?
[Qemu-devel] [virtio] use virtqueue async
Can I use a virtqueue asynchronously between guest and backend? By that I mean that any end of the communication channel (virtqueue) can push/pop how much data he wants anytime, without worrying that it will pop a thing that it pushed on the queue, and in fact the other end should have popped it? Or do I have to use 2 virtqueues for this to happen? What I'm trying to do is to process jobs in the backend continuously and then send a little more detailed result of the job. The job itself will run async on some piece of hardware while I continue popping jobs from the queue (the backend itself will just schedule them, will not busy wait till they get done). When a job is done, a callback will be called which will push the result one the virtqueue. What I am worrying about is what if I use the same queue for job request and job result? Is there a possibility for a job result to be processed by the same entity that put it there, instead of the other end?
[Qemu-devel] [virtio-vhost] virtqueue, async vq push from vhost
Can virtqueues be used asynchronous from the vhost end? From what I've seen there are some primitives with *add_used_and_signal* which update some indexes behind the scene. Could marking and signaling used buffers (in another order other than the way they were received) mess up cleaning and/or other functionalities?
[Qemu-devel] physical memory address in guest (from his point of view)
Does a physical memory address in a guest points to the same memory as qemu? For example: if I have a physical pointer 0x34 in the guest, does it point to the same location that qemu points with 0x34 address?
Re: [Qemu-devel] physical memory address in guest (from his point of view)
On Fri, Apr 24, 2015 at 12:05 PM, Paolo Bonzini pbonz...@redhat.com wrote: On 24/04/2015 10:46, Catalin Vasile wrote: Does a physical memory address in a guest points to the same memory as qemu? For example: if I have a physical pointer 0x34 in the guest, does it point to the same location that qemu points with 0x34 address? No. QEMU converts guest pointers (which are just uint64_t) to host pointers using address_space_map/address_space_unmap. Paolo That being said. Does a message that passes from guest to a backend need to pass through qemu everytime? I'm trying to understand how virtio virtqueues work. Namely I'm trying to pass as data itself a pointer in which the backend will write, namely use copy_to_user() if the backend is a kernel module.
Re: [Qemu-devel] vhost/virtio migration planning
Sorry for not respoding there. When I started researching and working on this qemu-devel was the only mailing list where I actually got support (answers) for my questions. On Tue, Apr 21, 2015 at 4:33 PM, Stefan Hajnoczi stefa...@gmail.com wrote: On Fri, Apr 17, 2015 at 02:03:59PM +0300, Catalin Vasile wrote: I am working on a virtio-crypto with vhost backend. You didn't respond in the [GSoC] project proposal thread where I explained why vhost isn't appropriate for this device: My suggestion is to work on the gnutls driver. Then, if you have time left, get cryptodev upstream (it can be part of your GSoC project plan). That approach is more beneficial in the long run. It will allow other applications to use the Crypto API too. vhost is good for exploiting kernel-only functionality (usually due to security/reliability boundaries). In this case the only reason for vhost is that the userspace API isn't ready yet. Use the opportunity to contribute to that effort instead of working around it. Stefan
[Qemu-devel] [virtio-vhost communication] memory mapping
If I send a pointer to a vhost server from a virtio driver, do I have to do anything else before I can call copy_from_user() or copy_to_user() on that pointer?
[Qemu-devel] register_savevm()
When are save() and load() handlers actually used? When are they trying to save and restore state?
Re: [Qemu-devel] register_savevm()
Does loadvm create new virtual devices (dev.init(), dev.realize(), etc.) for the vm to run onto and then load VMStateDescription? On Fri, Apr 17, 2015 at 4:40 PM, Peter Maydell peter.mayd...@linaro.org wrote: On 17 April 2015 at 14:03, Catalin Vasile catalinvasil...@gmail.com wrote: When are save() and load() handlers actually used? When are they trying to save and restore state? These are used for: * live migration * VM state save/load (eg via monitor 'savevm' and 'loadvm' commands or command line -loadvm option). Note that the register_savevm() function is legacy and should not be used in any new code. Devices should use VMStateDescription structs instead. -- PMM
[Qemu-devel] vhost/virtio migration planning
I am working on a virtio-crypto with vhost backend. This functionality should expose a virtual cryptographic device to the guest which would send crypto jobs to the host. One problem I am facing is where to store session data (such as keys, IVs, etc.). For performance improvements, I would be ideal for these to be stored inside the vhost module, so that the guest won't have to transfer session data every time it needs to get a job done. While this would be great for performance, this might represent a problem when migrating VMs, because I would need to query the vhost module to get my session data back in order to migrate successfully to a new host. The problem with this is that it doesn't sound like the virtio-crypto model will remain very backend agnostic. Can I get some opinions of this situation, please?
[Qemu-devel] [virtio] qemu vs vhost virtqueue consumer
I am using the virtio-net vhost-net model to make another virtio/vhost solution. I can't seem to find how does one make a virtqueue to only be consumed by vhost when adding notifiers to vhost. virtio_add_queue() seems to always ask for a valid handle for qemu, and I don't see any disable function primitives called in either vhost or virtio part in qemu. I then thought the qemu part checks a flag every time a message appears, and it stops it from popping data from the virtqueue, but I don't see that either. How does the virtio/vhost-net solution make sure qemu doesn't consume data from a virtqueue destined for vhost?
Re: [Qemu-devel] [virtio] qemu vs vhost virtqueue consumer
Awesome. Thanks On Tue, Apr 7, 2015 at 11:51 AM, Greg Kurz gk...@linux.vnet.ibm.com wrote: On Tue, 7 Apr 2015 11:10:07 +0300 Catalin Vasile catalinvasil...@gmail.com wrote: I am using the virtio-net vhost-net model to make another virtio/vhost solution. I can't seem to find how does one make a virtqueue to only be consumed by vhost when adding notifiers to vhost. virtio_add_queue() seems to always ask for a valid handle for qemu, and I don't see any disable function primitives called in either vhost or virtio part in qemu. I then thought the qemu part checks a flag every time a message appears, and it stops it from popping data from the virtqueue, but I don't see that either. How does the virtio/vhost-net solution make sure qemu doesn't consume data from a virtqueue destined for vhost? The idea behind vhost is we want to avoid QEMU to be involved when data is being copied to/from the guest: adding notifiers to vhost basically means that MMIO will be redirected to vhost instead of QEMU. In other words, QEMU cannot consume data because it is simply not notified at all in this case. Cheers. -- Greg
[Qemu-devel] virtio/vhost find queues, which virtqueue is which
How can I tell which virtqueue is which? I'm trying to make 2 virtqueues: 1) control vq between qemu and guest 2) data/job vq between vhost and guest I went through find_vqs() inside the guest (virtio-net), but I can't tell how it associates which one is the rx queue, which one is tx, and which one is the control queue. Inside qemu-vhost module, when it sets the notifiers for vhost, I see that it only reaches the virtio-dev's bus, but I don't actually see it referencing/matching a virtio-dev queue with a vhost-dev queue(I don't see anything like set_notifiers(vhost-dev.vq[i], virtio-dev.vq[i]). I don't see how it manages NOT to install a notifier for the control virtqueue along with those for rx/tx virtqueues.
[Qemu-devel] [valgrind] getauxval AT_DCACHEBSIZE failed
I'm trying to test something I'm working on qemu-kvm and I ran into a segfault. I usually use valgrind to deal with these kind of issues, but when I try to used it on qemu it goes nuts and shows me this error and closes: getauxval AT_DCACHEBSIZE failed Is it because I use hugetlb ? Or do any of you guys have an idea why this is happening? I've found the same issue on google, but it was related to a bad Ubuntu valgrind build in an official repo, but I'm using everything built from yocto. This is the full command I'm issuing: valgrind ./qemu-system-ppc -enable-kvm -nographic -m 1024 -M ppce500 -kernel uImage-p3041ds.bin -initrd fsl-image-core-p3041ds.ext2.gz -append root=/dev/ram rw console=ttyS0,115200 ramdisk_size=67186297 -mem-path /var/lib/hugetlbfs/pagesize-4MB -device virtio-crypto-pci
[Qemu-devel] unhandled signal 11
How can I debug errors like this: qemu-system-ppc[2273]: unhandled signal 11 at 0008 nip 1007ed08 lr 103404e0 code 30001 ? I know it is a ppc binary, but I am asking a general question. From a first impression it looks a little like Linux Kernel Oops messages. How can I actually find out the line in my code that is generating this thing?
[Qemu-devel] [GSoC] project proposal
Hi, My name is Catalin Vasile and I want to participate with a project for qemu at GSoC. From what I understand from the rules, I can participate with things I could also use for my college projects. This is my last bachelor year and I'm doing my diploma project, which is related to virtualization, more specific qemu-kvm. I'm trying to do a paravirtualized device using virtio and vhost. I've already done some work. To be more exact, I want to make a virtio-crypto device to emulate a virtual cryptographic offloading device that will send jobs from the guest to a vhost that will process the jobs. This mechanism will link CryptoAPI from the guest to the CryptoAPI from the host. This way, whatever it's beneath CryptoAPI from the host will be used as offloading for the guest. Is there a mentor interested in getting involved in this kind of project?
[Qemu-devel] sanity checks and error management
Is there any documentation on error reporting in qemu? I've seen a lot of diverse methods used to report errors in sanity checks and I don't know which one should I use and when.
[Qemu-devel] [vhost-net kernel module]: file.fd = -1 does nothing ?
Why do vhost init/start functions put out with the work of using a vhost_call(VHOST_NET_SET_BACKEND) with file.fd = -1 in qemu vhost-net sources? From what I see in vhost-net kernel module it just returns NULL from get_socket(), and that's it. It doesn't set anything.
[Qemu-devel] qemu sources and makefile system
Hi, I'm new to qemu-devel and I'm trying to add a .c source to qemu. To be more specific, I'm trying to add a file into qemu src/hw/virtio/. I've added common-obj-y += virtio-src.o to the Makefile.objs in that folder and when I'm compiling qemu it seems to compile the sources, but I don't know if they are added to the qemu binary. Is there anything else left to do in order for qemu build system to include my source file? Cata
[Qemu-devel] qemu sources and makefile system
Hi, I'm new to qemu-devel and I'm trying to add a .c source to qemu. To be more specific, I'm trying to add a file into qemu src/hw/virtio/. I've added common-obj-y += virtio-src.o to the Makefile.objs in that folder and when I'm compiling qemu it seems to compile the sources, but I don't know if they are added to the qemu binary. Is there anything else left to do in order for qemu build system to include my source file? Cata
