On Tue, 21 Apr 2026 at 18:06, Jorge Moreira <[email protected]> wrote: > > > Do you know other backends we should check? > > CrosVm's: > https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend
Which do exactly the same https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596 No? > > > On Tue, Apr 21, 2026 at 12:55 AM Stefano Garzarella <[email protected]> > wrote: > > > > On Tue, 21 Apr 2026 at 02:48, Jorge Moreira <[email protected]> wrote: > > > > > > > So it’s fine to continue in this direction, but I might add these > > > words more in the "Migration" section than here, since we’re talking > > > about an optional state migration here. For example after "No further > > > update must be done before rings are restarted." > > > Or in the "Ring states" section, where we can clarify how to restart a > > > ring after a migration. Or in both :-) > > > > > > I admit I don't know what that LOG_FD is for or what a "userfault-fd" > > > is, so I don't know how they relate to vring state. The paragraph I > > > modified discusses which part of the back-end state is in fact handled > > > by the front-end, which somewhat matches the "vring-can/should-kick" > > > idea. That being said, I don't have a strong preference for where this > > > should go. Repeating it in "Ring states" makes sense to me. > > > > I didn't express myself clearly, but my point is that discussing this > > in the "Migrating back-end state" section is misleading. We should > > address it first because it has nothing to do with state; it's also > > needed for devices that don't require any state transfer. > > > > > > > > > > > > QEMU's libvhost-user implementation already starts the ring when > > > VHOST_USER_SET_VRING_KICK is received. This makes more sense than > > > waiting for the kick fd since a back-end that uses polling (peeking at > > > the vring in memory) shouldn't need to monitor the kick fd. See below > > > though about races between the kick fd and vhost-user protocol messages. > > > > Just a note, we do the same in the rust-vmm vhost-user-backend crate as > > well. > > > > > Starting rings when SET_VRING_KICK is received technically violates > > > the vhost-user protocol as defined in the spec. QEMU may support > > > back-ends behaving this way, but other VMMs may not. Changing the spec > > > accordingly could potentially cause other implementations not > > > depending on QEMU's libvhost-user to become non-compliant, I prefer to > > > avoid this. > > > > This is a good point and essentially what changed my mind. > > > > However, this doesn't change anything for those frontends that also > > send the kick (I don't know which ones do this, but QEMU doesn't, and > > I don't think CH does either, since it uses the rust-vmm crates); they > > continue to work just the same. > > > > As for the backends, we’ve seen that both libvhost-user in QEMU and > > vhost-user-backend in rust-vmm do exactly this (start vrings once > > SET_VRING_KICK is received), so perhaps clarifying in this way isn't > > too risky. > > > > Do you know other backends we should check? > > > > > > > > > Note that monitoring the kick fd may be used to avoid races between > > > > the kick fd and vhost-user protocol messages, so in practice back-end > > > > implementors may still want to start the virtqueue when the kick fd > > > > becomes readable > > > > > > Implementations having to do in practice what the spec says today is > > > another reason not to change the it. > > > > > > While starting the vrings on SET_VRING_KICK could solve the state > > > machine issue, it still won't notify the back-end that buffers are > > > ready (the driver won't do this). Non-polling back-ends depend on this > > > kick, especially for queues where data flows only from the driver to > > > the back-end. Most implementations likely attempt to read from the > > > queue only after receiving the kick. > > > > Once the vrings are set, the VMM and the device have exactly the same > > view of the vring state, so IMO the kick is just some extra step (I'm > > fine to add it just because the spec wasn't clear). The device can > > start in the same way without it. > > > > > > > > ------ > > > Thanks for the feedback! I'll wait for consensus before making further > > > modifications. > > > > Thanks! > > Stefano > > > > > > > > > > > On Mon, Apr 20, 2026 at 11:18 AM Stefan Hajnoczi <[email protected]> > > > wrote: > > > > > > > > On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote: > > > > > Thanks for starting the discussion here, let me add also Hanna, > > > > > German, and > > > > > Stefan in CC that can help us. > > > > > > > > > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote: > > > > > > Migration of back-end state happens while the device is suspended > > > > > > (i.e > > > > > > all vrings are stopped). To resume normal operation on the > > > > > > destination, > > > > > > the vrings need to be started again with a kick (either a write on > > > > > > the > > > > > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). > > > > > > While > > > > > > > > > > It's true that in the spec we have: > > > > > "Each ring is initialized in a stopped and disabled state. The > > > > > back-end > > > > > must start a ring upon receiving a kick (that is, detecting that > > > > > file > > > > > descriptor is readable) on the descriptor specified by > > > > > VHOST_USER_SET_VRING_KICK or receiving the in-band message > > > > > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving > > > > > VHOST_USER_GET_VRING_BASE." > > > > > > > > > > But IMO this applies when a driver is not yet loaded. > > > > > When we are migrating, the driver could be already loaded. So, in the > > > > > new > > > > > device running in the destination, IMO we should consider the ring > > > > > already > > > > > started or add some messages to tell to the device: "hey, the device > > > > > was > > > > > already started, this is a migration and it's completed". > > > > > > > > > > Sending a kick from the frontend, seems more an hack here. > > > > > > > > > > That said, for example, in subprojects/libvhost-user/libvhost-user.c > > > > > IIUC > > > > > the virtqueue is started when the SET_VRING_KICK is handled by > > > > > vu_set_vring_kick_exec(), but not sure how compliant it is. > > > > > > > > > > > these notifications are typically sent by the driver, it has no > > > > > > reason > > > > > > to send them in the destination if it already sent them in the > > > > > > source as > > > > > > the driver is unaware that a migration took place. Therefore it > > > > > > should > > > > > > be the responsibility of the vhost-user front-end to ensure these > > > > > > vrings > > > > > > are started. This is particularly necessary for queues where data > > > > > > only > > > > > > flows from device to driver, such as those used by the vsock and > > > > > > input > > > > > > devices. > > > > > > > > > > Exactly, so IMO we should not use the kick, but maybe add something > > > > > new or > > > > > clarify what to do after the migration. > > > > > > > > > > For example in the "Migrating back-end state" we have: > > > > > "Migrating device state involves transferring the state from one > > > > > back-end, called the source, to another back-end, called the > > > > > destination. > > > > > After migration, the destination transparently resumes operation > > > > > without > > > > > requiring the driver to re-initialize the device at the VIRTIO > > > > > level." > > > > > > > > > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly > > > > > to > > > > > inform the new device about the state: "there isn't any state to > > > > > transfer, > > > > > but I notify you that the device was already initialized, so the > > > > > vrings can > > > > > be started". > > > > > > > > > > > > > > > > > This behavior is already used by some qemu vhost-user front-ends > > > > > > (e.g > > > > > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g > > > > > > CrosVm). > > > > > > > > > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see > > > > > the > > > > > code around the comment "/* Kick right away to begin processing > > > > > requests > > > > > already in vring */" but that one IIUC was introduced more to fix > > > > > devices > > > > > violating specs, so not sure it's a good example to follow: > > > > > > > > > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe > > > > > Author: Yongji Xie <[email protected]> > > > > > Date: Wed Jun 6 21:24:48 2018 +0800 > > > > > > > > > > vhost-user-blk: start vhost when guest kicks > > > > > Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs > > > > > early") > > > > > kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This > > > > > violates > > > > > the virtio spec. But virtio 1.0 transitional devices support this > > > > > behaviour. > > > > > So we should start vhost when guest kicks in this case. > > > > > Signed-off-by: Yongji Xie <[email protected]> > > > > > Signed-off-by: Chai Wen <[email protected]> > > > > > Signed-off-by: Ni Xun <[email protected]> > > > > > Reviewed-by: Stefan Hajnoczi <[email protected]> > > > > > Reviewed-by: Michael S. Tsirkin <[email protected]> > > > > > Signed-off-by: Michael S. Tsirkin <[email protected]> > > > > > > > > > > > > > > > > Adding it to the vhost-user documentation makes it explicit that > > > > > > this > > > > > > strategy is permitted and suggest it to vhost-user front-end > > > > > > authors. > > > > > > Explicitly documenting it is necessary because vring kicks appear > > > > > > designed to originate in the driver, so having some originate in the > > > > > > front-end can be counterintuitive and cause developers to waste time > > > > > > looking for other alternatives or face pushback during code review. > > > > > > > > > > As I pointed out in our discussion in > > > > > https://github.com/rust-vmm/vhost-device/pull/936 > > > > > IMO we should use some in-band messages and not relaying on kicks that > > > > > should be used only by the driver to notify the device about new > > > > > available > > > > > buffers. > > > > > > > > > > That said, I agree that we need to clarify in the specifications > > > > > exactly > > > > > what the backend and frontend should do after a migration to start > > > > > vrings if > > > > > there is no need to exchange a state. > > > > > > > > > > > > > > > Any other opinion? > > > > > > > > IMO no protocol changes are needed but the vhost-user spec should be > > > > tweaked. Hanna worked on device state migration and can confirm/deny > > > > what I'm about to describe. > > > > > > > > QEMU's libvhost-user implementation already starts the ring when > > > > VHOST_USER_SET_VRING_KICK is received. This makes more sense than > > > > waiting for the kick fd since a back-end that uses polling (peeking at > > > > the vring in memory) shouldn't need to monitor the kick fd. See below > > > > though about races between the kick fd and vhost-user protocol messages. > > > > > > > > All of this boils down to the ring state machine. libvhost-user's > > > > behavior is: > > > > 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK. > > > > 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE. > > > > 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE. > > > > > > > > The same sequence of vhost-user protocol messages that is used to > > > > start/stop a device locally (e.g. pause and resume a VM) is the same > > > > that can be used during migration. The ring state machine already exists > > > > and needs to be used when migrating device state. > > > > > > > > The following changes to the vhost-user spec would make this clearer: > > > > > > > > 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. > > > > Note that monitoring the kick fd may be used to avoid races between > > > > the kick fd and vhost-user protocol messages, so in practice back-end > > > > implementors may still want to start the virtqueue when the kick fd > > > > becomes readable. > > > > > > > > 2. Add a clarification to "Migrating back-end state" that the device > > > > must be suspended (see _suspended_device_state) when > > > > VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is > > > > transferred. This is already implicit in "Device state transfer > > > > parameters", but it's not obvious when reading the "Migrating > > > > back-end state" section. > > > > > > > > Stefan > > > > > >
