TL;DR: I'm working on the SEAPATH project, using QEMU with a Linux RT kernel. We faced real-time priority issues where the QEMU main process preempts the vhost thread, causing network latency. We fixed it by setting the vhost thread priority higher than QEMU's parent process using a script. Is it useful for QEMU to support setting vhost priorities directly? I would like to contribute this feature.
--- Hello community, I am working on a project called SEAPATH, which is hosted by the Linux Energy Foundation (https://github.com/seapath). This project primarily uses virtualization via QEMU with libvirt, running a Linux RT kernel (6.6). We utilize the vhost interface with the vhost kernel driver to achieve virtualization with real-time constraints. Specifically, we measure network latencies from the arrival of network packets on the hypervisor side to their processing on the VM side. On the VM side, we use a 3-core VM. On the hypervisor side, network packets are received on a dedicated interface, redirected to an OVS virtual port, and then sent to the VM using a vhost interface. This vhost interface is represented in the kernel by a process named vhost-<QEMU PID>. These processes are isolated on a dedicated hypervisor CPU core. Network packets interface (hypervisor -- IRQ RX process) --> OVS kernel driver (hypervisor -- IRQ RX process) --> Vhost kernel driver (hypervisor -- vhost-<QEMU PID> process) When using QEMU, we encountered some issues with real-time priority. Specifically, we observed that in certain cases, the QEMU main process would preempt the vhost thread, causing significant latency in our network processing pipeline since both QEMU and the vhost thread have the same priority. We resolved this issue by setting the vhost thread priority higher than the QEMU process parent. To make this fix permanent, we use a QEMU script hook that performs a chrt on the vhost thread after its creation. By default, the vhost thread is created with the same priority as the QEMU parent process. This behavior originates from the vhost kernel module (drivers/vhost/vhost.c), where the created kworker thread inherits the priority of the process responsible for the syscall (in our case, QEMU). Currently, there is no way for QEMU to instruct the vhost driver on the priority to assign to the created vhost thread. Is the issue we encountered a corner case? Would it be beneficial for the QEMU project to have the ability to set the vhost priority directly from QEMU (other than using a QEMU hook script)? If so, I would be very pleased to contribute this feature. Regards, Paul Le Guen de Kerneizon, Savoir-faire Linux
