This patch series implements the Transmit Rate Limiter (TRL) feature for the Intel 82576 (igb) network device emulation in QEMU.
The Transmit Rate Limiter allows the guest OS (or virtual machines in an IOV setup) to limit the transmission bandwidth of individual transmit queues to prevent a single queue from saturating the link. The specifications for TRL can be found in the Intel 82576eb datasheet. https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82576eb-gigabit-ethernet-controller-datasheet.pdf Motivation ========= Upstream Linux Kernel would like a universally accesible device to run tests for Live Update [1] (swapping kernel versions). The primary test case is verifying that a device can do continuous DMA for the duration of Live Update (about 4 minutes). IGB is a good candidate device for this use case because it's ubiquitous on many systems, emulated in QEMU, and can do DMA memcpy operations using its loopback functionality. I've written a VFIO selftest driver for IGB [2] which uses the loopback functionality do DMA. This driver works for both the real and emulated version of IGB. The problem is that the IGB driver can only do DMA for about 1 second when using the QEMU device, because QEMU does not emulate the line rate of the hardware, i.e. data transmission is instantaneous. However, with this implementation of TRL + enabling jumbo packets, I was able to get the driver to do DMA for 5 minutes, which easily meets the 4 minute requirement for Live Update testing. [1] https://docs.kernel.org/core-api/liveupdate.html [2] https://lore.kernel.org/kvm/[email protected]/ Implementation Details: ======================= TRL is implemented per-queue. When a queue is configured with a target rate via the TRLRC register, QEMU calculates the transmission delay dynamically for each packet based on its size (Layer 2 packet length) and the target bandwidth. If the calculated transmission time exceeds the current virtual time, QEMU's TX loop defers further packet processing, marks the queue as throttled, and schedules a timer. When the timer expires, the throttled state is cleared, packet transmission resumes, and a writeback interrupt is raised to notify the driver of the completion. Excluded Features / Simplifications: ==================================== I excluded the following TRL features that are mentioned in the IGB datasheet because they weren't necessary for my relatively simple use case. I figure they can be added later if people need them, or I can add them in a later version of the series if there's a use case for them now. The numbers in parentheses are citations for sections in the IGB datasheet. * Rate scheduling table (7.8): real hardware uses a rate scheudling table and a time stamp table instead of QEMU timers to determine when each rate-limited queue is allowed to transmit next. * Max Memory Window (7.8): MMW allows a queue to transmit a burst of traffic if it has been idle or delayed by other traffic, allowing "catch up" to its average target rate. * TRLDCS register (4.5.12.1): This register is used to avoid race conditions between hardware and software when re-enabling/configuring TRL when the link status/speed changes. However link changes in QEMU are synchronous, so this isn't needed. Testing ======= This series includes a Qtest for IGB which verifies that the transmission rate is throttled when TRL is enabled, and not throttled when TRL is disabled. Ran the following command to test it: ninja -C build && \ QTEST_QEMU_BINARY=./build/qemu-system-x86_64 \ ./build/tests/qtest/qos-test \ -p /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/igb/igb-tests Signed-off-by: Josh Hilke <[email protected]> Josh Hilke (5): igb: Define TRL registers and core state igb: Implement TRL register write and configuration igb: Refactor TX path to return processed byte count igb: Implement TRL throttling and timer resumption igb: Add a test for Transmit Rate Limiter hw/net/igb_common.h | 1 + hw/net/igb_core.c | 134 ++++++++++++++++++++++++++++++++-- hw/net/igb_core.h | 11 +++ hw/net/igb_regs.h | 11 +++ tests/qtest/igb-test.c | 161 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 313 insertions(+), 5 deletions(-) -- 2.54.0.1136.gdb2ca164c4-goog
