Re: [RFC] proposed 'lem' patch to improve behaviour under emulation
On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo ri...@iet.unipi.it wrote: This patch implements two features for the 'lem' driver that greatly improve the throughput under proper hypervisor. This is joint work with Vincenzo Maffione and Giuseppe Lettieri, I am posting it here for review, will then commit it if there are no objections. The first change is to implement a sysctl to access the 'itr' interrupt moderation register for the devices supported by this driver. It is little more than adding a struct into the device descriptor, and one line to create the dynamic sysctl entry, same as it is done for the other mitigation registers. The second change is more interesting and has huge benefits on througput. Under virtualization, VM exits (which happen every time there is an access to a register of the emulated peripheral) are extremely expensive. In the tx path of the 'lem' driver, there is a write to the TDT register on every packet sent. The patch we propose, if enabled through a sysctl (defaults off, so no change from current behaviour) defers writes to the TDT register when there is a pending transmit interrupt. This means that, together with proper emulation of interrupt mitigation on the hypervisor side, the number of VM exits is dramatically reduced. To give you an idea, on a modern system with qemu-kvm and companion patches, UDP throughput is KVM QEMU standard KVM, standard driver20 Kpps 6.3 Kpps modified KVM, standard driver37 Kpps28 Kpps modified KVM, modified driver 200 Kpps34 Kpps As you can see, on kvm this change gives a 5x speedup to the tx path, which combines nicely with the 2x speedup that comes from supporting interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) the benefits are much lower, as the guest becomes too slow. Patch follows. It would be good if people with real hardware using the 'lem' driver could test it to make sure it does no harm on their devices (in any case the sysctl variable dev.em.0.mit_enable must be set to explicitly enable it at runtime). (for those curious to test it under kvm, i am also attaching a patch that you need to apply to qemu in order to exploit the effect of interrupt mitigation; it is a followup of a similar patch i posted in july to the qemu mailing list, and i will post it the update there as well, shortly. Unfortunately we do not have kvm on freebsd..) A few comments. Thanks! -Garrett Index: if_lem.c === --- if_lem.c(revision 244673) +++ if_lem.c(working copy) @@ -32,6 +32,8 @@ **/ /*$FreeBSD$*/ +#define MITIGATION + gcooper Could you please make MITIGATION into a proper compile time flag via sys/conf/options with a more descript name? #ifdef HAVE_KERNEL_OPTION_HEADERS #include opt_device_polling.h #include opt_inet.h @@ -281,6 +283,9 @@ #define EM_TICKS_TO_USECS(ticks) ((1024 * (ticks) + 500) / 1000) #define EM_USECS_TO_TICKS(usecs) ((1000 * (usecs) + 512) / 1024) +#define MAX_INTS_PER_SEC 8000 +#define DEFAULT_ITR 10/(MAX_INTS_PER_SEC * 256) + gcooper Add parentheses around DEFAULT_ITR (I know the code was just shuffled around, but thought I could ask :)..)? ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] proposed 'lem' patch to improve behaviour under emulation
On Thu, Dec 27, 2012 at 02:26:44AM -0800, Garrett Cooper wrote: On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo ri...@iet.unipi.it wrote: This patch implements two features for the 'lem' driver that greatly improve the throughput under proper hypervisor. This is joint work with Vincenzo Maffione and Giuseppe Lettieri, I am posting it here for review, will then commit it if there are no objections. The first change is to implement a sysctl to access the 'itr' interrupt moderation register for the devices supported by this driver. It is little more than adding a struct into the device descriptor, and one line to create the dynamic sysctl entry, same as it is done for the other mitigation registers. The second change is more interesting and has huge benefits on througput. Under virtualization, VM exits (which happen every time there is an access to a register of the emulated peripheral) are extremely expensive. In the tx path of the 'lem' driver, there is a write to the TDT register on every packet sent. The patch we propose, if enabled through a sysctl (defaults off, so no change from current behaviour) defers writes to the TDT register when there is a pending transmit interrupt. This means that, together with proper emulation of interrupt mitigation on the hypervisor side, the number of VM exits is dramatically reduced. To give you an idea, on a modern system with qemu-kvm and companion patches, UDP throughput is KVM QEMU standard KVM, standard driver20 Kpps 6.3 Kpps modified KVM, standard driver37 Kpps28 Kpps modified KVM, modified driver 200 Kpps34 Kpps As you can see, on kvm this change gives a 5x speedup to the tx path, which combines nicely with the 2x speedup that comes from supporting interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) the benefits are much lower, as the guest becomes too slow. Patch follows. It would be good if people with real hardware using the 'lem' driver could test it to make sure it does no harm on their devices (in any case the sysctl variable dev.em.0.mit_enable must be set to explicitly enable it at runtime). (for those curious to test it under kvm, i am also attaching a patch that you need to apply to qemu in order to exploit the effect of interrupt mitigation; it is a followup of a similar patch i posted in july to the qemu mailing list, and i will post it the update there as well, shortly. Unfortunately we do not have kvm on freebsd..) A few comments. Thanks! -Garrett Index: if_lem.c === --- if_lem.c (revision 244673) +++ if_lem.c (working copy) @@ -32,6 +32,8 @@ **/ /*$FreeBSD$*/ +#define MITIGATION + gcooper Could you please make MITIGATION into a proper compile time flag via sys/conf/options with a more descript name? this is actually going away, with the code compiled in by default ... @@ -281,6 +283,9 @@ #define EM_TICKS_TO_USECS(ticks) ((1024 * (ticks) + 500) / 1000) #define EM_USECS_TO_TICKS(usecs) ((1000 * (usecs) + 512) / 1024) +#define MAX_INTS_PER_SEC 8000 +#define DEFAULT_ITR 10/(MAX_INTS_PER_SEC * 256) + gcooper Add parentheses around DEFAULT_ITR (I know the code was just shuffled around, but thought I could ask :)..)? will do thanks luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] proposed 'lem' patch to improve behaviour under emulation
LOL, it's ironic, my intention in creating lem was to isolate the old pre-PCIE driver from active changes so as to assure it's stability... but virtualization comes around to bit me in the butt :) I guess I'm agreeable in principle with what you're doing Luigi, but can you do me a favor and hold off until I'm technically back from vacation (after the new year) and let me review the code then? Thanks, Jack On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo ri...@iet.unipi.it wrote: This patch implements two features for the 'lem' driver that greatly improve the throughput under proper hypervisor. This is joint work with Vincenzo Maffione and Giuseppe Lettieri, I am posting it here for review, will then commit it if there are no objections. The first change is to implement a sysctl to access the 'itr' interrupt moderation register for the devices supported by this driver. It is little more than adding a struct into the device descriptor, and one line to create the dynamic sysctl entry, same as it is done for the other mitigation registers. The second change is more interesting and has huge benefits on througput. Under virtualization, VM exits (which happen every time there is an access to a register of the emulated peripheral) are extremely expensive. In the tx path of the 'lem' driver, there is a write to the TDT register on every packet sent. The patch we propose, if enabled through a sysctl (defaults off, so no change from current behaviour) defers writes to the TDT register when there is a pending transmit interrupt. This means that, together with proper emulation of interrupt mitigation on the hypervisor side, the number of VM exits is dramatically reduced. To give you an idea, on a modern system with qemu-kvm and companion patches, UDP throughput is KVM QEMU standard KVM, standard driver20 Kpps 6.3 Kpps modified KVM, standard driver37 Kpps28 Kpps modified KVM, modified driver 200 Kpps34 Kpps As you can see, on kvm this change gives a 5x speedup to the tx path, which combines nicely with the 2x speedup that comes from supporting interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) the benefits are much lower, as the guest becomes too slow. Patch follows. It would be good if people with real hardware using the 'lem' driver could test it to make sure it does no harm on their devices (in any case the sysctl variable dev.em.0.mit_enable must be set to explicitly enable it at runtime). (for those curious to test it under kvm, i am also attaching a patch that you need to apply to qemu in order to exploit the effect of interrupt mitigation; it is a followup of a similar patch i posted in july to the qemu mailing list, and i will post it the update there as well, shortly. Unfortunately we do not have kvm on freebsd..) cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] proposed 'lem' patch to improve behaviour under emulation
On Thu, Dec 27, 2012 at 10:46 AM, Jack Vogel jfvo...@gmail.com wrote: LOL, it's ironic, my intention in creating lem was to isolate the old pre-PCIE driver from active changes so as to assure it's stability... but virtualization comes around to bit me in the butt :) I guess I'm agreeable in principle with what you're doing Luigi, but can you do me a favor and hold off until I'm technically back from vacation (after the new year) and let me review the code then? sure, no rush -- i just wanted to have it out for review as it has been ready for a few weeks now. Regarding lem vs em i actually wonder if it wouldn't be better to consolidate the two drivers given the amount of common code. While i understand the desire for stability, i actually wonder if there is much if any leftover hw which uses 'lem' ... outside virtualization! cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] proposed 'lem' patch to improve behaviour under emulation
I don't know, in some ways it might be more interesting to make something just for a virtualized device, however reality is that I have way too many higher priority items to worry about after the new year gets underway than that, even so, we can see... Jack On Thu, Dec 27, 2012 at 6:52 PM, Luigi Rizzo ri...@iet.unipi.it wrote: On Thu, Dec 27, 2012 at 10:46 AM, Jack Vogel jfvo...@gmail.com wrote: LOL, it's ironic, my intention in creating lem was to isolate the old pre-PCIE driver from active changes so as to assure it's stability... but virtualization comes around to bit me in the butt :) I guess I'm agreeable in principle with what you're doing Luigi, but can you do me a favor and hold off until I'm technically back from vacation (after the new year) and let me review the code then? sure, no rush -- i just wanted to have it out for review as it has been ready for a few weeks now. Regarding lem vs em i actually wonder if it wouldn't be better to consolidate the two drivers given the amount of common code. While i understand the desire for stability, i actually wonder if there is much if any leftover hw which uses 'lem' ... outside virtualization! cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
