Re: Virtio network performance on Debian
On 12.04.2012 11:42, Hans-Kristian Bakke wrote: > Hi > > For some reason I am not able to get good network performance using > virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu > hosts then). The issue has been identified, after Hans-Kristian gave me access to his machine and I did alot of testing. And as usual, the root cause was very stupid... ;) In last release of debian qemu-kvm package I changed the way how debian package version string propagates to build procedure -- before it was a patch grabbing values from debian/version, but in last release I used --with-pkgversion configure flag. This resulted in the following change: - QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 2003-2008 Fabrice Bellard + QEMU emulator version 1.0 (Debian qemu-kvm 1.0+dfsg-9), Copyright (c) 2003-2008 Fabrice Bellard As it turns out, libvirt parses `qemu -version' output and looks for " (qemu-kvm-" string in there, and if it is found, libvirt enables some "extra" features. Important for us was setting IFF_VNET_HDR flag for a tap device, -- apparently this flag makes a HUGE difference in networking speed, especially when using vhost_net. Obviously this is a change unique to debian, and I never thought about such an.. "interesting" effect it may give us. This is a libvirt bug actually, since support of vnet_hdr can be determined by other means, and since upstream qemu now has almost everything from qemu-kvm, and it wants to be fast too. But since qemu[-kvm] has a long history of changing features, it is difficult to blame libvirt that much... Oh well. Thanks! /mjt -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance on Debian
Replacing the current qemu-kvm in Debian Wheezy (based on kernel 3.2.X) (qemu-kvm_1.0+dfsg-9_amd64.deb) with qemu-kvm_1.0+dfsg-8_amd64.deb gives me the following performance numbers: guest to guest: ~19.1 gbit/s host to guest: ~27.3 gbit/s In other words I think it is safe to say that there is nothing in the current Debian Wheezy kernel preventing KVM from performing exceptionally good. Hopefully the issue with the newest qemu-kvm Debian package gets solved so this will continue also in the future. Debian Squeeze isn't anywhere near Debian Wheezy in virtio/vhost performance for the previously given reasons from the Debian qemu-kvm maintainer. Regards, Hans-Kristian On Mon, Apr 16, 2012 at 13:29, Alexandre DERUMIER wrote: > > note: > proxmox2 kernel is based on 2.6.32-220.7.1.el6 RHEL6.2 kernel. > + qemu-kvm git. > > > - Mail original - > > De: "Stefan Pietsch" > À: "Hans-Kristian Bakke" > Cc: kvm@vger.kernel.org > Envoyé: Lundi 16 Avril 2012 11:01:16 > Objet: Re: Virtio network performance on Debian > > On 12.04.2012 09:42, Hans-Kristian Bakke wrote: > > Hi > > > > For some reason I am not able to get good network performance using > > virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu > > hosts then). > > Disc IO is very good and the guests feels snappy so it doesn't seem > > like there is something really wrong, just something suboptimal with > > the networking. > > [..] > > > I have tried: > > > > - Replacing Debian Wheezy with Debian Squeeze (stable, kernel > > 2.6.32-xx) - even worse results > > - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and > > config based on Debians included config - no apparent change > > - Extracted the kernel-config file from Fedora 17 alphas kernel and > > used this to compile a new kernel based on Debian Wheezys kernel > > source - slightly worse results > > - ...in addition to exchanging Debian with Fedora 17 alpha, Proxmox > > 1.9 and 2.0 and ESXi 5 which all have expected network performance > > using virtio. > > > > > > So, I am at a loss here. I does not seem to be kernel config related > > (as using Fedoras config on Debian kernel source didn't do anything > > good) so I think it must be either a kernel patch that red hat kernel > > based distros uses to make virtio/vhost much more efficient or perhaps > > something with Debians qemu-version, bridging or something. > > > I have made some tests with a Debian Squeeze KVM host running with the > Linux Kernel 2.6.39 from backports and the Kernel version 2.6.32-11-pve > from Proxmox. > > (http://download.proxmox.com/debian/dists/squeeze/pve/binary-amd64/pve-kernel-2.6.32-11-pve_2.6.32-66_amd64.deb) > > Network performance between two virtual machines on the same host is > significantly slower with the Debian kernel: > > 2.6.39-bpo.2-amd64 : 1.31 Gbits/sec > 2.6.32-11-pve : 2.20 Gbits/sec > > iperf tests between a virtual machine and the KVM host connected to the > same local bridge interface showed similar results. > > Are there other people who can confirm this? > > > Regards, > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > > -- > > > > > Alexandre D erumier > Ingénieur Système > Fixe : 03 20 68 88 90 > Fax : 03 20 68 90 81 > 45 Bvd du Général Leclerc 59100 Roubaix - France > 12 rue Marivaux 75002 Paris - France > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance on Debian
note: proxmox2 kernel is based on 2.6.32-220.7.1.el6 RHEL6.2 kernel. + qemu-kvm git. - Mail original - De: "Stefan Pietsch" À: "Hans-Kristian Bakke" Cc: kvm@vger.kernel.org Envoyé: Lundi 16 Avril 2012 11:01:16 Objet: Re: Virtio network performance on Debian On 12.04.2012 09:42, Hans-Kristian Bakke wrote: > Hi > > For some reason I am not able to get good network performance using > virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu > hosts then). > Disc IO is very good and the guests feels snappy so it doesn't seem > like there is something really wrong, just something suboptimal with > the networking. [..] > I have tried: > > - Replacing Debian Wheezy with Debian Squeeze (stable, kernel > 2.6.32-xx) - even worse results > - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and > config based on Debians included config - no apparent change > - Extracted the kernel-config file from Fedora 17 alphas kernel and > used this to compile a new kernel based on Debian Wheezys kernel > source - slightly worse results > - ...in addition to exchanging Debian with Fedora 17 alpha, Proxmox > 1.9 and 2.0 and ESXi 5 which all have expected network performance > using virtio. > > > So, I am at a loss here. I does not seem to be kernel config related > (as using Fedoras config on Debian kernel source didn't do anything > good) so I think it must be either a kernel patch that red hat kernel > based distros uses to make virtio/vhost much more efficient or perhaps > something with Debians qemu-version, bridging or something. I have made some tests with a Debian Squeeze KVM host running with the Linux Kernel 2.6.39 from backports and the Kernel version 2.6.32-11-pve from Proxmox. (http://download.proxmox.com/debian/dists/squeeze/pve/binary-amd64/pve-kernel-2.6.32-11-pve_2.6.32-66_amd64.deb) Network performance between two virtual machines on the same host is significantly slower with the Debian kernel: 2.6.39-bpo.2-amd64 : 1.31 Gbits/sec 2.6.32-11-pve : 2.20 Gbits/sec iperf tests between a virtual machine and the KVM host connected to the same local bridge interface showed similar results. Are there other people who can confirm this? Regards, Stefan -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- -- Alexandre D erumier Ingénieur Système Fixe : 03 20 68 88 90 Fax : 03 20 68 90 81 45 Bvd du Général Leclerc 59100 Roubaix - France 12 rue Marivaux 75002 Paris - France -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance on Debian
On 16.04.2012 13:01, Stefan Pietsch wrote: > On 12.04.2012 09:42, Hans-Kristian Bakke wrote: >> Hi >> >> For some reason I am not able to get good network performance using >> virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu >> hosts then). >> Disc IO is very good and the guests feels snappy so it doesn't seem >> like there is something really wrong, just something suboptimal with >> the networking. > > [..] > >> I have tried: >> >> - Replacing Debian Wheezy with Debian Squeeze (stable, kernel >> 2.6.32-xx) - even worse results >> - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and >> config based on Debians included config - no apparent change >> - Extracted the kernel-config file from Fedora 17 alphas kernel and >> used this to compile a new kernel based on Debian Wheezys kernel >> source - slightly worse results >> - ...in addition to exchanging Debian with Fedora 17 alpha, Proxmox >> 1.9 and 2.0 and ESXi 5 which all have expected network performance >> using virtio. >> >> So, I am at a loss here. I does not seem to be kernel config related >> (as using Fedoras config on Debian kernel source didn't do anything >> good) so I think it must be either a kernel patch that red hat kernel >> based distros uses to make virtio/vhost much more efficient or perhaps >> something with Debians qemu-version, bridging or something. Hans-Kristian submitted a bugreport to Debian BTS about this, http://bugs.debian.org/668594 . In this case it turns out to be a problem with particular qemu-kvm userspace binary build, maybe the toolchain used, maybe something else - I don't yet know. This is not the first time this version is mentioned as being slow (slower) with network transfers, here's another bugreport which I merely ignored till now -- see http://bugs.debian.org/665046 . I'm working on this, or trying to anyway. > I have made some tests with a Debian Squeeze KVM host running with the > Linux Kernel 2.6.39 from backports and the Kernel version 2.6.32-11-pve > from Proxmox. > > (http://download.proxmox.com/debian/dists/squeeze/pve/binary-amd64/pve-kernel-2.6.32-11-pve_2.6.32-66_amd64.deb) These are very different. Note that 2.6.32 does not have vhost-net support which is used by Hans-Kristian and speeds up network operations dramatically. > Network performance between two virtual machines on the same host is > significantly slower with the Debian kernel: > > 2.6.39-bpo.2-amd64 : 1.31 Gbits/sec > 2.6.32-11-pve : 2.20 Gbits/sec Please try current debian kernel. There were a few issues with that 2.6.32 backport, but I don't know the details and don't know if these affects qemu/kvm in any way or not. And again, if you really want good networking speeds, you should use vhost-net which is is not supported by 2.6.32 kernel. > iperf tests between a virtual machine and the KVM host connected to the > same local bridge interface showed similar results. > > Are there other people who can confirm this? Thanks, /mjt -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance on Debian
On 12.04.2012 09:42, Hans-Kristian Bakke wrote: > Hi > > For some reason I am not able to get good network performance using > virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu > hosts then). > Disc IO is very good and the guests feels snappy so it doesn't seem > like there is something really wrong, just something suboptimal with > the networking. [..] > I have tried: > > - Replacing Debian Wheezy with Debian Squeeze (stable, kernel > 2.6.32-xx) - even worse results > - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and > config based on Debians included config - no apparent change > - Extracted the kernel-config file from Fedora 17 alphas kernel and > used this to compile a new kernel based on Debian Wheezys kernel > source - slightly worse results > - ...in addition to exchanging Debian with Fedora 17 alpha, Proxmox > 1.9 and 2.0 and ESXi 5 which all have expected network performance > using virtio. > > > So, I am at a loss here. I does not seem to be kernel config related > (as using Fedoras config on Debian kernel source didn't do anything > good) so I think it must be either a kernel patch that red hat kernel > based distros uses to make virtio/vhost much more efficient or perhaps > something with Debians qemu-version, bridging or something. I have made some tests with a Debian Squeeze KVM host running with the Linux Kernel 2.6.39 from backports and the Kernel version 2.6.32-11-pve from Proxmox. (http://download.proxmox.com/debian/dists/squeeze/pve/binary-amd64/pve-kernel-2.6.32-11-pve_2.6.32-66_amd64.deb) Network performance between two virtual machines on the same host is significantly slower with the Debian kernel: 2.6.39-bpo.2-amd64 : 1.31 Gbits/sec 2.6.32-11-pve : 2.20 Gbits/sec iperf tests between a virtual machine and the KVM host connected to the same local bridge interface showed similar results. Are there other people who can confirm this? Regards, Stefan -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Virtio network performance on Debian
Hi For some reason I am not able to get good network performance using virtio/vhost-net on Debian KVM host (perhaps also valid for Ubuntu hosts then). Disc IO is very good and the guests feels snappy so it doesn't seem like there is something really wrong, just something suboptimal with the networking. Guests are Debian Wheezy (the performance issues seems to be related to the the host only) The test: iperf -s iperf -c -i 2 -t 3 Problem description: Guest to guest performance via local bridge: ~2.3 gbit/s, very high CPU usage on vhost-$PID and kvm process on host Physical server to guest on same subnet: ~940 mbit/s but with very high CPU usage on vhost-$PID and kvm process on host Physical server to guest via router: ~850 mbit/s with very high CPU usage on vhost-$PID and kvm process on host (why is routed traffic slower than switched on the guest??) Physical server to kvm host via router (just to verify that the router is not the issue): ~940 mbit/s with almost no CPU usage Expected results: - Guest to guest performance via local bridge: ~20 gbit/s, high CPU usage Physical server to guest on same subnet: ~940 mbit/s with low CPU usage on vhost-$PID and a bit higher on kvm process on host Physical server to guest via router: ~940 mbit/s with low CPU usage on vhost-$PID and a bit higher on kvm process on host Physical server to kvm host via router (just to verify that the router is not the issue): ~940 mbit/s with almost no CPU usage (the same as my current results..) The expected results are based on this (identical guests, network setup and hardware in all cases, only the host OS changes): - Fedora 17 alpha as 1:1 replacement for Debian Wheezy as KVM host gives expected results - Proxmox 1.9 and 2.0 (Debian based distro, but using RHEL 6.xx based kernels as far as I know) gives expected results - VMware ESXi 5 with VMXNET3 gives even slightly better network performance Details on host: - OS: Debian Wheezy (testing), kernel 3.2.0-2-amd64, currently based on 3.2.12 virsh qemu-monitor-command --hmp mail 'info version' 1.0.0 (Debian qemu-kvm 1.0+dfsg-9) virsh qemu-monitor-command --hmp mail 'info kvm': kvm support: enabled lsmod | grep kvm: kvm_intel 121968 9 kvm 287572 1 kvm_intel lsmod | grep vhost: vhost_net 27436 3 tun18337 7 vhost_net macvtap17598 1 vhost_net Output from ps -ef of running guest: /usr/bin/kvm -S -M pc-0.15 -cpu core2duo,+lahf_lm,+rdtscp,+avx,+osxsave,+xsave,+aes,+popcnt,+x2apic,+sse4.2,+sse4.1,+pdcm,+xtpr,+cx16,+tm2,+est,+smx,+vmx,+ds_cpl,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name mail -uuid ccace357-783d-ce9f-444a-419445ee601d -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/mail.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/raid10/mail,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f7:25:33,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Server hardware (seems to be the same issue regardless of server used): - Intel(R) Xeon(R) CPU E31220 @ 3.10GHz Quad Core - 16 GB ECC RAM - Supermicro X9SCI-LN4F (Quad Intel Server NICs using e1000e) - System disc: Corsair SSD Force Series 3 60GB - Storage for guests: LVM images on directly attached RAID10 Guest: - OS: Debian Wheezy (testing), kernel 3.2.0-2-amd64, currently based on 3.2.12 root@mail:~# lsmod | grep virtio: virtio_balloon 12832 0 virtio_blk 12874 3 virtio_net 17808 0 virtio_pci 13207 0 virtio_ring12969 4 virtio_pci,virtio_net,virtio_blk,virtio_balloon virtio 13093 5 virtio_ring,virtio_pci,virtio_net,virtio_blk,virtio_balloon I have tried: - Replacing Debian Wheezy with Debian Squeeze (stable, kernel 2.6.32-xx) - even worse results - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and config based on Debians included config - no apparent change - Extracted the kernel-config file from Fedora 17 alphas kernel and used this to compile a new kernel based on Debian Wheezys kernel source - slightly worse results - ...in addition to exchanging Debian with Fedora 17 alpha, Proxmox 1.9 and 2.0 and ESXi 5 which all have expected network performance using virtio.