--- Begin Message ---
Package: qemu-kvm
Version: 1.0+dfsg-9
Severity: normal
I cannot get optimal network througput on KVM guest using Debian Wheezy (and
stable) as KVM host.
It is not horribly bad, just not good compared to relevant alternatives.
I have tried Ubuntu Server 11.10, Proxmox 1.9, Proxmox 2.0 and Fedora 17 Alpha
as 1:1 replacement for the Debian KVM host using the same guests (just
preserving the LVM volumes between installs) and they all manage about 20
gbit/s guest to guest using a simple iperf test, while Debian only manages
about 2.3 gbit/s with high CPU usage. The CPU usage is generally much higher on
all guest network activity on Debian and are in some cases not able to even
saturate a gigabit link (when coming from other subnet) without maxing a CPU
core where the other KVM hosts barely breaks a sweat.
Disc IO is very good and the guests feels snappy so it doesn't seem like there
is something really wrong, just something suboptimal with the networking.
The issue follows only the host OS as the guests have been the same in all
comparisons (Debian Wheezy)
To reproduce:
------------
Install Debian Wheezy in guests (minimal with SSH and ntp)
Install iperf via apt-get
Configure network
Run test:
guest1: iperf -s
guest2: iperf -c <iperf-server> -i 2 -t 33333
My results:
----------------------------
- Guest to guest performance via local bridge: ~2.3 gbit/s, very high CPU usage
on vhost-$PID and kvm process on host
- Physical server to guest on same subnet: ~940 mbit/s but with very high CPU
usage on vhost-$PID and kvm process on host
- Physical server to guest via router: ~850 mbit/s with very high CPU usage on
vhost-$PID and kvm process on host (why is routed traffic slower than switched
on the guest??)
- Physical server to kvm host via router (just to verify that the router is not
the issue): ~940 mbit/s with almost no CPU usage
Expected results after comparison with other KVM hosts everything else the same:
-------------------------
- Guest to guest performance via local bridge: ~20 gbit/s, high CPU usage
- Physical server to guest on same subnet: ~940 mbit/s with low CPU usage on
vhost-$PID and a bit higher on kvm process on host
- Physical server to guest via router: ~940 mbit/s with low CPU usage on
vhost-$PID and a bit higher on kvm process on host
- Physical server to kvm host via router (just to verify that the router is not
the issue): ~940 mbit/s with almost no CPU usage (the same as my current
results)
Compare results with other OSes on same machine (guest to guest via bridge):
Ubuntu Server 11.10 (virtualization host): ~19 gbit
Proxmox VE 2.0: ~20 gbit/s
Fedora 17 alpha: ~20 gbit/s
VMWare ESXi 5 with VMXNET3: ~22 gbit/s
Host details:
---------------------
OS: Debian Wheezy (testing), kernel 3.2.0-2-amd64, currently based on 3.2.12
virsh qemu-monitor-command --hmp mail 'info version'
1.0.0 (Debian qemu-kvm 1.0+dfsg-9)
virsh qemu-monitor-command --hmp mail 'info kvm':
kvm support: enabled
lsmod | grep kvm:
kvm_intel 121968 9
kvm 287572 1 kvm_intel
lsmod | grep vhost:
vhost_net 27436 3
tun 18337 7 vhost_net
macvtap 17598 1 vhost_net
KSM enabled or disabled makes no difference on the results but here are my
parameters with it on:
echo "1" > /sys/kernel/mm/ksm/run
echo "200" > /sys/kernel/mm/ksm/sleep_millisecs
Output from ps -ef of running guest:
/usr/bin/kvm -S -M pc-0.15 -cpu
core2duo,+lahf_lm,+rdtscp,+avx,+osxsave,+xsave,+aes,+popcnt,+x2apic,+sse4.2,+sse4.1,+pdcm,+xtpr,+cx16,+tm2,+est,+smx,+vmx,+ds_cpl,+dtes64,+pclmuldq,+pbe,
+tm,+ht,+ss,+acpi,+ds
-enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name mail -uuid
ccace357-783d-ce9f-444a-419445ee601d -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/mail.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -drive
file=/dev/raid10/mail,if=none,id=drive-virtio-disk0,format=raw,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f7:25:33,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Server hardware (seems to be the same issue regardless of server used):
- Intel(R) Xeon(R) CPU E31220 @ 3.10GHz Quad Core
- 16 GB ECC RAM
- Supermicro X9SCI-LN4F (Quad Intel Server NICs using e1000e)
- System disc: Corsair SSD Force Series 3 60GB
- Storage for guests: LVM images on directly attached RAID10
Guest details:
---------
OS: Debian Wheezy (testing), kernel 3.2.0-2-amd64, currently based on 3.2.12
root@mail:~# lsmod | grep virtio:
virtio_balloon 12832 0
virtio_blk 12874 3
virtio_net 17808 0
virtio_pci 13207 0
virtio_ring 12969 4 virtio_pci,virtio_net,virtio_blk,virtio_balloon
virtio 13093 5
virtio_ring,virtio_pci,virtio_net,virtio_blk,virtio_balloon
I have tried:
----------------
- Replacing Debian Wheezy with Debian Squeeze (stable, kernel 2.6.32-xx) - even
worse results
- Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and config based
on Debians included config - no apparent change
- Extracted the kernel-config file from Fedora 17 alphas kernel and used this
to compile a new kernel based on Debian Wheezys kernel source - slightly worse
results
- Installing Proxmox VE 2.0 kernel in Debian. Results are the same
- ...in addition to exchanging Debian with Ubuntu Server 11.10, Fedora 17
alpha, Proxmox 1.9 and 2.0 and ESXi 5 which all have expected network
performance using virtio.
Please optimize KVM/vhost in Debian so it performs like the other alternatives.
-- Package-specific info:
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
stepping : 7
microcode : 0x1b
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm
ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 6186.08
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
stepping : 7
microcode : 0x1b
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm
ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 6185.89
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
stepping : 7
microcode : 0x1b
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm
ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 6185.90
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
stepping : 7
microcode : 0x1b
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm
ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 6185.90
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
-- System Information:
Debian Release: wheezy/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages qemu-kvm depends on:
ii adduser 3.113+nmu1
ii ipxe-qemu 1.0.0+git-20120202.f6840ba-3
ii libaio1 0.3.109-2
ii libasound2 1.0.25-2
ii libbluetooth3 4.99-2
ii libbrlapi0.5 4.3-2
ii libc6 2.13-27
ii libcurl3-gnutls 7.25.0-1
ii libglib2.0-0 2.30.2-6
ii libgnutls26 2.12.18-1
ii libiscsi1 1.0.1-1
ii libjpeg8 8d-1
ii libncurses5 5.9-4
ii libpng12-0 1.2.49-1
ii libpulse0 1.1-3+b1
ii librados2 0.43-1
ii librbd1 0.43-1
ii libsasl2-2 2.1.25.dfsg1-4
ii libsdl1.2debian 1.2.15-2
ii libspice-server1 0.10.1-2
ii libtinfo5 5.9-4
ii libuuid1 2.20.1-4
ii libvdeplug2 2.3.2-4
ii libx11-6 2:1.4.4-4
ii python 2.7.2-10
ii qemu-keymaps 1.0.1+dfsg-1
ii qemu-utils 1.0.1+dfsg-1
ii seabios 1.6.3-2
ii vgabios 0.7a-2
ii zlib1g 1:1.2.6.dfsg-2
Versions of packages qemu-kvm recommends:
ii bridge-utils 1.5-2
ii iproute 20120319-1
Versions of packages qemu-kvm suggests:
pn debootstrap <none>
pn samba <none>
pn vde2 <none>
-- no debconf information
--- End Message ---
