On Friday 22 January 2010 07:52:49 am Didier Moens wrote: > (initially posted to libvirt-us...@redhat.com, but by request of Daniel > P. Berrange cross-posted to this list) > > > Dear all, > > > I have been wrestling with this issue for the past few days ; googling > around doesn't seem to yield anything useful, hence this cry for help. > > > > Setup (RHEL5.4) : > > * kernel-2.6.18-164.10.1.el5 > * kvm-83-105.el5 > * libvirt-0.6.3-20.el5 > * net.bridge.bridge-nf-call-{arp,ip,ip6}tables = 0 > * tested with/without jumbo frames > > > - I am running several RHEL5.4 KVM virtio guest instances on a Dell PE > R805 RHEL5.4 host. Host and guests are fully updated ; I am using iperf > to test available bandwidth from 3 different locations (clients) in the > network to both the host and the guests . > > - To increase both bandwidth and fail-over, 3 1Gb network interfaces > (BCM5708, bnx2 driver) on the host are bonded (802.3ad) to a 3 Gb/s > bond0, which is bridged. As all guest interfaces are connected to the > bridge, I would expect total available bandwidth to all guests to be in > the range of 2-2.5 Gb/s. > > - Testing with one external client connection to the bare metal host > yields approx. 940 Mb/s ; > > - Testing with 3 simultaneous connections to the host yields 2.5 Gb/s, > which confirms a successful bonding setup. > > > Problem : > > Unfortunately, available bandwidth to the guests proves to be problematic : > > 1. One client to one guest : 250-600 Mb/s ; > 2a. One client to 3 guests : 300-350 Mb/s to each guest, total not > exceeding 980 Mb/s; > 2b. Three clients to 3 guests : 300-350 Mb/s to each guest ; > 2c. Three clients to host and 2 guests : 940 Mb/s (host) + 500 Mb/s to > each guest. > > > Conclusions : > > 1. I am experiencing a 40% performance hit (600 Mb/s) on each individual > virtio guest connection ;
I don't know what all features RHEL5.4 enables for kvm, but that doesn't seem outside the realm of possibility. Especially depending on what OS is running in the guest. I think RHEL5.4 has an older version of virtio, but I won't swear to it. Fwiw, I get ~1.5Gbps guest to host on a Ubuntu 9.10 guest, ~850mbit/s guest to host on a Windows 7 guest. To get those speeds, I have to up the window sizes a good bit (the default is 8K, those numbers are at 1M). At the default Windows 7 gets ~250mbit/s. > 2. Total simultaneous bandwidth to all guests seems to be capped at 1 > Gb/s ; quite problematic, as this renders my server consolidation almost > useless. I don't know about 802.3ad bonding, but I know the other linux bonding techniques are very hard to benchmark due to the way the mac's are handled. I would start by examining/describing your testing a little more. At the very least what tools you're using to test, etc. would be helpful. > > > I could bridge each host network interface separately and assign guest > interfaces by hand, but that would defy the whole idea of load balancing > and failover which is provided by the host bonding. > > > > Any ideas anyone, or am I peeking in the wrong direction (clueless > setup, flawed testing methodology, ...) ? > > > > I monitor the list; a CC: would be appreciated. > > Thanks in advance for any help, > Didier -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html