RE: virtio + vhost-net performance issue - preadv ?
David- Thanks for the followup. That is disappointing, and I wish I knew why the performance is so poor. With the kernel and qemu replaced, I don't know where the limitation is - raising the MTU makes no difference, and I also tried a few different kernels inside the guest. The network stack can natively support high throughput (~23Gbps on IP over Infiniband on these nodes), so I'm at a loss. Maybe it's the lack of preadv? Ben Clay rbc...@ncsu.edu -Original Message- From: David Cruz [mailto:david.c...@gigas.com] Sent: Wednesday, November 14, 2012 1:27 AM To: Ben Clay Subject: Re: virtio + vhost-net performance issue - preadv ? Got the same results here last year when we were testing. In the end, we use only CentOS6. And even more, we changed the kernel to 3.5.5 due to unstable Windows virtualization when using several Windows Server in the same Hypervisor. 2-4 GBit/s in Centos5 is acceptable. I think that's the max you can get in that version. David 2012/11/14 Ben Clay : > I have a working copy of libvirt 0.10.2 + qemu 1.2 installed on a > vanilla up-to-date (2.6.32-279.9.1) CentOS 6 host, and get very good > VM <-> VM network performance (both running on the same host) using > virtio. I have cgroups set to cap the VMs at 10Gbps and iperf shows > I'm getting exactly 10Gbps. > > I copied these VMs to a CentOS 5 host and installed libvirt 1.0 + qemu 1.2. > However, the best performance I can get in between the VMs (again > running on the same host) is ~2Gbps. In both cases, this is over a > bridged interface with static IPs assigned to each VM. I've also > tried virtual networking with NAT or routing, yielding the same results. > > I figured it was due to vhost-net missing on the older CentOS 5 > kernel, so I installed 2.6.39-4.2 from ELRepo and got the > /dev/vhost-net device and vhost processes associated with each VM: > > ]$ lsmod | grep vhost > vhost_net 28446 2 > tun23888 7 vhost_net > > ]$ ps aux | grep vhost- > root 9628 0.0 0.0 0 0 ?S17:57 0:00 > [vhost-9626] > root 9671 0.0 0.0 0 0 ?S17:57 0:00 > [vhost-9670] > > ]$ ls /dev/vhost-net -al > crw--- 1 root root 10, 58 Nov 13 15:19 /dev/vhost-net > > After installing the new kernel, I also tried rebuilding libvirt and > qemu, to no avail. I also disabled cgroups, just in case it was > getting in the way, as well as iptables. I can see the virtio_net > module loaded inside the guest, and using virtio raises my performance > from <400Mbps to 2Gbps, so it does make some improvement. > > The only differences between the two physical hosts that I can find are: > > - qemu on the CentOS 5 host builds without preadv support - would this > make such a huge performance difference? CentOS5 only comes with an > old version of glibc, which is missing preadv > - qemu on the CentOS 5 host builds without PIE > - libvirt 1.0 was required on the CentOS 5 host, since 0.10.2 had a > build bug. This shouldn't matter I don't think. > - I haven't tried rebuilding the VMs from scratch on the CentOS5 host, > which I guess is worth a try. > > The qemu process is being started with virtio + vhost: > > /usr/bin/qemu-system-x86_64 -name vmname -S -M pc-1.2 -enable-kvm -m > 4096 -smp 8,sockets=8,cores=1,threads=1 -uuid > 212915ed-a34a-4d6d-68f5-2216083a7693 -no-user-config -nodefaults > -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/vmname.monitor,server > ,nowai t -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=utc -no-shutdown -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=/mnt/vmname/disk.img,if=none,id=drive-virtio-disk0,format=raw,cac > he=non > e -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id > =virti > o-disk0,bootindex=1 -netdev tap,fd=16,id=hostnet0,vhost=on,vhostfd=18 > -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0 > ,addr= > 0x3 -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 > -vnc > 127.0.0.1:1 -vga cirrus -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > > The relevant part of my libvirt config, of which I've tried omitting > the target, alias and address elements with no difference in performance: > > > > > > > >function="0x0"/> > > > Is there something else which could be getting in the way here? > > Thanks! > > Ben Clay > rbc...@ncsu.edu > > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in the > body of a message to majord...@vger.kernel.org More majordomo info at > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
virtio + vhost-net performance issue - preadv ?
I have a working copy of libvirt 0.10.2 + qemu 1.2 installed on a vanilla up-to-date (2.6.32-279.9.1) CentOS 6 host, and get very good VM <-> VM network performance (both running on the same host) using virtio. I have cgroups set to cap the VMs at 10Gbps and iperf shows I'm getting exactly 10Gbps. I copied these VMs to a CentOS 5 host and installed libvirt 1.0 + qemu 1.2. However, the best performance I can get in between the VMs (again running on the same host) is ~2Gbps. In both cases, this is over a bridged interface with static IPs assigned to each VM. I've also tried virtual networking with NAT or routing, yielding the same results. I figured it was due to vhost-net missing on the older CentOS 5 kernel, so I installed 2.6.39-4.2 from ELRepo and got the /dev/vhost-net device and vhost processes associated with each VM: ]$ lsmod | grep vhost vhost_net 28446 2 tun23888 7 vhost_net ]$ ps aux | grep vhost- root 9628 0.0 0.0 0 0 ?S17:57 0:00 [vhost-9626] root 9671 0.0 0.0 0 0 ?S17:57 0:00 [vhost-9670] ]$ ls /dev/vhost-net -al crw--- 1 root root 10, 58 Nov 13 15:19 /dev/vhost-net After installing the new kernel, I also tried rebuilding libvirt and qemu, to no avail. I also disabled cgroups, just in case it was getting in the way, as well as iptables. I can see the virtio_net module loaded inside the guest, and using virtio raises my performance from <400Mbps to 2Gbps, so it does make some improvement. The only differences between the two physical hosts that I can find are: - qemu on the CentOS 5 host builds without preadv support - would this make such a huge performance difference? CentOS5 only comes with an old version of glibc, which is missing preadv - qemu on the CentOS 5 host builds without PIE - libvirt 1.0 was required on the CentOS 5 host, since 0.10.2 had a build bug. This shouldn't matter I don't think. - I haven't tried rebuilding the VMs from scratch on the CentOS5 host, which I guess is worth a try. The qemu process is being started with virtio + vhost: /usr/bin/qemu-system-x86_64 -name vmname -S -M pc-1.2 -enable-kvm -m 4096 -smp 8,sockets=8,cores=1,threads=1 -uuid 212915ed-a34a-4d6d-68f5-2216083a7693 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vmname.monitor,server,nowai t -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/vmname/disk.img,if=none,id=drive-virtio-disk0,format=raw,cache=non e -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virti o-disk0,bootindex=1 -netdev tap,fd=16,id=hostnet0,vhost=on,vhostfd=18 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0,addr= 0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 The relevant part of my libvirt config, of which I've tried omitting the target, alias and address elements with no difference in performance: Is there something else which could be getting in the way here? Thanks! Ben Clay rbc...@ncsu.edu -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: FW: cgroup blkio.weight working, but not for KVM guests
Stefan- Thanks for the hand-holding, it looks like the disk file is indeed open with O_DIRECT: [root@host ~]# grep ^flags: /proc/$PID/fdinfo/$FD flags: 02140002 Since this is not an issue, I guess another source of problems could be that all the virtio threads attached to this domain are not being placed within the cgroup. I will look through libvirt to see if they're setting the guest's process's cgroup classification as sticky (I can't imagine they wouldn't be), but this raises another question: are virtio kernel threads child processes of the guest's main process? Are you aware of any other factor which I should be considering here? I reran the dd tests inside the guest with iflag=fullblock set to make sure the guest buffer cache wasn't messing with throughput values (based on this: http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=5929322cc b1f9d27c1b07b746d37419d17a7cbf6), and got the same results listed earlier. Thanks again! Ben Clay rbc...@ncsu.edu -Original Message- From: Stefan Hajnoczi [mailto:stefa...@gmail.com] Sent: Tuesday, October 23, 2012 6:35 AM To: Ben Clay Cc: kvm@vger.kernel.org Subject: Re: FW: cgroup blkio.weight working, but not for KVM guests On Mon, Oct 22, 2012 at 07:36:34AM -0600, Ben Clay wrote: > Forwarding this to the KVM general list. I doubt you folks can help > me with libvirt, but I was wondering if theres some way to verify if > the cache=none parameter is being respected for my KVM guests disk > image, or if there are any other configuration/debug steps appropriate for KVM + virtio + cgroup. Here's how you can double-check the O_DIRECT flag: Find the QEMU process PID on the host: ps aux | grep qemu Then find the file descriptor of the image file which the QEMU process has open: ls -l /proc/$PID/fd Finally look at the file descriptor flags to confirm it is O_DIRECT: grep ^flags: /proc/$PID/fdinfo/$FD Note the flags field is in octal and you're looking for: #define O_DIRECT0004/* direct disk access hint */ Stefan -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
FW: cgroup blkio.weight working, but not for KVM guests
Forwarding this to the KVM general list. I doubt you folks can help me with libvirt, but I was wondering if theres some way to verify if the cache=none parameter is being respected for my KVM guests disk image, or if there are any other configuration/debug steps appropriate for KVM + virtio + cgroup. Thanks. Ben Clay rbc...@ncsu.edu From: Ben Clay [mailto:rbc...@ncsu.edu] Sent: Wednesday, October 17, 2012 11:31 AM To: libvirt-us...@redhat.com Subject: cgroup blkio.weight working, but not for KVM guests Im running libvirt 0.10.2 and qemu-kvm-1.2.0, both compiled from source, on CentOS 6. Ive got a working blkio cgroup hierarchy which Im attaching guests to using the following XML guest configs: VM1 (foreground): 2048 1000 VM2 (background): 2 100 Ive tested write throughput on the host using cgexec and dd, demonstrating that libvirt has correctly set up the cgroups: cgexec -g blkio:libvirt/qemu/foreground time dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096 & cgexec -g blkio:libvirt/qemu/background time dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096 & Snap from iotop, showing an 8:1 ratio (should be 10:1, but 8:1 is acceptable): Total DISK READ: 0.00 B/s | Total DISK WRITE: 91.52 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 9602 be/4 root 0.00 B/s 10.71 M/s 0.00 % 98.54 % dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096 9601 be/4 root 0.00 B/s 80.81 M/s 0.00 % 97.76 % dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096 Further, checking the task list inside each cgroup shows the guests main PID, plus those of the virtio kernel threads. Its hard to tell if all the virtio kernel threads are listed, but all the ones Ive hunted down appear to be there. However, when running the same dd commands inside the guests, I get roughly-equal performance nowhere near the ~8:1 relative bandwidth enforcement I get from the host: (background ctrl-cd right after foreground finishes, both started within 1s of each other) [ben@foreground ~]$ dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB) copied, 104.645 s, 41.0 MB/s [ben@background ~]$ dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096 ^C4052+0 records in 4052+0 records out 4248829952 bytes (4.2 GB) copied, 106.318 s, 40.0 MB/s I thought based on this statement: Currently, the Block I/O subsystem does not work for buffered write operations. It is primarily targeted at direct I/O, although it works for buffered read operations. from this page: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/ht ml/Resource_Management_Guide/ch-Subsystems_and_Tunable_Parameters.html that this problem might be due to host-side buffering, but I have that explicitly disabled in my guest configs: /usr/bin/qemu-kvm Here is the qemu line from ps, showing that its clearly being passed through from the guest XML config: root 5110 20.8 4.3 4491352 349312 ? Sl 11:58 0:38 /usr/bin/qemu-kvm -name background -S -M pc-1.2 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid ea632741-c7be-36ab-bd69-da3cbe505b38 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/background.monitor,server,n owait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/path/to/disk.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virti o-disk0,bootindex=1 -netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0,addr= 0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 For fun I tried a few different cache options to try to force a bypass the host buffercache, including writethough and directsync, but the number of virtio kernel threads appeared to explode (especially for directsync) and the throughput dropped quite low: ~50% of none for writethrough and ~5% for directsync. With cache=none, when I generate write loads inside the VMs, I do see growth in the hosts buffer cache. Further, if I use non-direct I/O inside the VMs, and inflate the balloon (forcing the guests buffer cache to flush), I dont see a corresponding drop in background throughput. Is it possible that the cache="none" directive is not being respected? Since cgroups is working for host-side processes I think my blkio subsystem is correctly set up (using cfq, group_isolation=1 etc). Maybe I miscompiled qemu, without some needed direct I/O support? Has anyone s
