Re: virtio disk slower than IDE?
[prior attempts from elsewhere kept bouncing, apologies for any replication] Gordan Bobic wrote: The test is building the Linux kernel (only taking the second run to give the test the benefit of local cache): make clean; make -j8 all; make clean; sync; time make -j8 all This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I ran the tests multiple time with most non-essential services on the host switched off (including cron/atd), and the guest in single-user mode to reduce the noise in the test to the minimum, and the results are pretty consistent, with virtio being about 30% behind. I'd expect for an observed 30% wall clock time difference of an operation as complex as a kernel build the base i/o throughput disparity is substantially greater. Did you try a more simple/regular load, eg: a streaming dd read of various block sizes from guest raw disk devices? This is also considerably easier to debug vs. the complex i/o load generated by a build. One way to chop up the problem space is using blktrace on the host to observe both the i/o patterns coming out of qemu and the host's response to them in terms of turn around time. I expect you'll see somewhat different nature requests generated by qemu w/r/t blocking and number of threads serving virtio_blk requests relative to ide but the host response should be essentially the same in terms of data returned per unit time. If the host looks to be turning around i/o request with similar latency in both cases, the problem would be lower frequency of requests generated by qemu in the case of virtio_blk. Here it would be useful to know the host load generated by the guest for both cases. -john -- john.coo...@redhat.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
Gordan Bobic wrote: The test is building the Linux kernel (only taking the second run to give the test the benefit of local cache): make clean; make -j8 all; make clean; sync; time make -j8 all This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I ran the tests multiple time with most non-essential services on the host switched off (including cron/atd), and the guest in single-user mode to reduce the noise in the test to the minimum, and the results are pretty consistent, with virtio being about 30% behind. I'd expect for an observed 30% wall clock time difference of an operation as complex as a kernel build the base i/o throughput disparity is substantially greater. Did you try a more simple/regular load, eg: a streaming dd read of various block sizes from guest raw disk devices? This is also considerably easier to debug vs. the complex i/o load generated by a build. One way to chop up the problem space is using blktrace on the host to observe both the i/o patterns coming out of qemu and the host's response to them in terms of turn around time. I expect you'll see somewhat different nature requests generated by qemu w/r/t blocking and number of threads serving virtio_blk requests relative to ide but the host response should be essentially the same in terms of data returned per unit time. If the host looks to be turning around i/o request with similar latency in both cases, the problem would be lower frequency of requests generated by qemu in the case of virtio_blk. Here it would be useful to know the host load generated by the guest for both cases. -john -- john.coo...@third-harmonic.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
Gordan Bobic wrote: Lastly, do you use cache=wb on qemu? it's just a fun mode, we use cache=off only. I don't see the option being set in the logs, so I'd guess it's whatever qemu-kvm defaults to. You can set this through libvirt by putting an element such as the following within your disk element: driver name='qemu' type='qcow2' cache='none'/ (Setting the type is preferred to avoid security issues wherein a guest writes an arbitrary qcow2 header to the beginning of a raw disk, reboots and allows qemu's autodetection to decide that this formerly-raw disk should now be treated as a delta against a file they otherwise might not have access to read; as such, it's particularly important if you intend that the type be raw). -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
On 11/16/2009 08:11 PM, Charles Duffy wrote: Gordan Bobic wrote: Lastly, do you use cache=wb on qemu? it's just a fun mode, we use cache=off only. I don't see the option being set in the logs, so I'd guess it's whatever qemu-kvm defaults to. You can set this through libvirt by putting an element such as the following within your disk element: driver name='qemu' type='qcow2' cache='none'/ It's not needed on rhel5.4 qemu - we have cache=none as a default (Setting the type is preferred to avoid security issues wherein a guest writes an arbitrary qcow2 header to the beginning of a raw disk, reboots and allows qemu's autodetection to decide that this formerly-raw disk should now be treated as a delta against a file they otherwise might not have access to read; as such, it's particularly important if you intend that the type be raw). -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
john cooper wrote:
The test is building the Linux kernel (only taking the second run to give the
test the benefit of local cache):
make clean; make -j8 all; make clean; sync; time make -j8 all
This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I
ran the tests multiple time with most non-essential services on the host switched off
(including cron/atd), and the guest in single-user mode to reduce the noise
in the test to the minimum, and the results are pretty consistent, with virtio being
about 30% behind.
I'd expect for an observed 30% wall clock time difference
of an operation as complex as a kernel build the base i/o
throughput disparity is substantially greater. Did you
try a more simple/regular load, eg: a streaming dd read
of various block sizes from guest raw disk devices?
This is also considerably easier to debug vs. the complex
i/o load generated by a build.
I'm not convinced it's the read performance, since it's the second pass
that is time, by which time all the source files will be in the guest's
cache. I verified this by doing just one pass and priming it with:
find . -type f -exec cat '{}' /dev/null \;
The execution times are indistinguishable from the second pass in the
two-pass test.
To me that would indicate the the problem is with write performance,
rather than read performance.
One way to chop up the problem space is using blktrace
on the host to observe both the i/o patterns coming out
of qemu and the host's response to them in terms of
turn around time. I expect you'll see somewhat different
nature requests generated by qemu w/r/t blocking and
number of threads serving virtio_blk requests relative
to ide but the host response should be essentially the
same in terms of data returned per unit time.
If the host looks to be turning around i/o request with
similar latency in both cases, the problem would be lower
frequency of requests generated by qemu in the case of
virtio_blk. Here it would be useful to know the host
load generated by the guest for both cases.
With virtio the CPU usage did seem to be noticeably lower. I figured
that was because it was spending more time waiting for I/O to finish,
since it was clearly bottlenecking on disk I/O (since that's the only
thing that changed).
I'll try iozone's write tests and see how that compares. If I'm right
about write performance being problematic, iozone might show the same
performance deterioration on write tests compared to the IDE emulation.
Gordan
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
On 11/14/2009 04:23 PM, Gordan Bobic wrote: I just tried paravirtualized virtio block devices, and my tests show that they are approximately 30% slower than emulated IDE devices. I'm guessing this isn't normal. Is this a known issue or am I likely to have mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and guest). Please try to change the io scheduler on the host to io scheduler, it should boost your performance back. Thanks. Gordan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
Dor Laor wrote: On 11/14/2009 04:23 PM, Gordan Bobic wrote: I just tried paravirtualized virtio block devices, and my tests show that they are approximately 30% slower than emulated IDE devices. I'm guessing this isn't normal. Is this a known issue or am I likely to have mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and guest). Please try to change the io scheduler on the host to io scheduler, it should boost your performance back. I presume you mean the deadline io scheduler. I tried that (kernel parameter elevator=deadline) and it made no measurable difference compared to the cfq scheduler. Gordan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
On 11/15/2009 02:00 PM, Gordan Bobic wrote: Dor Laor wrote: On 11/14/2009 04:23 PM, Gordan Bobic wrote: I just tried paravirtualized virtio block devices, and my tests show that they are approximately 30% slower than emulated IDE devices. I'm guessing this isn't normal. Is this a known issue or am I likely to have mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and guest). Please try to change the io scheduler on the host to io scheduler, it should boost your performance back. I presume you mean the deadline io scheduler. I tried that (kernel parameter elevator=deadline) and it made no measurable difference compared to the cfq scheduler. What version of kvm do you use? Is it rhel5.4? Can you post the qemu cmdline and the perf test in the guest? Lastly, do you use cache=wb on qemu? it's just a fun mode, we use cache=off only. Gordan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: virtio disk slower than IDE?
Dor Laor wrote: On 11/15/2009 02:00 PM, Gordan Bobic wrote: Dor Laor wrote: On 11/14/2009 04:23 PM, Gordan Bobic wrote: I just tried paravirtualized virtio block devices, and my tests show that they are approximately 30% slower than emulated IDE devices. I'm guessing this isn't normal. Is this a known issue or am I likely to have mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and guest). Please try to change the io scheduler on the host to io scheduler, it should boost your performance back. I presume you mean the deadline io scheduler. I tried that (kernel parameter elevator=deadline) and it made no measurable difference compared to the cfq scheduler. What version of kvm do you use? Is it rhel5.4? It's RHEL 5.4. $ rpm -qa | grep -i kvm kmod-kvm-83-105.el5_4.9 kvm-83-105.el5_4.9 Can you post the qemu cmdline and the perf test in the guest? Here is what is in the libvirt log: For IDE emulation: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 2048 -smp 4 -name RHEL_5_x86-64 -uuid cb44b2c5-e64b-848f-77af-f8e7f02fa2ca -no-kvm-pit-reinjection -monitor pty -pidfile /var/run/libvirt/qemu//RHEL_5_x86-64.pid -boot c -drive file=/var/lib/libvirt/images/RHEL_5_x86-64.img,if=ide,index=0,boot=on -net nic,macaddr=54:52:00:5a:67:4b,vlan=0,model=e1000 -net tap,fd=15,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-gb For virtio: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 2048 -smp 4 -name RHEL_5_x86-64 -uuid cb44b2c5-e64b-848f-77af-f8e7f02fa2ca -no-kvm-pit-reinjection -monitor pty -pidfile /var/run/libvirt/qemu//RHEL_5_x86-64.pid -boot c -drive file=/var/lib/libvirt/images/CentOS_5_x86-64.img,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:5a:67:4b,vlan=0,model=e1000 -net tap,fd=15,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-gb The test is building the Linux kernel (only taking the second run to give the test the benefit of local cache): make clean; make -j8 all; make clean; sync; time make -j8 all This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I ran the tests multiple time with most non-essential services on the host switched off (including cron/atd), and the guest in single-user mode to reduce the noise in the test to the minimum, and the results are pretty consistent, with virtio being about 30% behind. Lastly, do you use cache=wb on qemu? it's just a fun mode, we use cache=off only. I don't see the option being set in the logs, so I'd guess it's whatever qemu-kvm defaults to. Gordan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
virtio disk slower than IDE?
I just tried paravirtualized virtio block devices, and my tests show that they are approximately 30% slower than emulated IDE devices. I'm guessing this isn't normal. Is this a known issue or am I likely to have mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and guest). Thanks. Gordan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
