Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
Hi Michael, so your tests wer now exectued on a bloody host ? indicating that the performance went back up in bloody ? cheers tobi Today Michael Mounteney wrote: > Sorry to take so long to get back to you Tobias and I hope this is > still relevant. As described elsewhere in this list, I had temporarily > to downgrade ssh to achieve interoperability between the OmniOS (bloody) > host and the Gentoo Linux guests. > > First, ssh imposes some overhead: > > mounty@pantry ~ $ time ssh people exit > > real 0m0.724s > user 0m0.032s > sys 0m0.012s > > that real figure averages around the 0.750s mark. So I decided to > perform much bigger transfers to minimise its effect: > > mounty@pantry ~ $ dd if=/dev/zero bs=1M count=2000 | ssh people dd > of=/dev/null > 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.1 GB) copied, 138.436 s, 15.1 MB/s > 4096000+0 records in > 4096000+0 records out > 2097152000 bytes transferred in 137.657582 secs (15234555 bytes/sec) > > mounty@pantry ~ $ ssh people dd if=/dev/zero bs=1M count=2000 | dd > of=/dev/null > 2000+0 records in > 2000+0 records out > 2097152000 bytes transferred in 51.692313 secs (40569901 bytes/sec) > 4096000+0 records in > 4096000+0 records out > 2097152000 bytes (2.1 GB) copied, 52.4503 s, 40.0 MB/s > > It is puzzling that the in and out figures are so different but I did > perform each test three times and the results were approximately the > same each time. On the read-off-disk test, here are all three runs: > > pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB) copied, 65.3406 s, 16.0 MB/s > pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB) copied, 1.19789 s, 875 MB/s > pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB) copied, 1.85877 s, 564 MB/s > > which I've quoted to show that the disk must be cached. So I tried > again with more data to eliminate that effect: > > pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=10240 > 10240+0 records in > 10240+0 records out > 10737418240 bytes (11 GB) copied, 710.215 s, 15.1 MB/s > > I hope that's helpful. > > Michael. > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
Sorry to take so long to get back to you Tobias and I hope this is still relevant. As described elsewhere in this list, I had temporarily to downgrade ssh to achieve interoperability between the OmniOS (bloody) host and the Gentoo Linux guests. First, ssh imposes some overhead: mounty@pantry ~ $ time ssh people exit real0m0.724s user0m0.032s sys 0m0.012s that real figure averages around the 0.750s mark. So I decided to perform much bigger transfers to minimise its effect: mounty@pantry ~ $ dd if=/dev/zero bs=1M count=2000 | ssh people dd of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 138.436 s, 15.1 MB/s 4096000+0 records in 4096000+0 records out 2097152000 bytes transferred in 137.657582 secs (15234555 bytes/sec) mounty@pantry ~ $ ssh people dd if=/dev/zero bs=1M count=2000 | dd of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 51.692313 secs (40569901 bytes/sec) 4096000+0 records in 4096000+0 records out 2097152000 bytes (2.1 GB) copied, 52.4503 s, 40.0 MB/s It is puzzling that the in and out figures are so different but I did perform each test three times and the results were approximately the same each time. On the read-off-disk test, here are all three runs: pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 65.3406 s, 16.0 MB/s pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 1.19789 s, 875 MB/s pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 1.85877 s, 564 MB/s which I've quoted to show that the disk must be cached. So I tried again with more data to eliminate that effect: pantry ~ # dd if=/dev/vda of=/dev/zero bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 710.215 s, 15.1 MB/s I hope that's helpful. Michael. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
Hi Michael,
Today Michael Mounteney wrote:
> On Wed, 10 Dec 2014 06:14:56 +0100 (CET)
> Tobias Oetiker wrote:
>
> > This leads me to suspect, that either only very few people are
> > using omnios as a kvm server OR it is also a hardware dependent
> > problem.
>
> I think it must be. I'm running KVM (Gentoo Linux guests) and have
> just gone from 151010 to 151012. I haven't carried-out any
> quantitative assessment, but didn't notice any slowdown. For the
> record, my KVM invocation is:
>
> /usr/bin/qemu-system-x86_64 \
> -name "Gentoo "$WHAT \
> -cpu host \
> -boot order=d \
> -enable-kvm \
> -vnc cortex:$VNC \
> -smp 1,maxcpus=1,cores=2 \
> -m 1024 \
> -no-hpet \
> -localtime \
> -kernel
> /gentoo/kernel-source/linux-3.17.4-gentoo-vbox/arch/x86/boot/bzImage \
> -append "root=/dev/vda1 init=/usr/lib/systemd/systemd quiet" \
> -drive
> file=/dev/zvol/dsk/rpool/vol/Gentoo-KVM-${WHAT},cache=none,if=virtio,index=0 \
> -drive
> file=/dev/zvol/dsk/rpool/vol/Linuswap-${WHAT},cache=none,if=virtio,index=1 \
> -net nic,vlan=0,macaddr=$mac,model=virtio,name=ncard1 \
> -net vnic,vlan=0,name=net0,ifname=$VNIC,macaddr=$mac \
> -monitor telnet:127.0.0.1:${monitor},server,nowait \
> -vga std \
> -daemonize
>
> where ${WHAT} is either KDE or XFCE. Machine is a Supermicro 5017C-LF with 1
> x Intel Xeon E3-1240V2 3.40 GHz.4 Cores , 4 Threads,8Mb cache and 8 GiB RAM.
>
> If you are interested in any performance figures, let me know any tar or dd
> etc. commands you'd like me to run.
yes, a very simple test:
guest$ dd if=/dev/zero bs=1M count=20 | ssh host dd of=/dev/null
guest$ ssh host dd if=/dev/zero bs=1M count=20 | dd of=/dev/null
and since I suspect that the disk io suffers too but due to
caching, maybe reading just a bit off the disk device might be an
interesting test:
dd if=/dev/vda of=/dev/zero bs=1M count=1000
cheers
tobi
> Michael.
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch t...@oetiker.ch +41 62 775 9902
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
On Wed, 10 Dec 2014 06:14:56 +0100 (CET)
Tobias Oetiker wrote:
> This leads me to suspect, that either only very few people are
> using omnios as a kvm server OR it is also a hardware dependent
> problem.
I think it must be. I'm running KVM (Gentoo Linux guests) and have
just gone from 151010 to 151012. I haven't carried-out any
quantitative assessment, but didn't notice any slowdown. For the
record, my KVM invocation is:
/usr/bin/qemu-system-x86_64 \
-name "Gentoo "$WHAT \
-cpu host \
-boot order=d \
-enable-kvm \
-vnc cortex:$VNC \
-smp 1,maxcpus=1,cores=2 \
-m 1024 \
-no-hpet \
-localtime \
-kernel
/gentoo/kernel-source/linux-3.17.4-gentoo-vbox/arch/x86/boot/bzImage \
-append "root=/dev/vda1 init=/usr/lib/systemd/systemd quiet" \
-drive
file=/dev/zvol/dsk/rpool/vol/Gentoo-KVM-${WHAT},cache=none,if=virtio,index=0 \
-drive
file=/dev/zvol/dsk/rpool/vol/Linuswap-${WHAT},cache=none,if=virtio,index=1 \
-net nic,vlan=0,macaddr=$mac,model=virtio,name=ncard1 \
-net vnic,vlan=0,name=net0,ifname=$VNIC,macaddr=$mac \
-monitor telnet:127.0.0.1:${monitor},server,nowait \
-vga std \
-daemonize
where ${WHAT} is either KDE or XFCE. Machine is a Supermicro 5017C-LF with 1 x
Intel Xeon E3-1240V2 3.40 GHz.4 Cores , 4 Threads,8Mb cache and 8 GiB RAM.
If you are interested in any performance figures, let me know any tar or dd
etc. commands you'd like me to run.
Michael.
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
Yesterday Dan McDonald wrote: > > > > I have not tested disk performance explicitly, but even booting a > > windows host took ages ... so I suspect whatever is causing this > > influences all kvm guest IO. > > What's really REALLY weird about this is that we did not alter > anything about how we built KVM between these releases. > > Tell me, can you run "lockstat sleep " in the > global zone while you run your KVM tests? They will produce a > lot of output, but they may be very informative about what's > going on. > Unfortunately I don't have a spare 'big' iron box to play with, but we should be able to do some downtime tonight to run that lockstat sleep experiment and also to do some simple disk io test (with dd). > Also, I'd be curious if you might (BEs and rpool space being > available) upgrade a BE to bloody and repeat your tests? > We don't have the facilities to stress out VMs like this, which > is why we didn't notice this before 012 went out the door. > Clearly something's messing up KVM performance (you're not the > first to report this, but you seem to have a decent environment > for comparisons). Before the next stable (and incidentally > long-term-support as well) release, I hope to have these problems > cleared up. One thing that should happen soon is that Joyent is > upstreaming the VND changes into illumos-gate, which will allow > us to be fully caught up to their illumos-kvm-cmd source, which > we've frozen at revision 1c6181be55d1cadc4426069960688307a6083131 > since r151010. I know that there was no kvm change ... so this must be some side effect of another modificiation ... What seems odd, is that only 2 (or 3) few people reported this problem on the list. After all, it's not something that was difficult to notice. After the upgrade the kvm guests really are almost un-usable for interactive work involing network or disk IO, especially when compared to before. This leads me to suspect, that either only very few people are using omnios as a kvm server OR it is also a hardware dependent problem. I was also wondering if we should try to boot the current smaros on the box just to see what it does to kvm perf. But as I said, it is a production machine, so it is all a bit tricky. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
> > I have not tested disk performance explicitly, but even booting a > windows host took ages ... so I suspect whatever is causing this > influences all kvm guest IO. What's really REALLY weird about this is that we did not alter anything about how we built KVM between these releases. Tell me, can you run "lockstat sleep " in the global zone while you run your KVM tests? They will produce a lot of output, but they may be very informative about what's going on. Also, I'd be curious if you might (BEs and rpool space being available) upgrade a BE to bloody and repeat your tests? We don't have the facilities to stress out VMs like this, which is why we didn't notice this before 012 went out the door. Clearly something's messing up KVM performance (you're not the first to report this, but you seem to have a decent environment for comparisons). Before the next stable (and incidentally long-term-support as well) release, I hope to have these problems cleared up. One thing that should happen soon is that Joyent is upstreaming the VND changes into illumos-gate, which will allow us to be fully caught up to their illumos-kvm-cmd source, which we've frozen at revision 1c6181be55d1cadc4426069960688307a6083131 since r151010. Thanks, and I wish I could be of more immediate assistance! Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade
So tonight, we finally took the plunge and upgraded our zfs/kvm server to r151012 ... the results were terrible. The kvm booted very slowly and all networking felt really slow ... so I did a little test: ubutu-14.04-guest$ dd if=/dev/zero bs=1M count=20 | ssh omnios-r151012-host dd of=/dev/null 20971520 bytes (21 MB) copied, 6.27333 s, 3.3 MB/s ubutu-14.04-guest$ ssh omnios-r151012-host dd if=/dev/zero bs=1M count=20 | dd of=/dev/null 20971520 bytes transferred in 8.010208 secs (2618099 bytes/sec) These numbers were obtained using virtio net drivers but switching to e1000 did not significantly change things. So we booted back into r151010 again ... the difference was immediately apparent ... but there are also number to back this up. ubutu-14.04-guest$ dd if=/dev/zero bs=1M count=20 | ssh omnios-r151010-host dd of=/dev/null 20971520 bytes (21 MB) copied, 0.812479 s, 25.8 MB/s ubutu-14.04-guest$ ssh omnios-r151010-host dd if=/dev/zero bs=1M count=20 | dd of=/dev/null 20971520 bytes (21 MB) copied, 0.545423 s, 38.5 MB/s as you can see the difference in guest network performance is roughly one magnitude ... I have not tested disk performance explicitly, but even booting a windows host took ages ... so I suspect whatever is causing this influences all kvm guest IO. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
