Use default block size. Larger the better. As for pinpointing issues, try iostat -xnc and look at iowait. Another thing that can help is some dtrace in opt/dtt and look at your write sizes. Run atto disk bench in the vm if it's windows.
I would expect-a-guess a single sequential write around 200 and a read at about 300 mb/sec for that setup. Peel back the layers of KVM if needed to track the issue. On Jan 3, 2012, at 23:50, Russell Hansen <russ...@new-swankton.net> wrote: > I'm seeing absolutely terrible disk performance for my virtual machines > running under KVM. > > My ZFS pool is 6 Western Digital AV-25 500GB disks arranged in 3 mirrored > vdevs connected to an LSI 9210 (reflashed IBM M1015) in IT mode. > > My KVM startup script: > > /usr/bin/qemu-kvm \ > -enable-kvm \ > -smp 2 \ > -m 1024 \ > -no-hpet \ > -rtc base=localtime,driftfix=slew \ > -drive > file=/dev/zvol/dsk/oi_data/kvm/system/disk0,if=virtio,format=raw,cache=none,index=0 > \ > -drive > file=/dev/zvol/dsk/oi_data/kvm/system/disk1,if=virtio,format=raw,cache=none,index=1 > \ > -net nic,vlan=0,name=lan0,model=virtio,macaddr=$MAC0 \ > -net vnic,vlan=0,name=lan0,ifname=$ETH0 \ > -net nic,vlan=1,name=int0,model=virtio,macaddr=$MAC1 \ > -net vnic,vlan=1,name=int0,ifname=$ETH1 \ > -vnc $LAN_ADDR:$1 \ > -monitor telnet:127.0.0.1:444$1,server,nowait \ > -usbdevice tablet \ > -daemonize > > My zvol settings: > > NAME PROPERTY VALUE > SOURCE > oi_data/kvm/sachiel/disk1 type volume > - > oi_data/kvm/sachiel/disk1 creation Tue Nov 8 11:56 > 2011 - > oi_data/kvm/sachiel/disk1 used 298G > - > oi_data/kvm/sachiel/disk1 available 844G > - > oi_data/kvm/sachiel/disk1 referenced 139G > - > oi_data/kvm/sachiel/disk1 compressratio 1.00x > - > oi_data/kvm/sachiel/disk1 reservation none > default > oi_data/kvm/sachiel/disk1 volsize 144G > local > oi_data/kvm/sachiel/disk1 volblocksize 8K > - > oi_data/kvm/sachiel/disk1 checksum on > default > oi_data/kvm/sachiel/disk1 compression off > default > oi_data/kvm/sachiel/disk1 readonly off > default > oi_data/kvm/sachiel/disk1 copies 1 > default > oi_data/kvm/sachiel/disk1 refreservation 149G > local > oi_data/kvm/sachiel/disk1 primarycache all > default > oi_data/kvm/sachiel/disk1 secondarycache all > default > oi_data/kvm/sachiel/disk1 usedbysnapshots 11.8G > - > oi_data/kvm/sachiel/disk1 usedbydataset 139G > - > oi_data/kvm/sachiel/disk1 usedbychildren 0 > - > oi_data/kvm/sachiel/disk1 usedbyrefreservation 147G > - > oi_data/kvm/sachiel/disk1 logbias latency > default > oi_data/kvm/sachiel/disk1 dedup off > default > oi_data/kvm/sachiel/disk1 mlslabel none > default > oi_data/kvm/sachiel/disk1 sync standard > default > oi_data/kvm/sachiel/disk1 refcompressratio 1.00x > - > > Representative iostat output while moving some 4GB of digital photos (approx > 2MB each): > > capacity operations bandwidth > pool alloc free read write read write > ------------------------- ----- ----- ----- ----- ----- ----- > oi_data 373G 1019G 71 377 569K 3.98M > mirror 124G 340G 15 127 125K 1.34M > c2t50014EE2AFBDC3CBd0 - - 2 71 118K 1.34M > c2t50014EE25A683C10d0 - - 0 69 7.01K 1.34M > mirror 124G 340G 31 123 256K 1.33M > c2t50014EE20518D4F6d0 - - 4 49 132K 1.33M > c2t50014EE2AFBD87B4d0 - - 4 48 123K 1.33M > mirror 124G 340G 23 125 189K 1.32M > c2t50014EE205154E0Bd0 - - 3 52 35.1K 1.32M > c2t50014EE25A6829D6d0 - - 4 51 156K 1.32M > ------------------------- ----- ----- ----- ----- ----- ----- > > A Bonnie result from a napp-it install: > > Version 1.03c ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > %CP > 2011.12.12 32G 61586 99 151550 16 95819 17 49188 99 273833 18 1166 > 3 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 16 32070 99 +++++ +++ +++++ +++ 32736 100 +++++ +++ +++++ +++ > 2011.12.12,32G,61586,99,151550,16,95819,17,49188,99,273833,18,1166.1,3,16,32070,99,+++++,+++,+++++,+++,32736,100,+++++,+++,+++++,+++ > > > As you can see, it basically hovers around 4-6 MB/s average with an > approximate 10KB I/O size so it appears I'm 100% iops limited. Do I need to > recreate my zvols with a larger volblocksize to get decent performance or is > it something else I need to do on the KVM config side or within the virtual > machine? > > I have one Windows Server 2003 (32-bit) VM and 2x Windows Server 2008 R2 > (64-bit) VMs and they all behave the same way when it comes to disk perf. > > Thanks, > > -Russ > _______________________________________________ > OpenIndiana-discuss mailing list > OpenIndiana-discuss@openindiana.org > http://openindiana.org/mailman/listinfo/openindiana-discuss _______________________________________________ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
