I'm seeing absolutely terrible disk performance for my virtual machines running under KVM.
My ZFS pool is 6 Western Digital AV-25 500GB disks arranged in 3 mirrored vdevs connected to an LSI 9210 (reflashed IBM M1015) in IT mode. My KVM startup script: /usr/bin/qemu-kvm \ -enable-kvm \ -smp 2 \ -m 1024 \ -no-hpet \ -rtc base=localtime,driftfix=slew \ -drive file=/dev/zvol/dsk/oi_data/kvm/system/disk0,if=virtio,format=raw,cache=none,index=0 \ -drive file=/dev/zvol/dsk/oi_data/kvm/system/disk1,if=virtio,format=raw,cache=none,index=1 \ -net nic,vlan=0,name=lan0,model=virtio,macaddr=$MAC0 \ -net vnic,vlan=0,name=lan0,ifname=$ETH0 \ -net nic,vlan=1,name=int0,model=virtio,macaddr=$MAC1 \ -net vnic,vlan=1,name=int0,ifname=$ETH1 \ -vnc $LAN_ADDR:$1 \ -monitor telnet:127.0.0.1:444$1,server,nowait \ -usbdevice tablet \ -daemonize My zvol settings: NAME PROPERTY VALUE SOURCE oi_data/kvm/sachiel/disk1 type volume - oi_data/kvm/sachiel/disk1 creation Tue Nov 8 11:56 2011 - oi_data/kvm/sachiel/disk1 used 298G - oi_data/kvm/sachiel/disk1 available 844G - oi_data/kvm/sachiel/disk1 referenced 139G - oi_data/kvm/sachiel/disk1 compressratio 1.00x - oi_data/kvm/sachiel/disk1 reservation none default oi_data/kvm/sachiel/disk1 volsize 144G local oi_data/kvm/sachiel/disk1 volblocksize 8K - oi_data/kvm/sachiel/disk1 checksum on default oi_data/kvm/sachiel/disk1 compression off default oi_data/kvm/sachiel/disk1 readonly off default oi_data/kvm/sachiel/disk1 copies 1 default oi_data/kvm/sachiel/disk1 refreservation 149G local oi_data/kvm/sachiel/disk1 primarycache all default oi_data/kvm/sachiel/disk1 secondarycache all default oi_data/kvm/sachiel/disk1 usedbysnapshots 11.8G - oi_data/kvm/sachiel/disk1 usedbydataset 139G - oi_data/kvm/sachiel/disk1 usedbychildren 0 - oi_data/kvm/sachiel/disk1 usedbyrefreservation 147G - oi_data/kvm/sachiel/disk1 logbias latency default oi_data/kvm/sachiel/disk1 dedup off default oi_data/kvm/sachiel/disk1 mlslabel none default oi_data/kvm/sachiel/disk1 sync standard default oi_data/kvm/sachiel/disk1 refcompressratio 1.00x - Representative iostat output while moving some 4GB of digital photos (approx 2MB each): capacity operations bandwidth pool alloc free read write read write ------------------------- ----- ----- ----- ----- ----- ----- oi_data 373G 1019G 71 377 569K 3.98M mirror 124G 340G 15 127 125K 1.34M c2t50014EE2AFBDC3CBd0 - - 2 71 118K 1.34M c2t50014EE25A683C10d0 - - 0 69 7.01K 1.34M mirror 124G 340G 31 123 256K 1.33M c2t50014EE20518D4F6d0 - - 4 49 132K 1.33M c2t50014EE2AFBD87B4d0 - - 4 48 123K 1.33M mirror 124G 340G 23 125 189K 1.32M c2t50014EE205154E0Bd0 - - 3 52 35.1K 1.32M c2t50014EE25A6829D6d0 - - 4 51 156K 1.32M ------------------------- ----- ----- ----- ----- ----- ----- A Bonnie result from a napp-it install: Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP 2011.12.12 32G 61586 99 151550 16 95819 17 49188 99 273833 18 1166 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 32070 99 +++++ +++ +++++ +++ 32736 100 +++++ +++ +++++ +++ 2011.12.12,32G,61586,99,151550,16,95819,17,49188,99,273833,18,1166.1,3,16,32070,99,+++++,+++,+++++,+++,32736,100,+++++,+++,+++++,+++ As you can see, it basically hovers around 4-6 MB/s average with an approximate 10KB I/O size so it appears I'm 100% iops limited. Do I need to recreate my zvols with a larger volblocksize to get decent performance or is it something else I need to do on the KVM config side or within the virtual machine? I have one Windows Server 2003 (32-bit) VM and 2x Windows Server 2008 R2 (64-bit) VMs and they all behave the same way when it comes to disk perf. Thanks, -Russ _______________________________________________ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss