to compare fio results from host system with file places on the ssd (ext4,nodiscard, trimmed before run): randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32 2.0.8 Starting 10 processes Jobs: 10 (f=10): [mmmmmmmmmm] [100.0% done] [50892K/49944K /s] [12.8K/12.5K iops] [eta 00m:00s] randrw: (groupid=0, jobs=10): err= 0: pid=7439 read : io=739852KB, bw=18491KB/s, iops=4622 , runt= 40012msec slat (usec): min=2 , max=12573 , avg=308.09, stdev=955.59 clat (usec): min=137 , max=107321 , avg=12750.90, stdev=7865.72 lat (usec): min=167 , max=107364 , avg=13059.31, stdev=7932.79 clat percentiles (usec): | 1.00th=[ 2512], 5.00th=[ 4192], 10.00th=[ 5280], 20.00th=[ 6816], | 30.00th=[ 8096], 40.00th=[ 9408], 50.00th=[10816], 60.00th=[12480], | 70.00th=[14528], 80.00th=[17536], 90.00th=[22656], 95.00th=[28032], | 99.00th=[40192], 99.50th=[45824], 99.90th=[59648], 99.95th=[68096], | 99.99th=[94720] bw (KB/s) : min= 0, max= 6679, per=27.96%, avg=5170.37, stdev=665.70 write: io=737492KB, bw=18432KB/s, iops=4607 , runt= 40012msec slat (usec): min=4 , max=12670 , avg=305.68, stdev=945.24 clat (usec): min=31 , max=107147 , avg=11268.07, stdev=7542.08 lat (usec): min=72 , max=107181 , avg=11574.10, stdev=7615.99 clat percentiles (usec): | 1.00th=[ 1624], 5.00th=[ 3216], 10.00th=[ 4256], 20.00th=[ 5664], | 30.00th=[ 6816], 40.00th=[ 8032], 50.00th=[ 9408], 60.00th=[10944], | 70.00th=[12864], 80.00th=[15552], 90.00th=[20608], 95.00th=[26240], | 99.00th=[38144], 99.50th=[43264], 99.90th=[55552], 99.95th=[63744], | 99.99th=[89600] bw (KB/s) : min= 0, max= 6767, per=27.39%, avg=5047.81, stdev=992.62 lat (usec) : 50=0.01%, 100=0.01%, 250=0.06%, 500=0.12%, 750=0.06% lat (usec) : 1000=0.06% lat (msec) : 2=0.74%, 4=5.47%, 10=42.71%, 20=38.26%, 50=12.26% lat (msec) : 100=0.25%, 250=0.01% cpu : usr=0.27%, sys=67.69%, ctx=339142, majf=0, minf=70 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued : total=r=184963/w=184373/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs): READ: io=739852KB, aggrb=18490KB/s, minb=18490KB/s, maxb=18490KB/s, mint=40012msec, maxt=40012msec WRITE: io=737492KB, aggrb=18431KB/s, minb=18431KB/s, maxb=18431KB/s, mint=40012msec, maxt=40012msec Disk stats (read/write): dm-5: ios=183885/183290, merge=0/0, ticks=1070388/977428, in_queue=2053044, util=35.35%, aggrios=184963/184383, aggrmerge=0/3, aggrticks=1072368/978452, aggrin_queue=2052424, aggrutil=35.39% sdd: ios=184963/184383, merge=0/3, ticks=1072368/978452, in_queue=2052424, util=35.39% 2015-08-02 14:52 GMT+03:00 Vasiliy Tolstov <v.tols...@selfip.ru>: > 2015-07-31 15:08 GMT+03:00 Vasiliy Tolstov <v.tols...@selfip.ru>: >> Please wait to performance comparison. As i see Liu's patch may be >> more slow then Hitoshi. > > > I'm switch to local cluster driver to test only local ssd and not > network overhead. > > But now Liu's qemu does not able to completely install vps: > sheep runs as: > /usr/sbin/sheep --log level=debug format=server dir=/var/log/ > dst=default --bindaddr 0.0.0.0 --cluster local --directio --upgrade > --myaddr 192.168.240.134 --pidfile /var/run/sheepdog.pid > /var/lib/sheepdog/sd_meta > /var/lib/sheepdog/vg0-sd_data,/var/lib/sheepdog/vg1-sd_data,/var/lib/sheepdog/vg2-sd_data,/var/lib/sheepdog/vg3-sd_data,/var/lib/sheepdog/vg4-sd_data,/var/lib/sheepdog/vg5-sd_data > > qemu runs as: > qemu-system-x86_64 -enable-kvm -name 30681 -S -machine > pc-i440fx-1.7,accel=kvm,usb=off -m 1024 -realtime > mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid > 49771611-5e45-70c9-1a78-00000043bf91 -no-user-config -nodefaults > -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/30681.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown -boot strict=on -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-scsi-pci,id=scsi0,num_queues=1,bus=pci.0,addr=0x4 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive > file=sheepdog:30681,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,discard=ignore,aio=native > -device > scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 > -drive if=none,id=drive-scsi0-0-1-0,readonly=on,format=raw -device > scsi-cd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 > -netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=39 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:42:8d,bus=pci.0,addr=0x3,rombar=0 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -chardev > socket,id=charchannel0,path=/var/lib/libvirt/qemu/30681.agent,server,nowait > -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 > -device usb-mouse,id=input0 -device usb-kbd,id=input1 -vnc > [::]:0,password -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object > rng-random,id=objrng0,filename=/dev/random -device > virtio-rng-pci,rng=objrng0,id=rng0,max-bytes=1024,period=2000,bus=pci.0,addr=0x7 > -msg timestamp=on > > sheep.log: > https://gist.github.com/raw/31201454b0afe1b86d00 > > dmesg: > [Sun Aug 2 14:48:45 2015] qemu-system-x86[6894]: segfault at 401364 > ip 00007f212f060ff8 sp 00007f213efe5f40 error 4 in > qemu-system-x86_64[7f212ecbc000+518000] > > fio job file: > [randrw] > blocksize=4k > filename=/dev/sdb > rw=randrw > direct=1 > buffered=0 > ioengine=libaio > iodepth=32 > group_reporting > numjobs=10 > runtime=40 > > For clean test i'm create new vdi and attach it to running vm: qemu > cache=none, io=native, discard=ignore > Hitoshi patch: > randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 > ... > fio-2.1.11 > Starting 10 processes > Jobs: 10 (f=10): [m(10)] [100.0% done] [13414KB/12291KB/0KB /s] > [3353/3072/0 iops] [eta 00m:00s] > randrw: (groupid=0, jobs=10): err= 0: pid=1318: Sun Aug 2 11:51:25 2015 > read : io=368004KB, bw=9191.9KB/s, iops=2297, runt= 40036msec > slat (usec): min=1, max=408908, avg=1316.35, stdev=10057.52 > clat (usec): min=229, max=972302, avg=67203.98, stdev=65126.08 > lat (usec): min=294, max=972306, avg=68520.61, stdev=66741.02 > clat percentiles (msec): > | 1.00th=[ 4], 5.00th=[ 16], 10.00th=[ 30], 20.00th=[ 36], > | 30.00th=[ 40], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 52], > | 70.00th=[ 59], 80.00th=[ 85], 90.00th=[ 128], 95.00th=[ 186], > | 99.00th=[ 367], 99.50th=[ 441], 99.90th=[ 570], 99.95th=[ 619], > | 99.99th=[ 799] > bw (KB /s): min= 6, max= 1824, per=10.19%, avg=936.59, stdev=434.25 > write: io=366964KB, bw=9165.9KB/s, iops=2291, runt= 40036msec > slat (usec): min=1, max=338238, avg=1320.61, stdev=9586.81 > clat (usec): min=912, max=972334, avg=69411.58, stdev=64715.54 > lat (usec): min=920, max=972338, avg=70732.51, stdev=66193.96 > clat percentiles (msec): > | 1.00th=[ 8], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 38], > | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 53], > | 70.00th=[ 60], 80.00th=[ 86], 90.00th=[ 128], 95.00th=[ 184], > | 99.00th=[ 379], 99.50th=[ 441], 99.90th=[ 570], 99.95th=[ 635], > | 99.99th=[ 824] > bw (KB /s): min= 7, max= 1771, per=10.21%, avg=935.65, stdev=439.55 > lat (usec) : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.04% > lat (msec) : 2=0.20%, 4=0.57%, 10=1.53%, 20=1.78%, 50=50.50% > lat (msec) : 100=28.04%, 250=14.56%, 500=2.47%, 750=0.23%, 1000=0.03% > cpu : usr=0.17%, sys=0.26%, ctx=16905, majf=0, minf=78 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, > >=64=0.0% > issued : total=r=92001/w=91741/d=0, short=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=368004KB, aggrb=9191KB/s, minb=9191KB/s, maxb=9191KB/s, > mint=40036msec, maxt=40036msec > WRITE: io=366964KB, aggrb=9165KB/s, minb=9165KB/s, maxb=9165KB/s, > mint=40036msec, maxt=40036msec > > Disk stats (read/write): > sdb: ios=91739/91385, merge=55/0, ticks=2854936/3040936, > in_queue=5901740, util=99.82% > > > -- > Vasiliy Tolstov, > e-mail: v.tols...@selfip.ru -- Vasiliy Tolstov, e-mail: v.tols...@selfip.ru