On Fri, Oct 19, 2018 at 08:32:52PM +0200, Sergio Lopez wrote:
> Yes, I ran some random read and random write tests with fio. The Guest
> is Linux, mainly because its virtio implementation is more mature. Next
> week I'll try with OpenBSD.
Here are the (long due) numbers with an OpenBSD 6.4 Guest. While on
Linux all tests performed better with kickfd, this time randread gets a
~10% improvement, while randwrite numbers are ~5% worse.
Please keep in mind that those are just the inital numbers. There's
still a lot of optimizations that can be applied on both the vmd side
(specially on writes, with things like request merging and asynchronous
I/O) and the Guest drivers (OpenBSD performance is roughly half of
Linux's, which indicates there's plenty of room for improvement).
In any case, I'd like to know if you think this kickfd feature is
something worth pursuing, or should I move to something else.
Cheers,
Sergio.
========================
| OpenBSD 6.4 + kickfd |
========================
- randread
# fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=1 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
fio-3.12-17-g0fcbc0
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=24.5MiB/s][r=6278 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=245800000: Fri Nov 23 09:59:54 2018
read: IOPS=6120, BW=23.9MiB/s (25.1MB/s)(1024MiB/42831msec)
clat (usec): min=10, max=10126, avg=127.91, stdev=36.28
lat (usec): min=19, max=10134, avg=136.25, stdev=36.58
clat percentiles (usec):
| 1.00th=[ 110], 5.00th=[ 112], 10.00th=[ 113], 20.00th=[ 119],
| 30.00th=[ 121], 40.00th=[ 126], 50.00th=[ 129], 60.00th=[ 131],
| 70.00th=[ 131], 80.00th=[ 133], 90.00th=[ 135], 95.00th=[ 145],
| 99.00th=[ 190], 99.50th=[ 208], 99.90th=[ 255], 99.95th=[ 318],
| 99.99th=[ 766]
bw ( KiB/s): min= 2669, max=24660, per=61.01%, avg=14936.35, stdev=7445.54,
samples=75
iops : min= 667, max= 6165, avg=3733.80, stdev=1861.45, samples=75
lat (usec) : 20=0.03%, 50=0.01%, 100=0.01%, 250=99.84%, 500=0.08%
lat (usec) : 750=0.02%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=0.69%, sys=41.15%, ctx=261996, majf=1, minf=3
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=23.9MiB/s (25.1MB/s), 23.9MiB/s-23.9MiB/s (25.1MB/s-25.1MB/s),
io=1024MiB (1074MB), run=42831-42831msec
# ./fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=2 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 2 threads
Jobs: 1 (f=1): [r(1),_(1)][98.8%][r=23.2MiB/s][r=5947 IOPS][eta 00m:01s]
randread: (groupid=0, jobs=2): err= 0: pid=1989935168: Fri Nov 23 10:03:27 2018
read: IOPS=6366, BW=24.9MiB/s (26.1MB/s)(2048MiB/82356msec)
clat (usec): min=10, max=1104.9k, avg=187.28, stdev=4505.64
lat (usec): min=19, max=1104.9k, avg=221.12, stdev=5417.17
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 110], 10.00th=[ 113], 20.00th=[ 118],
| 30.00th=[ 119], 40.00th=[ 120], 50.00th=[ 121], 60.00th=[ 122],
| 70.00th=[ 127], 80.00th=[ 133], 90.00th=[ 135], 95.00th=[ 135],
| 99.00th=[ 184], 99.50th=[ 221], 99.90th=[ 537], 99.95th=[ 906],
| 99.99th=[225444]
bw ( KiB/s): min= 110, max=24515, per=18.61%, avg=4738.70, stdev=5790.82,
samples=242
iops : min= 27, max= 6128, avg=1184.30, stdev=1447.69, samples=242
lat (usec) : 20=1.53%, 50=1.88%, 100=0.04%, 250=96.25%, 500=0.17%
lat (usec) : 750=0.07%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.02%, 100=0.01%
lat (msec) : 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.32%, sys=25.10%, ctx=638508, majf=0, minf=4
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=24.9MiB/s (26.1MB/s), 24.9MiB/s-24.9MiB/s (26.1MB/s-26.1MB/s),
io=2048MiB (2147MB), run=82356-82356msec
# ./fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=4 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 4 threads
Jobs: 1 (f=1): [_(2),r(1),_(1)][98.2%][r=21.4MiB/s][r=5488 IOPS][eta 00m:03s]
randread: (groupid=0, jobs=4): err= 0: pid=-805060032: Fri Nov 23 10:07:09 2018
read: IOPS=6266, BW=24.5MiB/s (25.7MB/s)(4096MiB/167342msec)
clat (usec): min=10, max=3119.3k, avg=341.60, stdev=13523.41
lat (usec): min=19, max=3119.3k, avg=409.38, stdev=14924.32
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 21], 10.00th=[ 111], 20.00th=[ 117],
| 30.00th=[ 120], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 122],
| 70.00th=[ 123], 80.00th=[ 133], 90.00th=[ 135], 95.00th=[ 137],
| 99.00th=[ 478], 99.50th=[ 523], 99.90th=[ 742], 99.95th=[ 23987],
| 99.99th=[700449]
bw ( KiB/s): min= 3, max=16050, per=2.45%, avg=614.96, stdev=1501.01,
samples=742
iops : min= 0, max= 4012, avg=153.37, stdev=375.28, samples=742
lat (usec) : 20=1.97%, 50=5.62%, 100=0.16%, 250=90.54%, 500=1.15%
lat (usec) : 750=0.45%, 1000=0.04%
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (msec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=0.21%, sys=14.03%, ctx=1393669, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=24.5MiB/s (25.7MB/s), 24.5MiB/s-24.5MiB/s (25.7MB/s-25.7MB/s),
io=4096MiB (4295MB), run=167342-167342msec
- randwrite
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=1 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
fio-3.12-17-g0fcbc0
Starting 1 thread
Jobs: 1 (f=1): [w(1)][100.0%][w=12.9MiB/s][w=3291 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1318987840: Fri Nov 23 10:11:10 2018
write: IOPS=3532, BW=13.8MiB/s (14.5MB/s)(1024MiB/74204msec)
clat (usec): min=116, max=383707, avg=208.57, stdev=1131.99
lat (usec): min=125, max=383715, avg=216.99, stdev=1132.04
clat percentiles (usec):
| 1.00th=[ 124], 5.00th=[ 127], 10.00th=[ 130], 20.00th=[ 133],
| 30.00th=[ 135], 40.00th=[ 139], 50.00th=[ 141], 60.00th=[ 145],
| 70.00th=[ 153], 80.00th=[ 265], 90.00th=[ 273], 95.00th=[ 277],
| 99.00th=[ 379], 99.50th=[ 562], 99.90th=[ 9372], 99.95th=[ 9503],
| 99.99th=[ 9896]
bw ( KiB/s): min= 3056, max=17819, per=95.00%, avg=13423.97, stdev=2726.80,
samples=148
iops : min= 764, max= 4454, avg=3355.61, stdev=681.70, samples=148
lat (usec) : 250=75.90%, 500=23.51%, 750=0.13%, 1000=0.04%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.38%, 20=0.01%, 500=0.01%
cpu : usr=0.48%, sys=33.40%, ctx=263585, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s),
io=1024MiB (1074MB), run=74204-74204msec
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=2 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 2 threads
Jobs: 1 (f=1): [_(1),w(1)][100.0%][w=14.8MiB/s][w=3799 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=2): err= 0: pid=-135734720: Fri Nov 23 10:16:36 2018
write: IOPS=3447, BW=13.5MiB/s (14.1MB/s)(2048MiB/152097msec)
clat (usec): min=15, max=429974, avg=505.90, stdev=1912.65
lat (usec): min=24, max=429982, avg=517.04, stdev=1916.41
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 124], 10.00th=[ 128], 20.00th=[ 133],
| 30.00th=[ 135], 40.00th=[ 139], 50.00th=[ 326], 60.00th=[ 338],
| 70.00th=[ 400], 80.00th=[ 619], 90.00th=[ 1012], 95.00th=[ 1418],
| 99.00th=[ 2999], 99.50th=[ 9110], 99.90th=[18482], 99.95th=[19268],
| 99.99th=[20317]
bw ( KiB/s): min= 2128, max=15030, per=47.84%, avg=6595.79, stdev=1414.59,
samples=604
iops : min= 532, max= 3757, avg=1648.59, stdev=353.63, samples=604
lat (usec) : 20=0.01%, 50=3.19%, 100=0.38%, 250=42.04%, 500=27.28%
lat (usec) : 750=9.69%, 1000=6.81%
lat (msec) : 2=8.46%, 4=1.36%, 10=0.59%, 20=0.18%, 50=0.01%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=0.29%, sys=22.79%, ctx=1151553, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=13.5MiB/s (14.1MB/s), 13.5MiB/s-13.5MiB/s (14.1MB/s-14.1MB/s),
io=2048MiB (2147MB), run=152097-152097msec
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=4 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 4 threads
Jobs: 1 (f=1): [_(2),w(1),_(1)][100.0%][w=11.1MiB/s][w=2846 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=-1510592448: Fri Nov 23 10:22:24
2018
write: IOPS=3284, BW=12.8MiB/s (13.5MB/s)(4096MiB/319238msec)
clat (usec): min=15, max=504371, avg=1032.12, stdev=3293.43
lat (usec): min=23, max=512815, avg=1088.02, stdev=3381.10
clat percentiles (usec):
| 1.00th=[ 36], 5.00th=[ 125], 10.00th=[ 133], 20.00th=[ 137],
| 30.00th=[ 396], 40.00th=[ 486], 50.00th=[ 644], 60.00th=[ 807],
| 70.00th=[ 996], 80.00th=[ 1287], 90.00th=[ 1844], 95.00th=[ 2573],
| 99.00th=[ 9110], 99.50th=[17695], 99.90th=[28967], 99.95th=[34341],
| 99.99th=[71828]
bw ( KiB/s): min= 138, max= 7895, per=23.84%, avg=3132.35, stdev=890.04,
samples=2541
iops : min= 34, max= 1973, avg=782.71, stdev=222.50, samples=2541
lat (usec) : 20=0.02%, 50=3.47%, 100=0.53%, 250=22.41%, 500=14.76%
lat (usec) : 750=15.11%, 1000=14.03%
lat (msec) : 2=21.20%, 4=6.21%, 10=1.52%, 20=0.49%, 50=0.23%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=0.15%, sys=12.27%, ctx=3122891, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=12.8MiB/s (13.5MB/s), 12.8MiB/s-12.8MiB/s (13.5MB/s-13.5MB/s),
io=4096MiB (4295MB), run=319238-319238msec
===========================
| OpenBSD 6.4 + NO kickfd |
===========================
- randread
# ./fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=1 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
fio-3.12-17-g0fcbc0
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=22.2MiB/s][r=5689 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=759733312: Fri Nov 23 10:28:56 2018
read: IOPS=5411, BW=21.1MiB/s (22.2MB/s)(1024MiB/48440msec)
clat (usec): min=10, max=986, avg=149.36, stdev=87.15
lat (usec): min=18, max=994, avg=157.53, stdev=87.24
clat percentiles (usec):
| 1.00th=[ 77], 5.00th=[ 78], 10.00th=[ 79], 20.00th=[ 86],
| 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 90],
| 70.00th=[ 253], 80.00th=[ 260], 90.00th=[ 269], 95.00th=[ 273],
| 99.00th=[ 318], 99.50th=[ 351], 99.90th=[ 416], 99.95th=[ 482],
| 99.99th=[ 766]
bw ( KiB/s): min=19734, max=22110, per=94.89%, avg=20540.47, stdev=478.77,
samples=96
iops : min= 4933, max= 5527, avg=5134.75, stdev=119.67, samples=96
lat (usec) : 20=0.03%, 50=0.01%, 100=63.54%, 250=2.01%, 500=34.37%
lat (usec) : 750=0.03%, 1000=0.01%
cpu : usr=0.60%, sys=77.27%, ctx=2596, majf=6, minf=3
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=21.1MiB/s (22.2MB/s), 21.1MiB/s-21.1MiB/s (22.2MB/s-22.2MB/s),
io=1024MiB (1074MB), run=48440-48440msec
# ./fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=2 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 2 threads
Jobs: 1 (f=1): [_(1),r(1)][97.9%][r=21.6MiB/s][r=5528 IOPS][eta 00m:02s]
randread: (groupid=0, jobs=2): err= 0: pid=-140517824: Fri Nov 23 10:31:39 2018
read: IOPS=5589, BW=21.8MiB/s (22.9MB/s)(2048MiB/93797msec)
clat (usec): min=10, max=154982, avg=283.67, stdev=2193.12
lat (usec): min=18, max=154991, avg=299.75, stdev=2253.64
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 77], 10.00th=[ 79], 20.00th=[ 86],
| 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 89], 60.00th=[ 90],
| 70.00th=[ 253], 80.00th=[ 258], 90.00th=[ 269], 95.00th=[ 273],
| 99.00th=[ 330], 99.50th=[12256], 99.90th=[24249], 99.95th=[24249],
| 99.99th=[94897]
bw ( KiB/s): min= 5051, max=23174, per=47.90%, avg=10708.88, stdev=1767.97,
samples=372
iops : min= 1262, max= 5793, avg=2676.88, stdev=442.01, samples=372
lat (usec) : 20=1.47%, 50=2.04%, 100=62.66%, 250=1.32%, 500=31.79%
lat (usec) : 750=0.05%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.27%, 50=0.36%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.41%, sys=38.78%, ctx=5087, majf=0, minf=4
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=21.8MiB/s (22.9MB/s), 21.8MiB/s-21.8MiB/s (22.9MB/s-22.9MB/s),
io=2048MiB (2147MB), run=93797-93797msec
# ./fio --clocksource=clock_gettime --name=randread --rw=randread --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=4 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 4 threads
Jobs: 3 (f=3): [r(1),_(1),r(2)][99.0%][r=23.9MiB/s][r=6115 IOPS][eta 00m:02s]
randread: (groupid=0, jobs=4): err= 0: pid=-318713792: Fri Nov 23 10:35:12 2018
read: IOPS=5503, BW=21.5MiB/s (22.5MB/s)(4096MiB/190525msec)
clat (usec): min=10, max=332015, avg=574.78, stdev=5598.22
lat (usec): min=18, max=332023, avg=608.63, stdev=5764.17
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 21], 10.00th=[ 79], 20.00th=[ 86],
| 30.00th=[ 88], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 90],
| 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 269], 95.00th=[ 273],
| 99.00th=[ 586], 99.50th=[ 47973], 99.90th=[ 71828], 99.95th=[ 71828],
| 99.99th=[166724]
bw ( KiB/s): min= 1782, max=10923, per=24.19%, avg=5325.29, stdev=1150.90,
samples=1514
iops : min= 445, max= 2730, avg=1330.96, stdev=287.72, samples=1514
lat (usec) : 20=2.48%, 50=5.45%, 100=58.13%, 250=1.19%, 500=31.43%
lat (usec) : 750=0.55%, 1000=0.04%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.29%
lat (msec) : 100=0.39%, 250=0.03%, 500=0.01%
cpu : usr=0.17%, sys=19.49%, ctx=9866, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=21.5MiB/s (22.5MB/s), 21.5MiB/s-21.5MiB/s (22.5MB/s-22.5MB/s),
io=4096MiB (4295MB), run=190525-190525msec
- randwrite
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=1 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
fio-3.12-17-g0fcbc0
Starting 1 thread
Jobs: 1 (f=1): [w(1)][100.0%][w=16.9MiB/s][w=4322 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1861118016: Fri Nov 23 10:40:57 2018
write: IOPS=3746, BW=14.6MiB/s (15.3MB/s)(1024MiB/69972msec)
clat (usec): min=119, max=19223, avg=231.25, stdev=564.33
lat (usec): min=127, max=19231, avg=239.64, stdev=564.38
clat percentiles (usec):
| 1.00th=[ 125], 5.00th=[ 127], 10.00th=[ 131], 20.00th=[ 135],
| 30.00th=[ 137], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 147],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 310], 95.00th=[ 322],
| 99.00th=[ 461], 99.50th=[ 709], 99.90th=[ 9503], 99.95th=[ 9634],
| 99.99th=[ 9896]
bw ( KiB/s): min= 9642, max=17950, per=95.15%, avg=14258.51, stdev=2772.35,
samples=139
iops : min= 2410, max= 4487, avg=3564.25, stdev=693.12, samples=139
lat (usec) : 250=65.41%, 500=33.77%, 750=0.33%, 1000=0.05%
lat (msec) : 2=0.02%, 4=0.04%, 10=0.39%, 20=0.01%
cpu : usr=0.76%, sys=78.02%, ctx=3507, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=14.6MiB/s (15.3MB/s), 14.6MiB/s-14.6MiB/s (15.3MB/s-15.3MB/s),
io=1024MiB (1074MB), run=69972-69972msec
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=2 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 2 threads
Jobs: 1 (f=1): [_(1),w(1)][99.3%][w=16.9MiB/s][w=4325 IOPS][eta 00m:01s]
randwrite: (groupid=0, jobs=2): err= 0: pid=1742875200: Fri Nov 23 10:44:00 2018
write: IOPS=3726, BW=14.6MiB/s (15.3MB/s)(2048MiB/140709msec)
clat (usec): min=60, max=173332, avg=461.77, stdev=2803.68
lat (usec): min=67, max=173346, avg=477.14, stdev=2837.11
clat percentiles (usec):
| 1.00th=[ 76], 5.00th=[ 126], 10.00th=[ 129], 20.00th=[ 135],
| 30.00th=[ 137], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 147],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 297], 95.00th=[ 322],
| 99.00th=[ 12125], 99.50th=[ 23987], 99.90th=[ 28705], 99.95th=[ 38011],
| 99.99th=[107480]
bw ( KiB/s): min= 2885, max=18075, per=47.39%, avg=7063.41, stdev=1745.48,
samples=559
iops : min= 721, max= 4518, avg=1765.48, stdev=436.35, samples=559
lat (usec) : 100=4.07%, 250=61.93%, 500=31.78%, 750=0.64%, 1000=0.17%
lat (msec) : 2=0.02%, 4=0.03%, 10=0.26%, 20=0.43%, 50=0.66%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.33%, sys=39.13%, ctx=7613, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=14.6MiB/s (15.3MB/s), 14.6MiB/s-14.6MiB/s (15.3MB/s-15.3MB/s),
io=2048MiB (2147MB), run=140709-140709msec
# ./fio --clocksource=clock_gettime --name=randwrite --rw=randwrite --bs=4k
--filename=/dev/sd1c --size=1g --numjobs=4 --ioengine=sync --iodepth=1
--direct=0 --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=sync, iodepth=1
...
fio-3.12-17-g0fcbc0
Starting 4 threads
Jobs: 3 (f=3): [_(1),w(3)][100.0%][w=13.3MiB/s][w=3405 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=-226515392: Fri Nov 23 10:49:27 2018
write: IOPS=3577, BW=13.0MiB/s (14.7MB/s)(4096MiB/293102msec)
clat (usec): min=60, max=356719, avg=961.03, stdev=6854.14
lat (usec): min=67, max=356726, avg=997.48, stdev=6991.00
clat percentiles (usec):
| 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 128], 20.00th=[ 137],
| 30.00th=[ 139], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 149],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 310], 95.00th=[ 351],
| 99.00th=[ 41157], 99.50th=[ 58459], 99.90th=[ 71828], 99.95th=[ 91751],
| 99.99th=[177210]
bw ( KiB/s): min= 746, max= 8118, per=23.52%, avg=3365.87, stdev=960.59,
samples=2341
iops : min= 186, max= 2029, avg=841.10, stdev=240.15, samples=2341
lat (usec) : 100=6.97%, 250=58.31%, 500=30.98%, 750=1.38%, 1000=0.52%
lat (msec) : 2=0.12%, 4=0.12%, 10=0.19%, 20=0.05%, 50=0.80%
lat (msec) : 100=0.54%, 250=0.04%, 500=0.01%
cpu : usr=0.16%, sys=19.53%, ctx=17858, majf=0, minf=2
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=13.0MiB/s (14.7MB/s), 13.0MiB/s-13.0MiB/s (14.7MB/s-14.7MB/s),
io=4096MiB (4295MB), run=293102-293102msec
