Hi Gilles,
Thanks a lot for the answer, i've made a new benchmark test with
bio, i've used the following configuration for the test:
; Four threads, two query, two writers.
[global]
rw=randread
size=256m
directory=/mnt/ceph-btrfs-test
ioengine=libaio
iodepth=4
invalidate=1
direct=1
[bgwriter]
rw=randwrite
iodepth=32
[queryA]
iodepth=1
ioengine=mmap
direct=0
thinktime=3
[queryB]
iodepth=1
ioengine=mmap
direct=0
thinktime=5
[bgupdater]
rw=randrw
iodepth=16
thinktime=40
size=32m
The results:
CEPH Cluster:
root@ceph-deploy01:/home/ceph/ceph-cluster# fio
four-threads-randio.fio
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio,
iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio,
iodepth=16
2.0.8
Starting 4 processes
bgwriter: Laying out IO file(s) (1 file(s) / 256MB)
queryA: Laying out IO file(s) (1 file(s) / 256MB)
queryB: Laying out IO file(s) (1 file(s) / 256MB)
bgupdater: Laying out IO file(s) (1 file(s) / 32MB)
Jobs: 2 (f=2): [_rr_] [99.8% done] [1164K/0K /s] [291 /0 iops] [eta
00m:01s]
bgwriter: (groupid=0, jobs=1): err= 0: pid=10481
write: io=262144KB, bw=3990.6KB/s, iops=997 , runt= 65691msec
slat (usec): min=35 , max=35877 , avg=131.07, stdev=284.30
clat (msec): min=3 , max=81 , avg=31.94, stdev= 6.72
lat (msec): min=3 , max=82 , avg=32.07, stdev= 6.70
clat percentiles (usec):
| 1.00th=[11456], 5.00th=[22400], 10.00th=[25472],
20.00th=[27520],
| 30.00th=[29056], 40.00th=[30080], 50.00th=[31104],
60.00th=[32384],
| 70.00th=[34048], 80.00th=[36608], 90.00th=[41216],
95.00th=[43776],
| 99.00th=[48896], 99.50th=[51968], 99.90th=[59136],
99.95th=[62208],
| 99.99th=[74240]
bw (KB/s) : min= 2909, max= 4525, per=94.09%, avg=3992.26,
stdev=481.59
lat (msec) : 4=0.01%, 10=0.78%, 20=2.23%, 50=96.20%, 100=0.78%
cpu : usr=0.85%, sys=11.90%, ctx=116709, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
queryA: (groupid=0, jobs=1): err= 0: pid=10482
read : io=262144KB, bw=517027 B/s, iops=126 , runt=519190msec
clat (usec): min=700 , max=46513 , avg=7845.19, stdev=13054.61
lat (usec): min=701 , max=46514 , avg=7845.79, stdev=13054.61
clat percentiles (usec):
| 1.00th=[ 1208], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[
1544],
| 30.00th=[ 1592], 40.00th=[ 1624], 50.00th=[ 1688], 60.00th=[
1752],
| 70.00th=[ 2256], 80.00th=[ 2992], 90.00th=[36608],
95.00th=[38656],
| 99.00th=[40704], 99.50th=[41216], 99.90th=[41728],
99.95th=[41728],
| 99.99th=[42752]
bw (KB/s) : min= 182, max= 769, per=48.49%, avg=504.32,
stdev=91.51
lat (usec) : 750=0.01%, 1000=0.09%
lat (msec) : 2=66.46%, 4=14.28%, 10=0.83%, 20=0.80%, 50=17.55%
cpu : usr=0.43%, sys=2.43%, ctx=196715, majf=65536, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
queryB: (groupid=0, jobs=1): err= 0: pid=10483
read : io=262144KB, bw=517127 B/s, iops=126 , runt=519090msec
clat (usec): min=688 , max=47423 , avg=7842.08, stdev=13027.06
lat (usec): min=689 , max=47424 , avg=7842.67, stdev=13027.06
clat percentiles (usec):
| 1.00th=[ 1224], 5.00th=[ 1416], 10.00th=[ 1448], 20.00th=[
1560],
| 30.00th=[ 1592], 40.00th=[ 1624], 50.00th=[ 1688], 60.00th=[
1768],
| 70.00th=[ 2256], 80.00th=[ 2992], 90.00th=[36608],
95.00th=[38656],
| 99.00th=[40704], 99.50th=[41216], 99.90th=[41728],
99.95th=[41728],
| 99.99th=[42240]
bw (KB/s) : min= 183, max= 782, per=48.55%, avg=504.87,
stdev=87.39
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=66.05%, 4=14.67%, 10=0.79%, 20=0.79%, 50=17.59%
cpu : usr=0.39%, sys=2.10%, ctx=196157, majf=65536, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
bgupdater: (groupid=0, jobs=1): err= 0: pid=10484
read : io=16164KB, bw=1095.9KB/s, iops=273 , runt= 14751msec
slat (usec): min=27 , max=263 , avg=78.80, stdev=26.74
clat (usec): min=574 , max=55711 , avg=16161.04, stdev=15577.04
lat (usec): min=660 , max=55765 , avg=16240.59, stdev=15578.88
clat percentiles (usec):
| 1.00th=[ 852], 5.00th=[ 1160], 10.00th=[ 1336], 20.00th=[
1576],
| 30.00th=[ 1992], 40.00th=[ 2864], 50.00th=[ 7648],
60.00th=[20352],
| 70.00th=[32128], 80.00th=[35584], 90.00th=[37632],
95.00th=[39168],
| 99.00th=[41216], 99.50th=[41728], 99.90th=[45312],
99.95th=[45824],
| 99.99th=[55552]
bw (KB/s) : min= 838, max= 1512, per=100.00%, avg=1098.41,
stdev=164.81
write: io=16604KB, bw=1125.7KB/s, iops=281 , runt= 14751msec
slat (usec): min=43 , max=281 , avg=119.68, stdev=33.39
clat (msec): min=5 , max=80 , avg=40.70, stdev= 6.48
lat (msec): min=5 , max=81 , avg=40.82, stdev= 6.48
clat percentiles (usec):
| 1.00th=[15296], 5.00th=[29056], 10.00th=[35584],
20.00th=[38144],
| 30.00th=[39168], 40.00th=[40192], 50.00th=[41216],
60.00th=[42240],
| 70.00th=[43264], 80.00th=[44800], 90.00th=[46848],
95.00th=[48896],
| 99.00th=[55040], 99.50th=[57600], 99.90th=[71168],
99.95th=[73216],
| 99.99th=[81408]
bw (KB/s) : min= 896, max= 1274, per=26.54%, avg=1126.14,
stdev=103.62
lat (usec) : 750=0.24%, 1000=1.05%
lat (msec) : 2=13.66%, 4=6.97%, 10=4.39%, 20=4.38%, 50=67.46%
lat (msec) : 100=1.84%
cpu : usr=0.76%, sys=5.91%, ctx=21589, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=4041/w=4151/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=540452KB, aggrb=1040KB/s, minb=504KB/s, maxb=1095KB/s,
mint=14751msec, maxt=519190msec
WRITE: io=278748KB, aggrb=4243KB/s, minb=1125KB/s, maxb=3990KB/s,
mint=14751msec, maxt=65691msec
root@ceph-deploy01:/home/ceph/ceph-cluster#
Also i've made the test on a NetApp FAS3250 cluster that we had:
root@e05-host05:/home/cloud# fio four-threats.fio
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio,
iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio,
iodepth=16
2.0.8
Starting 4 processes
bgwriter: Laying out IO file(s) (1 file(s) / 256MB)
queryA: Laying out IO file(s) (1 file(s) / 256MB)
queryB: Laying out IO file(s) (1 file(s) / 256MB)
bgupdater: Laying out IO file(s) (1 file(s) / 32MB)
Jobs: 3 (f=3): [wrr_] [2.7% done] [1454K/4075K /s] [363 /1018 iops]
[eta 07m:10
Jobs: 3 (f=3): [wrr_] [2.8% done] [359K/4007K /s] [89 /1001 iops]
[eta 07m:38s]
Jobs: 3 (f=3): [wrr_] [3.4% done] [1162K/3468K /s] [290 /867 iops]
[eta 07m:56s
Jobs: 3 (f=3): [wrr_] [3.7% done] [1214K/3440K /s] [303 /860 iops]
[eta 07m:51s
Jobs: 3 (f=3): [wrr_] [3.9% done] [1202K/3676K /s] [300 /919 iops]
[eta 07m:47s
Jobs: 3 (f=3): [wrr_] [4.1% done] [1194K/3264K /s] [298 /816 iops]
[eta 07m:43s
Jobs: 3 (f=3): [wrr_] [4.4% done] [1214K/3188K /s] [303 /797 iops]
[eta 07m:39s
Jobs: 3 (f=3): [wrr_] [4.6% done] [1234K/3080K /s] [308 /770 iops]
[eta 07m:35s
Jobs: 3 (f=3): [wrr_] [4.9% done] [1254K/3136K /s] [313 /784 iops]
[eta 07m:31s
Jobs: 3 (f=3): [wrr_] [5.1% done] [1246K/3192K /s] [311 /798 iops]
[eta 07m:27s
Jobs: 3 (f=3): [wrr_] [5.3% done] [1214K/3172K /s] [303 /793 iops]
[eta 07m:23s
Jobs: 3 (f=3): [wrr_] [5.6% done] [1238K/3172K /s] [309 /793 iops]
[eta 07m:20s
Jobs: 3 (f=3): [wrr_] [5.7% done] [535K/3164K /s] [133 /791 iops]
[eta 07m:29s]
Jobs: 3 (f=3): [wrr_] [6.2% done] [1150K/3100K /s] [287 /775 iops]
[eta 07m:37s
Jobs: 3 (f=3): [wrr_] [6.4% done] [1262K/3224K /s] [315 /806 iops]
[eta 07m:33s
Jobs: 3 (f=3): [wrr_] [6.6% done] [1110K/2817K /s] [277 /704 iops]
[eta 07m:32s
Jobs: 3 (f=3): [wrr_] [6.8% done] [931K/3052K /s] [232 /763 iops]
[eta 07m:33s]
Jobs: 1 (f=1): [_r__] [99.8% done] [1010K/0K /s] [252 /0 iops] [eta
00m:01s]
bgwriter: (groupid=0, jobs=1): err= 0: pid=29603
write: io=262144KB, bw=4563.9KB/s, iops=1140 , runt= 57439msec
slat (usec): min=9 , max=24719 , avg=855.55, stdev=1710.97
clat (usec): min=546 , max=281071 , avg=27186.12, stdev=14054.06
lat (usec): min=586 , max=281080 , avg=28042.59, stdev=14380.73
clat percentiles (usec):
| 1.00th=[ 1208], 5.00th=[ 4080], 10.00th=[ 7776],
20.00th=[13888],
| 30.00th=[19328], 40.00th=[23936], 50.00th=[27776],
60.00th=[31616],
| 70.00th=[35584], 80.00th=[39680], 90.00th=[44288],
95.00th=[48384],
| 99.00th=[56576], 99.50th=[59648], 99.90th=[72192],
99.95th=[116224],
| 99.99th=[257024]
bw (KB/s) : min= 2856, max=18376, per=91.18%, avg=4421.46,
stdev=2452.38
lat (usec) : 750=0.10%, 1000=0.50%
lat (msec) : 2=1.45%, 4=2.82%, 10=8.96%, 20=17.51%, 50=65.07%
lat (msec) : 100=3.51%, 250=0.06%, 500=0.01%
cpu : usr=0.73%, sys=51.81%, ctx=81663, majf=0, minf=756
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
queryA: (groupid=0, jobs=1): err= 0: pid=29604
read : io=262144KB, bw=535556 B/s, iops=130 , runt=501227msec
clat (usec): min=471 , max=664678 , avg=7580.25, stdev=9256.98
lat (usec): min=472 , max=664679 , avg=7580.55, stdev=9256.99
clat percentiles (usec):
| 1.00th=[ 820], 5.00th=[ 996], 10.00th=[ 2064], 20.00th=[
4256],
| 30.00th=[ 5280], 40.00th=[ 6112], 50.00th=[ 6816], 60.00th=[
7520],
| 70.00th=[ 8256], 80.00th=[ 9280], 90.00th=[11584],
95.00th=[14784],
| 99.00th=[28800], 99.50th=[41728], 99.90th=[109056],
99.95th=[173056],
| 99.99th=[370688]
bw (KB/s) : min= 7, max= 1136, per=48.75%, avg=525.52,
stdev=141.18
lat (usec) : 500=0.01%, 750=0.25%, 1000=4.82%
lat (msec) : 2=4.77%, 4=8.30%, 10=66.28%, 20=13.36%, 50=1.84%
lat (msec) : 100=0.26%, 250=0.10%, 500=0.01%, 750=0.01%
cpu : usr=0.14%, sys=0.35%, ctx=131440, majf=65536, minf=99
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
queryB: (groupid=0, jobs=1): err= 0: pid=29605
read : io=262144KB, bw=545259 B/s, iops=133 , runt=492308msec
clat (usec): min=366 , max=639108 , avg=7442.54, stdev=8790.57
lat (usec): min=366 , max=639108 , avg=7442.84, stdev=8790.57
clat percentiles (usec):
| 1.00th=[ 724], 5.00th=[ 964], 10.00th=[ 1992], 20.00th=[
4320],
| 30.00th=[ 5280], 40.00th=[ 5984], 50.00th=[ 6688], 60.00th=[
7328],
| 70.00th=[ 8096], 80.00th=[ 9024], 90.00th=[11328],
95.00th=[14400],
| 99.00th=[28800], 99.50th=[41216], 99.90th=[117248],
99.95th=[171008],
| 99.99th=[257024]
bw (KB/s) : min= 16, max= 1009, per=49.66%, avg=535.34,
stdev=148.02
lat (usec) : 500=0.11%, 750=1.04%, 1000=4.57%
lat (msec) : 2=4.30%, 4=7.47%, 10=68.26%, 20=12.12%, 50=1.75%
lat (msec) : 100=0.24%, 250=0.12%, 500=0.01%, 750=0.01%
cpu : usr=0.14%, sys=0.41%, ctx=131449, majf=65536,
minf=101
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
bgupdater: (groupid=0, jobs=1): err= 0: pid=29606
read : io=16368KB, bw=13918KB/s, iops=3479 , runt= 1176msec
slat (usec): min=7 , max=226 , avg=17.68, stdev=13.13
clat (usec): min=569 , max=15737 , avg=2111.01, stdev=985.20
lat (usec): min=588 , max=15812 , avg=2129.16, stdev=985.67
clat percentiles (usec):
| 1.00th=[ 1192], 5.00th=[ 1464], 10.00th=[ 1576], 20.00th=[
1672],
| 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1848], 60.00th=[
1960],
| 70.00th=[ 2128], 80.00th=[ 2384], 90.00th=[ 2768], 95.00th=[
3280],
| 99.00th=[ 7008], 99.50th=[ 8768], 99.90th=[11968],
99.95th=[12992],
| 99.99th=[15680]
bw (KB/s) : min=12840, max=14888, per=100.00%, avg=13864.00,
stdev=1448.15
write: io=16400KB, bw=13946KB/s, iops=3486 , runt= 1176msec
slat (usec): min=8 , max=287 , avg=19.21, stdev=14.16
clat (usec): min=594 , max=13975 , avg=2217.93, stdev=1060.31
lat (usec): min=615 , max=13989 , avg=2237.63, stdev=1060.73
clat percentiles (usec):
| 1.00th=[ 1240], 5.00th=[ 1576], 10.00th=[ 1688], 20.00th=[
1720],
| 30.00th=[ 1784], 40.00th=[ 1848], 50.00th=[ 1928], 60.00th=[
2040],
| 70.00th=[ 2224], 80.00th=[ 2448], 90.00th=[ 2832], 95.00th=[
3440],
| 99.00th=[ 7456], 99.50th=[ 9408], 99.90th=[12224],
99.95th=[12992],
| 99.99th=[14016]
bw (KB/s) : min=12632, max=15224, per=100.00%, avg=13928.00,
stdev=1832.82
lat (usec) : 750=0.09%, 1000=0.27%
lat (msec) : 2=59.50%, 4=36.61%, 10=3.25%, 20=0.29%
cpu : usr=3.74%, sys=11.91%, ctx=8588, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
=64=0.0%
issued : total=r=4092/w=4100/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=540656KB, aggrb=1078KB/s, minb=523KB/s, maxb=13918KB/s,
mint=1176msec, maxt=501227msec
WRITE: io=278544KB, aggrb=4849KB/s, minb=4563KB/s, maxb=13945KB/s,
mint=1176msec, maxt=57439msec
Disk stats (read/write):
dm-4: ios=135140/69692, merge=0/2236, ticks=984912/74076,
in_queue=1058872, util=100.00%, aggrios=67607/34846, aggrmerge=0/0,
aggrticks=490164/35990, aggrin_queue=526150, aggrutil=51.05%
sdc: ios=67522/34956, merge=0/0, ticks=489872/37288,
in_queue=527156, util=50.86%
sdd: ios=67692/34736, merge=0/0, ticks=490456/34692,
in_queue=525144, util=51.05%
root@e05-host05:/home/cloud#
German Anders
--- Original message ---
Asunto: Re: [ceph-users] Ceph Performance MB/sec
De: Gilles Mocellin <gilles.mocel...@nuagelibre.org>
Para: <ceph-users@lists.ceph.com>
Fecha: Sunday, 01/12/2013 13:59
Le 01/12/2013 15:22, German Anders a écrit :
[...]
ceph@ceph-deploy01:/mnt/ceph-btrfs-test$ for i in 1 2 3 4; do
sudo dd if=/dev/zero of=./a bs=1M count=1000; done
Hello,
You should really write anything but zeros.
I suspect that nothing is really written to disk, specially on
btrfs, a cow filesystem with thin provisioning features.
Perhaps Ceph itself handle that like a sparse file.
Write a real file (iso image) or use a benchmark tool made for it,
like fio.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
