Hi, bluefs_buffered_io was disabled in Ceph version 14.2.11.
The cluster started last year with 14.2.5 and got upgraded over the year now
running 14.2.16.
The performance was OK first but got abysmal bad at the end of 2020.
We checked the components and HDDs and SSDs seem to be fine. Single disk
benchmarks showed performance according the specs.
Today we (re-)enabled bluefs_buffered_io and restarted all OSD processes on 248
HDDs distributed over 12 nodes.
Now the benchmarks are fine again: 434MB/s write instead of 60MB/s, 960MB/s
read instead of 123MB/s.
This setting was disabled in 14.2.11 because "in some test cases it appears to
cause excessive swap utilization by the linux kernel and a large negative
performance impact after several hours of run time."
We have to monitor if this will happen in our cluster. Is there any other
negative side effect currently known?
Here are the rados bench values, first with bluefs_buffered_io=false, then with
bluefs_buffered_io=true:
Bench Total Total Write Object Band Stddev Max Min
Average Stddev Max Min Average Stddev Max Min
time writes Read size width Bandwidth
IOPS Latency (s)
run reads size (MB/sec)
made
false write 33,081 490 4194304 4194304 59,2485 71,3829 264 0
14 17,8702 66 0 1,07362 2,83017 20,71 0,0741089
false seq 15,8226 490 4194304 4194304 123,874
30 46,8659 174 0 0,51453 9,53873 0,00343417
false rand 38,2615 2131 4194304 4194304 222,782
55 109,374 415 0 0,28191 12,1039 0,00327948
true write 30,4612 3308 4194304 4194304 434,389 26,0323 480 376
108 6,50809 120 94 0,14683 0,07368 0,99791 0,0751249
true seq 13,7628 3308 4194304 4194304 961,429
240 22,544 280 184 0,06528 0,88676 0,00338191
true rand 30,1007 8247 4194304 4194304 1095,92
273 25,5066 313 213 0,05719 0,99140 0,00325295
Regards
--
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
