date:20171205

[ceph-users] ceph.conf tuning ... please comment

2017-12-05 Thread Stefan Kooman

Dear list,

In a ceph blog post about the new Luminous release there is a paragraph
on the need for ceph tuning [1]:

"If you are a Ceph power user and believe there is some setting that you
need to change for your environment to get the best performance, please
tell uswed like to either adjust our defaults so that your change isnt
necessary or have a go at convincing you that you shouldnt be tuning
that option."

We have been tuning several ceph.conf parameters in order to allow for
"fast failure" when an entire datacenter goes offline. We now have
continued operation (no pending IO) after ~ 7 seconds. We have changed
the following parameters:

[global]
# http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/
osd heartbeat grace = 4 # default 6
# Do _NOT_ scale based on laggy estimations
mon osd adjust heartbeat grace = false

^^ without this setting it could take up to two minutes before ceph
flagged a whole datacenter down (after we cut connectivity to the DC).
Not sure how the estimation is done, but not good enough for us.

[mon]
# http://docs.ceph.com/docs/master/rados/configuration/mon-config-ref/
# TUNING #
mon lease = 1.0# default 5
mon election timeout = 2 # default 5
mon lease renew interval factor = 0.4 # default 0.6
mon lease ack timeout factor = 1.5 # default 2.0
mon timecheck interval = 60# default 300

Above checks are there to make the whole process faster. After a DC
failure the monitors will need a re-election (depending on what DC and
who was a leader and who were peon). While going through mon
debug logging we have observed that this whole process is really fast
(things happen to be done in milliseconds). We have a quite low latency
network, so I guess we can cut some slack here. Ceph won't make any
decisions while there is no consensus, so better get that consensus as
soon as possible.

#
http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/#monitor-settings

mon osd reporter subtree level = datacenter

^^ We do want to make sure at least two datacenters are seeing a
datacenter go down, not individual hosts.

[osd]
# http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/
osd crush update on start = false
osd heartbeat interval = 1 # default 6
osd mon heartbeat interval = 10# default 30
osd mon report interval min = 1# default 5
osd mon report interval max = 15 # default 120

The osd would almost immediately see a "cut off" to their partner OSD's
in the placement group. By default they wait 6 seconds before sending
their report to the monitors. During our analysis this is exactly the
time the monitors were keeping an election. By tuning all of the above
we could get them to send their reports faster, and by the time the
election process was finished the monitors would handle the reports from
the OSDs and come to the conclusion that a DC is down, flag it down
and allow for normal client IO again.

Of course, stability and data safety is most important to us. So if any
of these settings make you worry please let us know.

Gr. Stefan

[1]: http://ceph.com/community/new-luminous-rados-improvements/

--
| BIT BV http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

39 matches

Mail list logo