Faidon has submitted this change and it was merged. Change subject: Ceph: tune config knobs in response to mini-outage ......................................................................
Ceph: tune config knobs in response to mini-outage gdb showed threads blocked on a pthread lock; further investigation showed the thread pool has 100 max threads which is too few (it /might/ not be yet, but it will certainly be in the future). out'ing two OSDs resulted in maxing out the GbE of two ms-be due to a large number of backfills (256). Tune a couple of settings to sacrifice recovery speed in favor of cluster stability. Change-Id: I3945fff1cabfffde20519eed66670d62d89b2724 --- M manifests/role/ceph.pp 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Faidon: Looks good to me, approved jenkins-bot: Verified diff --git a/manifests/role/ceph.pp b/manifests/role/ceph.pp index 68b4e9c..caa5f62 100644 --- a/manifests/role/ceph.pp +++ b/manifests/role/ceph.pp @@ -37,11 +37,17 @@ 'osd' => { 'osd journal' => '/var/lib/ceph/journal/$cluster-$id', 'osd journal size' => '10240', + # lower from 5->3 and 10->5 respectively to ease up on the + # recovery traffic; GbE has easilly been maxed out before + 'osd recovery max active' => '3', + 'osd max backfills' => '5', }, 'radosgw' => { 'rgw print continue' => 'false', 'rgw enable ops log' => 'false', 'rgw enable usage log' => 'false', + # default is 100, far too small for the nr. of req that we want + 'rgw thread pool size' => '600', 'rgw extended http attrs' => 'x_content_duration', 'debug rgw' => '1', }, -- To view, visit https://gerrit.wikimedia.org/r/80219 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I3945fff1cabfffde20519eed66670d62d89b2724 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Faidon <fai...@wikimedia.org> Gerrit-Reviewer: Faidon <fai...@wikimedia.org> Gerrit-Reviewer: jenkins-bot _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits