Faidon has submitted this change and it was merged.
Change subject: Ceph: tune config knobs in response to mini-outage
......................................................................
Ceph: tune config knobs in response to mini-outage
gdb showed threads blocked on a pthread lock; further investigation
showed the thread pool has 100 max threads which is too few (it /might/
not be yet, but it will certainly be in the future).
out'ing two OSDs resulted in maxing out the GbE of two ms-be due to a
large number of backfills (256). Tune a couple of settings to sacrifice
recovery speed in favor of cluster stability.
Change-Id: I3945fff1cabfffde20519eed66670d62d89b2724
---
M manifests/role/ceph.pp
1 file changed, 6 insertions(+), 0 deletions(-)
Approvals:
Faidon: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/role/ceph.pp b/manifests/role/ceph.pp
index 68b4e9c..caa5f62 100644
--- a/manifests/role/ceph.pp
+++ b/manifests/role/ceph.pp
@@ -37,11 +37,17 @@
'osd' => {
'osd journal' =>
'/var/lib/ceph/journal/$cluster-$id',
'osd journal size' => '10240',
+ # lower from 5->3 and 10->5 respectively to ease up on the
+ # recovery traffic; GbE has easilly been maxed out before
+ 'osd recovery max active' => '3',
+ 'osd max backfills' => '5',
},
'radosgw' => {
'rgw print continue' => 'false',
'rgw enable ops log' => 'false',
'rgw enable usage log' => 'false',
+ # default is 100, far too small for the nr. of req that we want
+ 'rgw thread pool size' => '600',
'rgw extended http attrs' => 'x_content_duration',
'debug rgw' => '1',
},
--
To view, visit https://gerrit.wikimedia.org/r/80219
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I3945fff1cabfffde20519eed66670d62d89b2724
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon <fai...@wikimedia.org>
Gerrit-Reviewer: Faidon <fai...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
