Discuss: New default recovery config settings

2015-05-29 Thread Samuel Just
Many people have reported that they need to lower the osd recovery config options to minimize the impact of recovery on client io. We are talking about changing the defaults as follows: osd_max_backfills to 1 (from 10) osd_recovery_max_active to 3 (from 15) osd_recovery_op_priority to 1 (from 1

Re: Discuss: New default recovery config settings

2015-05-29 Thread Milosz Tanski
On Fri, May 29, 2015 at 5:47 PM, Samuel Just wrote: > Many people have reported that they need to lower the osd recovery config > options to minimize the impact of recovery on client io. We are talking > about changing the defaults as follows: > > osd_max_backfills to 1 (from 10) > osd_recovery

RE: Discuss: New default recovery config settings

2015-05-29 Thread Somnath Roy
7;ceph-us...@lists.ceph.com' (ceph-us...@lists.ceph.com) Subject: Discuss: New default recovery config settings Many people have reported that they need to lower the osd recovery config options to minimize the impact of recovery on client io. We are talking about changing the defaults as follows: osd_ma

Re: Discuss: New default recovery config settings

2015-05-29 Thread Gregory Farnum
On Fri, May 29, 2015 at 2:47 PM, Samuel Just wrote: > Many people have reported that they need to lower the osd recovery config > options to minimize the impact of recovery on client io. We are talking > about changing the defaults as follows: > > osd_max_backfills to 1 (from 10) > osd_recovery

Re: Discuss: New default recovery config settings

2015-05-31 Thread Justin Erenkrantz
On Fri, May 29, 2015 at 5:47 PM, Samuel Just wrote: > Many people have reported that they need to lower the osd recovery config > options to minimize the impact of recovery on client io. We are talking > about changing the defaults as follows: > > osd_max_backfills to 1 (from 10) > osd_recovery

RE: Discuss: New default recovery config settings

2015-06-01 Thread Paul Von-Stamwitz
On Fri, May 29, 2015 at 4:18 PM, Gregory Farnum wrote: > On Fri, May 29, 2015 at 2:47 PM, Samuel Just wrote: > > Many people have reported that they need to lower the osd recovery config > > options to minimize the impact of recovery on client io. We are talking > > about changing the defaults

Re: Discuss: New default recovery config settings

2015-06-01 Thread Gregory Farnum
On Mon, Jun 1, 2015 at 6:39 PM, Paul Von-Stamwitz wrote: > On Fri, May 29, 2015 at 4:18 PM, Gregory Farnum wrote: >> On Fri, May 29, 2015 at 2:47 PM, Samuel Just wrote: >> > Many people have reported that they need to lower the osd recovery config >> > options to minimize the impact of recovery

Re: Discuss: New default recovery config settings

2015-06-03 Thread Sage Weil
On Mon, 1 Jun 2015, Gregory Farnum wrote: > On Mon, Jun 1, 2015 at 6:39 PM, Paul Von-Stamwitz > wrote: > > On Fri, May 29, 2015 at 4:18 PM, Gregory Farnum wrote: > >> On Fri, May 29, 2015 at 2:47 PM, Samuel Just wrote: > >> > Many people have reported that they need to lower the osd recovery >

Re: Discuss: New default recovery config settings

2015-06-03 Thread Gregory Farnum
On Wed, Jun 3, 2015 at 3:44 PM, Sage Weil wrote: > On Mon, 1 Jun 2015, Gregory Farnum wrote: >> On Mon, Jun 1, 2015 at 6:39 PM, Paul Von-Stamwitz >> wrote: >> > On Fri, May 29, 2015 at 4:18 PM, Gregory Farnum wrote: >> >> On Fri, May 29, 2015 at 2:47 PM, Samuel Just wrote: >> >> > Many people h

Re: Discuss: New default recovery config settings

2015-06-04 Thread Mike Dawson
With a write-heavy RBD workload, I add the following to ceph.conf: osd_max_backfills = 2 osd_recovery_max_active = 2 If things are going well during recovery (i.e. guests happy and no slow requests), I will often bump both up to three: # ceph tell osd.* injectargs '--osd-max-backfills 3 --os

Re: [ceph-users] Discuss: New default recovery config settings

2015-06-01 Thread Jan Schermer
We had to disable deep scrub or the cluster would me unusable - we need to turn it back on sooner or later, though. With minimal scrubbing and recovery settings, everything is mostly good. Turned out many issues we had were due to too few PGs - once we increased them from 4K to 16K everything sp

Re: [ceph-users] Discuss: New default recovery config settings

2015-06-01 Thread Lionel Bouton
On 06/01/15 09:43, Jan Schermer wrote: > We had to disable deep scrub or the cluster would me unusable - we need to > turn it back on sooner or later, though. > With minimal scrubbing and recovery settings, everything is mostly good. > Turned out many issues we had were due to too few PGs - once

Re: [ceph-users] Discuss: New default recovery config settings

2015-06-01 Thread huang jun
hi,jan 2015-06-01 15:43 GMT+08:00 Jan Schermer : > We had to disable deep scrub or the cluster would me unusable - we need to > turn it back on sooner or later, though. > With minimal scrubbing and recovery settings, everything is mostly good. > Turned out many issues we had were due to too few

Re: [ceph-users] Discuss: New default recovery config settings

2015-06-01 Thread Jan Schermer
Slow requests are not exactly tied to the PG number, but we were getting slow requests whenever backfills or recoveries fired up - increasing the number of PGs helped with this as the “blocks” of work are much smaller than before. We have roughly the same number of OSDs as you but only one reall