[ceph-users] Re: Odd 10-minute delay before recovery IO begins

2022-12-05 Thread Sean Matheny
Hi all, Thanks for the great responses. Confirming that this was the issue (feature). No idea why this was set differently for us in Nautilus. This should make the recovery benchmarking a bit faster now. :) Cheers, Sean > On 6/12/2022, at 3:09 PM, Wesley Dillingham wrote: > > I think you

[ceph-users] Re: Odd 10-minute delay before recovery IO begins

2022-12-05 Thread Wesley Dillingham
I think you are experiencing the mon_osd_down_out_interval https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/#confval-mon_osd_down_out_interval Ceph waits 10 minutes before marking a down osd as out for the reasons you mention, but this would have been the case in nautilus

[ceph-users] Re: Odd 10-minute delay before recovery IO begins

2022-12-05 Thread Tyler Brekke
Sounds like your OSDs were down, but not marked out. Recovery will only occur once they are actually marked out. The default mon_osd_down_out_interval is 10 minutes. You can mark them out explicitly with ceph osd out On Mon, Dec 5, 2022 at 2:20 PM Sean Matheny wrote: > Hi all, > > New Quincy

[ceph-users] Re: Odd 10-minute delay before recovery IO begins

2022-12-05 Thread Stephen Smith6
The 10 minute delay is the default wait period Ceph allows before it attempts to heal the data. See "mon_osd_report_timeout" – I believe the default is 900 seconds. From: Sean Matheny Date: Monday, December 5, 2022 at 5:20 PM To: ceph-users@ceph.io Cc: Blair Bethwaite , pi...@stackhpc.com ,