It's osd recover clone overlap (see http://tracker.ceph.com/issues/5401)
-----Original Message----- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Samuel Just Sent: mercredi 21 août 2013 17:33 To: Mike Dawson Cc: Stefan Priebe - Profihost AG; josh.dur...@inktank.com; ceph-devel@vger.kernel.org Subject: Re: still recovery issues with cuttlefish Have you tried setting osd_recovery_clone_overlap to false? That seemed to help with Stefan's issue. -Sam On Wed, Aug 21, 2013 at 8:28 AM, Mike Dawson <mike.daw...@cloudapt.com> wrote: > Sam/Josh, > > We upgraded from 0.61.7 to 0.67.1 during a maintenance window this > morning, hoping it would improve this situation, but there was no appreciable > change. > > One node in our cluster fsck'ed after a reboot and got a bit behind. > Our instances backed by RBD volumes were OK at that point, but once > the node booted fully and the OSDs started, all Windows instances with > rbd volumes experienced very choppy performance and were unable to > ingest video surveillance traffic and commit it to disk. Once the > cluster got back to HEALTH_OK, they resumed normal operation. > > I tried for a time with conservative recovery settings (osd max > backfills = 1, osd recovery op priority = 1, and osd recovery max > active = 1). No improvement for the guests. So I went to more > aggressive settings to get things moving faster. That decreased the duration > of the outage. > > During the entire period of recovery/backfill, the network looked > fine...no where close to saturation. iowait on all drives look fine as well. > > Any ideas? > > Thanks, > Mike Dawson > > > > On 8/14/2013 3:04 AM, Stefan Priebe - Profihost AG wrote: >> >> the same problem still occours. Will need to check when i've time to >> gather logs again. >> >> Am 14.08.2013 01:11, schrieb Samuel Just: >>> >>> I'm not sure, but your logs did show that you had >16 recovery ops >>> in flight, so it's worth a try. If it doesn't help, you should >>> collect the same set of logs I'll look again. Also, there are a few >>> other patches between 61.7 and current cuttlefish which may help. >>> -Sam >>> >>> On Tue, Aug 13, 2013 at 2:03 PM, Stefan Priebe - Profihost AG >>> <s.pri...@profihost.ag> wrote: >>>> >>>> >>>> Am 13.08.2013 um 22:43 schrieb Samuel Just <sam.j...@inktank.com>: >>>> >>>>> I just backported a couple of patches from next to fix a bug where >>>>> we weren't respecting the osd_recovery_max_active config in some >>>>> cases (1ea6b56170fc9e223e7c30635db02fa2ad8f4b4e). You can either >>>>> try the current cuttlefish branch or wait for a 61.8 release. >>>> >>>> >>>> Thanks! Are you sure that this is the issue? I don't believe that >>>> but i'll give it a try. I already tested a branch from sage where >>>> he fixed a race regarding max active some weeks ago. So active >>>> recovering was max 1 but the issue didn't went away. >>>> >>>> Stefan >>>> >>>>> -Sam >>>>> >>>>> On Mon, Aug 12, 2013 at 10:34 PM, Samuel Just >>>>> <sam.j...@inktank.com> >>>>> wrote: >>>>>> >>>>>> I got swamped today. I should be able to look tomorrow. Sorry! >>>>>> -Sam >>>>>> >>>>>> On Mon, Aug 12, 2013 at 9:39 PM, Stefan Priebe - Profihost AG >>>>>> <s.pri...@profihost.ag> wrote: >>>>>>> >>>>>>> Did you take a look? >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> Am 11.08.2013 um 05:50 schrieb Samuel Just <sam.j...@inktank.com>: >>>>>>> >>>>>>>> Great! I'll take a look on Monday. >>>>>>>> -Sam >>>>>>>> >>>>>>>> On Sat, Aug 10, 2013 at 12:08 PM, Stefan Priebe >>>>>>>> <s.pri...@profihost.ag> wrote: >>>>>>>>> >>>>>>>>> Hi Samual, >>>>>>>>> >>>>>>>>> Am 09.08.2013 23:44, schrieb Samuel Just: >>>>>>>>> >>>>>>>>>> I think Stefan's problem is probably distinct from Mike's. >>>>>>>>>> >>>>>>>>>> Stefan: Can you reproduce the problem with >>>>>>>>>> >>>>>>>>>> debug osd = 20 >>>>>>>>>> debug filestore = 20 >>>>>>>>>> debug ms = 1 >>>>>>>>>> debug optracker = 20 >>>>>>>>>> >>>>>>>>>> on a few osds (including the restarted osd), and upload those >>>>>>>>>> osd logs along with the ceph.log from before killing the osd >>>>>>>>>> until after the cluster becomes clean again? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> done - you'll find the logs at cephdrop folder: >>>>>>>>> slow_requests_recovering_cuttlefish >>>>>>>>> >>>>>>>>> osd.52 was the one recovering >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> Greets, >>>>>>>>> Stefan >>>>>>>> >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>> ceph-devel" in the body of a message to >>>>>>>> majord...@vger.kernel.org More majordomo info at >>>>>>>> http://vger.kernel.org/majordomo-info.html >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>>> in >>>>> the body of a message to majord...@vger.kernel.org More majordomo >>>>> info at http://vger.kernel.org/majordomo-info.html >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> ceph-devel" in the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
