[ceph-users] Re: Lousy recovery for mclock and reef

> > Each simultaneously. These are SATA3 with 128mb cache drives. Turn off the cache. > The bus is 6 gb/s. I expect usage to be in the 90+% range not the 50% range. "usage" as measured how? > > On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri wrote: > >> >> >> >> hdd iops on the

[ceph-users] Re: Lousy recovery for mclock and reef

Each simultaneously. These are SATA3 with 128mb cache drives. The bus is 6 gb/s. I expect usage to be in the 90+% range not the 50% range. On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri wrote: > > > > hdd iops on the three discs hover around 80 +/- 5. > > > Each or total? I wouldn’t

[ceph-users] Re: Lousy recovery for mclock and reef

> > hdd iops on the three discs hover around 80 +/- 5. Each or total? I wouldn’t expect much more than 80 per drive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

I suspect my initial spike in performance was pg's balancing between the three osd of the one host. host load is very low, under 1. hdd iops on the three discs hover around 80 +/- 5. atop shows about 20% business. Gig-ethernet shows about 20% utilized according to atop. I find it extremely

[ceph-users] Re: Lousy recovery for mclock and reef

With mClock, osd_max_backfills and osd_recovery_max_active can be modified at runtime after setting osd_mclock_override_recovery_settings to true. See the docs for more

[ceph-users] Re: Lousy recovery for mclock and reef

Also osd_max_backfills and osd_recovery_max_active can plays a role, but I wonder if they still has effect with the new mpq feature. On Sun, May 26, 2024, 09:37 Mazzystr wrote: > I can't explain the problem. I have to recover three discs that are hdds. > I figured on just replacing one to give

[ceph-users] Re: Lousy recovery for mclock and reef

What about drives IOPS ? Hdd tops at an average 150, you can use iostat -xmt to get these values (also last column show disk utilization which is very usefull) On Sun, May 26, 2024, 09:37 Mazzystr wrote: > I can't explain the problem. I have to recover three discs that are hdds. > I figured on

[ceph-users] Re: Lousy recovery for mclock and reef

Hi Isn't this just the limit of one HDD or the other HDD's for providing the data? Don't forget, recovery will drop even more for the last few objects. At least I noticed this when replacing a drive in my (little) cluster. Kind regards, Sake > Op 26-05-2024 09:36 CEST schreef Mazzystr : >

[ceph-users] Re: Lousy recovery for mclock and reef

I can't explain the problem. I have to recover three discs that are hdds. I figured on just replacing one to give the full recovery capacity of the cluster to that one disc. I was never able to achieve a higher recovery rate than about 22 MiB/sec so I just added the other two discs. Recovery

[ceph-users] Re: Lousy recovery for mclock and reef

Hi! Could you please elaborate what you meant by "adding another disc to the recovery process"? /Z On Sat, 25 May 2024, 22:49 Mazzystr, wrote: > Well this was an interesting journey through the bowels of Ceph. I have > about 6 hours into tweaking every setting imaginable just to circle back

[ceph-users] Re: Lousy recovery for mclock and reef

Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey. I did end up fixing

[ceph-users] Re: Lousy recovery for mclock and reef

On 24.05.2024 21:07, Mazzystr wrote: I did the obnoxious task of updating ceph.conf and restarting all my osds. ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" } I have some spare memory on my target host/osd and increased the target

[ceph-users] Re: Lousy recovery for mclock and reef

Now that you're on wpq, you can try tweaking osd_max_backfills (up) and osd_recovery_sleep (down). Josh On Fri, May 24, 2024 at 1:07 PM Mazzystr wrote: > > I did the obnoxious task of updating ceph.conf and restarting all my osds. > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get

[ceph-users] Re: Lousy recovery for mclock and reef

I did the obnoxious task of updating ceph.conf and restarting all my osds. ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" } I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted.

[ceph-users] Re: Lousy recovery for mclock and reef

It requires an OSD restart, unfortunately. Josh On Fri, May 24, 2024 at 11:03 AM Mazzystr wrote: > > Is that a setting that can be applied runtime or does it req osd restart? > > On Fri, May 24, 2024 at 9:59 AM Joshua Baergen > wrote: > > > Hey Chris, > > > > A number of users have been

[ceph-users] Re: Lousy recovery for mclock and reef

Is that a setting that can be applied runtime or does it req osd restart? On Fri, May 24, 2024 at 9:59 AM Joshua Baergen wrote: > Hey Chris, > > A number of users have been reporting issues with recovery on Reef > with mClock. Most folks have had success reverting to > osd_op_queue=wpq. AIUI

[ceph-users] Re: Lousy recovery for mclock and reef

Hey Chris, A number of users have been reporting issues with recovery on Reef with mClock. Most folks have had success reverting to osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but I haven't looked at the list myself yet. Josh On Fri, May 24, 2024 at 10:55 AM Mazzystr