[ceph-users] Re: Lousy recovery for mclock and reef

[Joshua Baergen]

Hey Chris,

A number of users have been reporting issues with recovery on Reef
with mClock. Most folks have had success reverting to
osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but
I haven't looked at the list myself yet.

Josh

On Fri, May 24, 2024 at 10:55 AM Mazzystr  wrote:
>
> Hi all,
> Goodness I'd say it's been at least 3 major releases since I had to do a
> recovery.  I have disks with 60-75,000 power_on_hours.  I just updated from
> Octopus to Reef last month and I'm hit with 3 disk failures and the mclock
> ugliness.  My recovery is moving at a wondrous 21 mb/sec after some serious
> hacking.  It started out at 9 mb/sec.
>
> My hosts are showing minimal cpu use.  normal mem use.  0-6% disk
> business.  Load is minimal so processes aren't blocked by disk io.
>
> I tried the changing all the sleeps and recovery_max and
> setting osd_mclock_profile high_recovery_ops to no change in performance.
>
> Does anyone have any suggestions to improve performance?
>
> Thanks,
> /Chris C
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

Is that a setting that can be applied runtime or does it req osd restart?

On Fri, May 24, 2024 at 9:59 AM Joshua Baergen 
wrote:

> Hey Chris,
>
> A number of users have been reporting issues with recovery on Reef
> with mClock. Most folks have had success reverting to
> osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but
> I haven't looked at the list myself yet.
>
> Josh
>
> On Fri, May 24, 2024 at 10:55 AM Mazzystr  wrote:
> >
> > Hi all,
> > Goodness I'd say it's been at least 3 major releases since I had to do a
> > recovery.  I have disks with 60-75,000 power_on_hours.  I just updated
> from
> > Octopus to Reef last month and I'm hit with 3 disk failures and the
> mclock
> > ugliness.  My recovery is moving at a wondrous 21 mb/sec after some
> serious
> > hacking.  It started out at 9 mb/sec.
> >
> > My hosts are showing minimal cpu use.  normal mem use.  0-6% disk
> > business.  Load is minimal so processes aren't blocked by disk io.
> >
> > I tried the changing all the sleeps and recovery_max and
> > setting osd_mclock_profile high_recovery_ops to no change in performance.
> >
> > Does anyone have any suggestions to improve performance?
> >
> > Thanks,
> > /Chris C
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Joshua Baergen]

It requires an OSD restart, unfortunately.

Josh

On Fri, May 24, 2024 at 11:03 AM Mazzystr  wrote:
>
> Is that a setting that can be applied runtime or does it req osd restart?
>
> On Fri, May 24, 2024 at 9:59 AM Joshua Baergen 
> wrote:
>
> > Hey Chris,
> >
> > A number of users have been reporting issues with recovery on Reef
> > with mClock. Most folks have had success reverting to
> > osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but
> > I haven't looked at the list myself yet.
> >
> > Josh
> >
> > On Fri, May 24, 2024 at 10:55 AM Mazzystr  wrote:
> > >
> > > Hi all,
> > > Goodness I'd say it's been at least 3 major releases since I had to do a
> > > recovery.  I have disks with 60-75,000 power_on_hours.  I just updated
> > from
> > > Octopus to Reef last month and I'm hit with 3 disk failures and the
> > mclock
> > > ugliness.  My recovery is moving at a wondrous 21 mb/sec after some
> > serious
> > > hacking.  It started out at 9 mb/sec.
> > >
> > > My hosts are showing minimal cpu use.  normal mem use.  0-6% disk
> > > business.  Load is minimal so processes aren't blocked by disk io.
> > >
> > > I tried the changing all the sleeps and recovery_max and
> > > setting osd_mclock_profile high_recovery_ops to no change in performance.
> > >
> > > Does anyone have any suggestions to improve performance?
> > >
> > > Thanks,
> > > /Chris C
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

I did the obnoxious task of updating ceph.conf and restarting all my osds.

ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue
{
"osd_op_queue": "wpq"
}

I have some spare memory on my target host/osd and increased the target
memory of that OSD to 10 Gb and restarted.  No effect observed.  In fact
mem usage on the host is stable so I don't think the change took effect
even with updating ceph.conf, restart and a direct asok config set.  target
memory value is confirmed to be set via asok config get

Nothing has helped.  I still cannot break the 21 MiB/s barrier.

Does anyone have any more ideas?

/C

On Fri, May 24, 2024 at 10:20 AM Joshua Baergen 
wrote:

> It requires an OSD restart, unfortunately.
>
> Josh
>
> On Fri, May 24, 2024 at 11:03 AM Mazzystr  wrote:
> >
> > Is that a setting that can be applied runtime or does it req osd restart?
> >
> > On Fri, May 24, 2024 at 9:59 AM Joshua Baergen <
> jbaer...@digitalocean.com>
> > wrote:
> >
> > > Hey Chris,
> > >
> > > A number of users have been reporting issues with recovery on Reef
> > > with mClock. Most folks have had success reverting to
> > > osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but
> > > I haven't looked at the list myself yet.
> > >
> > > Josh
> > >
> > > On Fri, May 24, 2024 at 10:55 AM Mazzystr  wrote:
> > > >
> > > > Hi all,
> > > > Goodness I'd say it's been at least 3 major releases since I had to
> do a
> > > > recovery.  I have disks with 60-75,000 power_on_hours.  I just
> updated
> > > from
> > > > Octopus to Reef last month and I'm hit with 3 disk failures and the
> > > mclock
> > > > ugliness.  My recovery is moving at a wondrous 21 mb/sec after some
> > > serious
> > > > hacking.  It started out at 9 mb/sec.
> > > >
> > > > My hosts are showing minimal cpu use.  normal mem use.  0-6% disk
> > > > business.  Load is minimal so processes aren't blocked by disk io.
> > > >
> > > > I tried the changing all the sleeps and recovery_max and
> > > > setting osd_mclock_profile high_recovery_ops to no change in
> performance.
> > > >
> > > > Does anyone have any suggestions to improve performance?
> > > >
> > > > Thanks,
> > > > /Chris C
> > > > ___
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Joshua Baergen]

Now that you're on wpq, you can try tweaking osd_max_backfills (up)
and osd_recovery_sleep (down).

Josh

On Fri, May 24, 2024 at 1:07 PM Mazzystr  wrote:
>
> I did the obnoxious task of updating ceph.conf and restarting all my osds.
>
> ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue
> {
> "osd_op_queue": "wpq"
> }
>
> I have some spare memory on my target host/osd and increased the target 
> memory of that OSD to 10 Gb and restarted.  No effect observed.  In fact mem 
> usage on the host is stable so I don't think the change took effect even with 
> updating ceph.conf, restart and a direct asok config set.  target memory 
> value is confirmed to be set via asok config get
>
> Nothing has helped.  I still cannot break the 21 MiB/s barrier.
>
> Does anyone have any more ideas?
>
> /C
>
> On Fri, May 24, 2024 at 10:20 AM Joshua Baergen  
> wrote:
>>
>> It requires an OSD restart, unfortunately.
>>
>> Josh
>>
>> On Fri, May 24, 2024 at 11:03 AM Mazzystr  wrote:
>> >
>> > Is that a setting that can be applied runtime or does it req osd restart?
>> >
>> > On Fri, May 24, 2024 at 9:59 AM Joshua Baergen 
>> > wrote:
>> >
>> > > Hey Chris,
>> > >
>> > > A number of users have been reporting issues with recovery on Reef
>> > > with mClock. Most folks have had success reverting to
>> > > osd_op_queue=wpq. AIUI 18.2.3 should have some mClock improvements but
>> > > I haven't looked at the list myself yet.
>> > >
>> > > Josh
>> > >
>> > > On Fri, May 24, 2024 at 10:55 AM Mazzystr  wrote:
>> > > >
>> > > > Hi all,
>> > > > Goodness I'd say it's been at least 3 major releases since I had to do 
>> > > > a
>> > > > recovery.  I have disks with 60-75,000 power_on_hours.  I just updated
>> > > from
>> > > > Octopus to Reef last month and I'm hit with 3 disk failures and the
>> > > mclock
>> > > > ugliness.  My recovery is moving at a wondrous 21 mb/sec after some
>> > > serious
>> > > > hacking.  It started out at 9 mb/sec.
>> > > >
>> > > > My hosts are showing minimal cpu use.  normal mem use.  0-6% disk
>> > > > business.  Load is minimal so processes aren't blocked by disk io.
>> > > >
>> > > > I tried the changing all the sleeps and recovery_max and
>> > > > setting osd_mclock_profile high_recovery_ops to no change in 
>> > > > performance.
>> > > >
>> > > > Does anyone have any suggestions to improve performance?
>> > > >
>> > > > Thanks,
>> > > > /Chris C
>> > > > ___
>> > > > ceph-users mailing list -- ceph-users@ceph.io
>> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
>> > >
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Kai Stian Olstad]

On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my 
osds.


ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get 
osd_op_queue

{
"osd_op_queue": "wpq"
}

I have some spare memory on my target host/osd and increased the target
memory of that OSD to 10 Gb and restarted.  No effect observed.  In 
fact

mem usage on the host is stable so I don't think the change took effect
even with updating ceph.conf, restart and a direct asok config set.  
target

memory value is confirmed to be set via asok config get

Nothing has helped.  I still cannot break the 21 MiB/s barrier.

Does anyone have any more ideas?


For recovery you can adjust the following.

osd_max_backfills default is 1, in my system I get the best performance 
with 3 and wpq.


The following I have not adjusted myself, but you can try.
osd_recovery_max_active is default to 3.
osd_recovery_op_priority is default to 3, a lower number increases the 
priority for recovery.


All of them can be runtime adjusted.


--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

Well this was an interesting journey through the bowels of Ceph.  I have
about 6 hours into tweaking every setting imaginable just to circle back to
my basic configuration and 2G memory target per osd.  I was never able to
exceed 22 Mib/Sec recovery time during that journey.

I did end up fixing the issue and now I see the following -

  io:
recovery: 129 MiB/s, 33 objects/s

This is normal for my measly cluster.  I like micro ceph clusters.  I have
a lot of them. :)

What was the fix?  Adding another disc to the recovery process!  I was
recovering to one disc now I'm recovering to two.  I have three total that
need to be recovered.  Somehow that one disc was completely swamped.  I was
unable to see it in htop, atop, iostat.  Disc business was 6% max.

My config is back to mclock scheduler, profile high_recovery_ops, and
backfills of 256.

Thank you everyone that took the time to review and contribute.  Hopefully
this provides some modern information for the next person that has slow
recovery.

/Chris C





On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
wrote:

> On 24.05.2024 21:07, Mazzystr wrote:
> > I did the obnoxious task of updating ceph.conf and restarting all my
> > osds.
> >
> > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
> > osd_op_queue
> > {
> > "osd_op_queue": "wpq"
> > }
> >
> > I have some spare memory on my target host/osd and increased the target
> > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
> > fact
> > mem usage on the host is stable so I don't think the change took effect
> > even with updating ceph.conf, restart and a direct asok config set.
> > target
> > memory value is confirmed to be set via asok config get
> >
> > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
> >
> > Does anyone have any more ideas?
>
> For recovery you can adjust the following.
>
> osd_max_backfills default is 1, in my system I get the best performance
> with 3 and wpq.
>
> The following I have not adjusted myself, but you can try.
> osd_recovery_max_active is default to 3.
> osd_recovery_op_priority is default to 3, a lower number increases the
> priority for recovery.
>
> All of them can be runtime adjusted.
>
>
> --
> Kai Stian Olstad
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Zakhar Kirpichenko]

Hi!

Could you please elaborate what you meant by "adding another disc to the
recovery process"?

/Z


On Sat, 25 May 2024, 22:49 Mazzystr,  wrote:

> Well this was an interesting journey through the bowels of Ceph.  I have
> about 6 hours into tweaking every setting imaginable just to circle back to
> my basic configuration and 2G memory target per osd.  I was never able to
> exceed 22 Mib/Sec recovery time during that journey.
>
> I did end up fixing the issue and now I see the following -
>
>   io:
> recovery: 129 MiB/s, 33 objects/s
>
> This is normal for my measly cluster.  I like micro ceph clusters.  I have
> a lot of them. :)
>
> What was the fix?  Adding another disc to the recovery process!  I was
> recovering to one disc now I'm recovering to two.  I have three total that
> need to be recovered.  Somehow that one disc was completely swamped.  I was
> unable to see it in htop, atop, iostat.  Disc business was 6% max.
>
> My config is back to mclock scheduler, profile high_recovery_ops, and
> backfills of 256.
>
> Thank you everyone that took the time to review and contribute.  Hopefully
> this provides some modern information for the next person that has slow
> recovery.
>
> /Chris C
>
>
>
>
>
> On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
> wrote:
>
> > On 24.05.2024 21:07, Mazzystr wrote:
> > > I did the obnoxious task of updating ceph.conf and restarting all my
> > > osds.
> > >
> > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
> > > osd_op_queue
> > > {
> > > "osd_op_queue": "wpq"
> > > }
> > >
> > > I have some spare memory on my target host/osd and increased the target
> > > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
> > > fact
> > > mem usage on the host is stable so I don't think the change took effect
> > > even with updating ceph.conf, restart and a direct asok config set.
> > > target
> > > memory value is confirmed to be set via asok config get
> > >
> > > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
> > >
> > > Does anyone have any more ideas?
> >
> > For recovery you can adjust the following.
> >
> > osd_max_backfills default is 1, in my system I get the best performance
> > with 3 and wpq.
> >
> > The following I have not adjusted myself, but you can try.
> > osd_recovery_max_active is default to 3.
> > osd_recovery_op_priority is default to 3, a lower number increases the
> > priority for recovery.
> >
> > All of them can be runtime adjusted.
> >
> >
> > --
> > Kai Stian Olstad
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

I can't explain the problem.  I have to recover three discs that are hdds.
I figured on just replacing one to give the full recovery capacity of the
cluster to that one disc.  I was never able to achieve a higher recovery
rate than about 22 MiB/sec so I just added the other two discs.  Recovery
bounced up to 129 MiB/sec for a while.  Then things settled at 50 MiB/sec.
I kept tinkering to try to get back to 120 and now things are back to 23
MiB/Sec again.  This is very irritating.

Cpu usage is minimal in the single digit %.  Mem is right on target per
target setting in ceph.conf.  Disc's and network appear to be 20%
utilized.

I'm not a normal Ceph user.  I don't care about client access at all.  The
mclock assumptions are wrong for me.  I want my data to be replicated
correctly as fast as possible.

How do I open up the floodgates for maximum recovery performance?




On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko  wrote:

> Hi!
>
> Could you please elaborate what you meant by "adding another disc to the
> recovery process"?
>
> /Z
>
>
> On Sat, 25 May 2024, 22:49 Mazzystr,  wrote:
>
>> Well this was an interesting journey through the bowels of Ceph.  I have
>> about 6 hours into tweaking every setting imaginable just to circle back
>> to
>> my basic configuration and 2G memory target per osd.  I was never able to
>> exceed 22 Mib/Sec recovery time during that journey.
>>
>> I did end up fixing the issue and now I see the following -
>>
>>   io:
>> recovery: 129 MiB/s, 33 objects/s
>>
>> This is normal for my measly cluster.  I like micro ceph clusters.  I have
>> a lot of them. :)
>>
>> What was the fix?  Adding another disc to the recovery process!  I was
>> recovering to one disc now I'm recovering to two.  I have three total that
>> need to be recovered.  Somehow that one disc was completely swamped.  I
>> was
>> unable to see it in htop, atop, iostat.  Disc business was 6% max.
>>
>> My config is back to mclock scheduler, profile high_recovery_ops, and
>> backfills of 256.
>>
>> Thank you everyone that took the time to review and contribute.  Hopefully
>> this provides some modern information for the next person that has slow
>> recovery.
>>
>> /Chris C
>>
>>
>>
>>
>>
>> On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
>> wrote:
>>
>> > On 24.05.2024 21:07, Mazzystr wrote:
>> > > I did the obnoxious task of updating ceph.conf and restarting all my
>> > > osds.
>> > >
>> > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
>> > > osd_op_queue
>> > > {
>> > > "osd_op_queue": "wpq"
>> > > }
>> > >
>> > > I have some spare memory on my target host/osd and increased the
>> target
>> > > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
>> > > fact
>> > > mem usage on the host is stable so I don't think the change took
>> effect
>> > > even with updating ceph.conf, restart and a direct asok config set.
>> > > target
>> > > memory value is confirmed to be set via asok config get
>> > >
>> > > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
>> > >
>> > > Does anyone have any more ideas?
>> >
>> > For recovery you can adjust the following.
>> >
>> > osd_max_backfills default is 1, in my system I get the best performance
>> > with 3 and wpq.
>> >
>> > The following I have not adjusted myself, but you can try.
>> > osd_recovery_max_active is default to 3.
>> > osd_recovery_op_priority is default to 3, a lower number increases the
>> > priority for recovery.
>> >
>> > All of them can be runtime adjusted.
>> >
>> >
>> > --
>> > Kai Stian Olstad
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Sake Ceph]

Hi

Isn't this just the limit of one HDD or the other HDD's for providing the data? 
Don't forget, recovery will drop even more for the last few objects. At least I 
noticed this when replacing a drive in my (little) cluster. 

Kind regards, 
Sake 

> Op 26-05-2024 09:36 CEST schreef Mazzystr :
> 
>  
> I can't explain the problem.  I have to recover three discs that are hdds.
> I figured on just replacing one to give the full recovery capacity of the
> cluster to that one disc.  I was never able to achieve a higher recovery
> rate than about 22 MiB/sec so I just added the other two discs.  Recovery
> bounced up to 129 MiB/sec for a while.  Then things settled at 50 MiB/sec.
> I kept tinkering to try to get back to 120 and now things are back to 23
> MiB/Sec again.  This is very irritating.
> 
> Cpu usage is minimal in the single digit %.  Mem is right on target per
> target setting in ceph.conf.  Disc's and network appear to be 20%
> utilized.
> 
> I'm not a normal Ceph user.  I don't care about client access at all.  The
> mclock assumptions are wrong for me.  I want my data to be replicated
> correctly as fast as possible.
> 
> How do I open up the floodgates for maximum recovery performance?
> 
> 
> 
> 
> On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko  wrote:
> 
> > Hi!
> >
> > Could you please elaborate what you meant by "adding another disc to the
> > recovery process"?
> >
> > /Z
> >
> >
> > On Sat, 25 May 2024, 22:49 Mazzystr,  wrote:
> >
> >> Well this was an interesting journey through the bowels of Ceph.  I have
> >> about 6 hours into tweaking every setting imaginable just to circle back
> >> to
> >> my basic configuration and 2G memory target per osd.  I was never able to
> >> exceed 22 Mib/Sec recovery time during that journey.
> >>
> >> I did end up fixing the issue and now I see the following -
> >>
> >>   io:
> >> recovery: 129 MiB/s, 33 objects/s
> >>
> >> This is normal for my measly cluster.  I like micro ceph clusters.  I have
> >> a lot of them. :)
> >>
> >> What was the fix?  Adding another disc to the recovery process!  I was
> >> recovering to one disc now I'm recovering to two.  I have three total that
> >> need to be recovered.  Somehow that one disc was completely swamped.  I
> >> was
> >> unable to see it in htop, atop, iostat.  Disc business was 6% max.
> >>
> >> My config is back to mclock scheduler, profile high_recovery_ops, and
> >> backfills of 256.
> >>
> >> Thank you everyone that took the time to review and contribute.  Hopefully
> >> this provides some modern information for the next person that has slow
> >> recovery.
> >>
> >> /Chris C
> >>
> >>
> >>
> >>
> >>
> >> On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
> >> wrote:
> >>
> >> > On 24.05.2024 21:07, Mazzystr wrote:
> >> > > I did the obnoxious task of updating ceph.conf and restarting all my
> >> > > osds.
> >> > >
> >> > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
> >> > > osd_op_queue
> >> > > {
> >> > > "osd_op_queue": "wpq"
> >> > > }
> >> > >
> >> > > I have some spare memory on my target host/osd and increased the
> >> target
> >> > > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
> >> > > fact
> >> > > mem usage on the host is stable so I don't think the change took
> >> effect
> >> > > even with updating ceph.conf, restart and a direct asok config set.
> >> > > target
> >> > > memory value is confirmed to be set via asok config get
> >> > >
> >> > > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
> >> > >
> >> > > Does anyone have any more ideas?
> >> >
> >> > For recovery you can adjust the following.
> >> >
> >> > osd_max_backfills default is 1, in my system I get the best performance
> >> > with 3 and wpq.
> >> >
> >> > The following I have not adjusted myself, but you can try.
> >> > osd_recovery_max_active is default to 3.
> >> > osd_recovery_op_priority is default to 3, a lower number increases the
> >> > priority for recovery.
> >> >
> >> > All of them can be runtime adjusted.
> >> >
> >> >
> >> > --
> >> > Kai Stian Olstad
> >> >
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Cedric]

What about drives IOPS ? Hdd tops at an average 150, you can use iostat
-xmt to get these values (also last column show disk utilization which is
very usefull)

On Sun, May 26, 2024, 09:37 Mazzystr  wrote:

> I can't explain the problem.  I have to recover three discs that are hdds.
> I figured on just replacing one to give the full recovery capacity of the
> cluster to that one disc.  I was never able to achieve a higher recovery
> rate than about 22 MiB/sec so I just added the other two discs.  Recovery
> bounced up to 129 MiB/sec for a while.  Then things settled at 50 MiB/sec.
> I kept tinkering to try to get back to 120 and now things are back to 23
> MiB/Sec again.  This is very irritating.
>
> Cpu usage is minimal in the single digit %.  Mem is right on target per
> target setting in ceph.conf.  Disc's and network appear to be 20%
> utilized.
>
> I'm not a normal Ceph user.  I don't care about client access at all.  The
> mclock assumptions are wrong for me.  I want my data to be replicated
> correctly as fast as possible.
>
> How do I open up the floodgates for maximum recovery performance?
>
>
>
>
> On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko 
> wrote:
>
> > Hi!
> >
> > Could you please elaborate what you meant by "adding another disc to the
> > recovery process"?
> >
> > /Z
> >
> >
> > On Sat, 25 May 2024, 22:49 Mazzystr,  wrote:
> >
> >> Well this was an interesting journey through the bowels of Ceph.  I have
> >> about 6 hours into tweaking every setting imaginable just to circle back
> >> to
> >> my basic configuration and 2G memory target per osd.  I was never able
> to
> >> exceed 22 Mib/Sec recovery time during that journey.
> >>
> >> I did end up fixing the issue and now I see the following -
> >>
> >>   io:
> >> recovery: 129 MiB/s, 33 objects/s
> >>
> >> This is normal for my measly cluster.  I like micro ceph clusters.  I
> have
> >> a lot of them. :)
> >>
> >> What was the fix?  Adding another disc to the recovery process!  I was
> >> recovering to one disc now I'm recovering to two.  I have three total
> that
> >> need to be recovered.  Somehow that one disc was completely swamped.  I
> >> was
> >> unable to see it in htop, atop, iostat.  Disc business was 6% max.
> >>
> >> My config is back to mclock scheduler, profile high_recovery_ops, and
> >> backfills of 256.
> >>
> >> Thank you everyone that took the time to review and contribute.
> Hopefully
> >> this provides some modern information for the next person that has slow
> >> recovery.
> >>
> >> /Chris C
> >>
> >>
> >>
> >>
> >>
> >> On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
> >> wrote:
> >>
> >> > On 24.05.2024 21:07, Mazzystr wrote:
> >> > > I did the obnoxious task of updating ceph.conf and restarting all my
> >> > > osds.
> >> > >
> >> > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
> >> > > osd_op_queue
> >> > > {
> >> > > "osd_op_queue": "wpq"
> >> > > }
> >> > >
> >> > > I have some spare memory on my target host/osd and increased the
> >> target
> >> > > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
> >> > > fact
> >> > > mem usage on the host is stable so I don't think the change took
> >> effect
> >> > > even with updating ceph.conf, restart and a direct asok config set.
> >> > > target
> >> > > memory value is confirmed to be set via asok config get
> >> > >
> >> > > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
> >> > >
> >> > > Does anyone have any more ideas?
> >> >
> >> > For recovery you can adjust the following.
> >> >
> >> > osd_max_backfills default is 1, in my system I get the best
> performance
> >> > with 3 and wpq.
> >> >
> >> > The following I have not adjusted myself, but you can try.
> >> > osd_recovery_max_active is default to 3.
> >> > osd_recovery_op_priority is default to 3, a lower number increases the
> >> > priority for recovery.
> >> >
> >> > All of them can be runtime adjusted.
> >> >
> >> >
> >> > --
> >> > Kai Stian Olstad
> >> >
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Cedric]

Also osd_max_backfills and osd_recovery_max_active can plays a role, but I
wonder if they still has effect with the new mpq feature.

On Sun, May 26, 2024, 09:37 Mazzystr  wrote:

> I can't explain the problem.  I have to recover three discs that are hdds.
> I figured on just replacing one to give the full recovery capacity of the
> cluster to that one disc.  I was never able to achieve a higher recovery
> rate than about 22 MiB/sec so I just added the other two discs.  Recovery
> bounced up to 129 MiB/sec for a while.  Then things settled at 50 MiB/sec.
> I kept tinkering to try to get back to 120 and now things are back to 23
> MiB/Sec again.  This is very irritating.
>
> Cpu usage is minimal in the single digit %.  Mem is right on target per
> target setting in ceph.conf.  Disc's and network appear to be 20%
> utilized.
>
> I'm not a normal Ceph user.  I don't care about client access at all.  The
> mclock assumptions are wrong for me.  I want my data to be replicated
> correctly as fast as possible.
>
> How do I open up the floodgates for maximum recovery performance?
>
>
>
>
> On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko 
> wrote:
>
> > Hi!
> >
> > Could you please elaborate what you meant by "adding another disc to the
> > recovery process"?
> >
> > /Z
> >
> >
> > On Sat, 25 May 2024, 22:49 Mazzystr,  wrote:
> >
> >> Well this was an interesting journey through the bowels of Ceph.  I have
> >> about 6 hours into tweaking every setting imaginable just to circle back
> >> to
> >> my basic configuration and 2G memory target per osd.  I was never able
> to
> >> exceed 22 Mib/Sec recovery time during that journey.
> >>
> >> I did end up fixing the issue and now I see the following -
> >>
> >>   io:
> >> recovery: 129 MiB/s, 33 objects/s
> >>
> >> This is normal for my measly cluster.  I like micro ceph clusters.  I
> have
> >> a lot of them. :)
> >>
> >> What was the fix?  Adding another disc to the recovery process!  I was
> >> recovering to one disc now I'm recovering to two.  I have three total
> that
> >> need to be recovered.  Somehow that one disc was completely swamped.  I
> >> was
> >> unable to see it in htop, atop, iostat.  Disc business was 6% max.
> >>
> >> My config is back to mclock scheduler, profile high_recovery_ops, and
> >> backfills of 256.
> >>
> >> Thank you everyone that took the time to review and contribute.
> Hopefully
> >> this provides some modern information for the next person that has slow
> >> recovery.
> >>
> >> /Chris C
> >>
> >>
> >>
> >>
> >>
> >> On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad 
> >> wrote:
> >>
> >> > On 24.05.2024 21:07, Mazzystr wrote:
> >> > > I did the obnoxious task of updating ceph.conf and restarting all my
> >> > > osds.
> >> > >
> >> > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get
> >> > > osd_op_queue
> >> > > {
> >> > > "osd_op_queue": "wpq"
> >> > > }
> >> > >
> >> > > I have some spare memory on my target host/osd and increased the
> >> target
> >> > > memory of that OSD to 10 Gb and restarted.  No effect observed.  In
> >> > > fact
> >> > > mem usage on the host is stable so I don't think the change took
> >> effect
> >> > > even with updating ceph.conf, restart and a direct asok config set.
> >> > > target
> >> > > memory value is confirmed to be set via asok config get
> >> > >
> >> > > Nothing has helped.  I still cannot break the 21 MiB/s barrier.
> >> > >
> >> > > Does anyone have any more ideas?
> >> >
> >> > For recovery you can adjust the following.
> >> >
> >> > osd_max_backfills default is 1, in my system I get the best
> performance
> >> > with 3 and wpq.
> >> >
> >> > The following I have not adjusted myself, but you can try.
> >> > osd_recovery_max_active is default to 3.
> >> > osd_recovery_op_priority is default to 3, a lower number increases the
> >> > priority for recovery.
> >> >
> >> > All of them can be runtime adjusted.
> >> >
> >> >
> >> > --
> >> > Kai Stian Olstad
> >> >
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Sridhar Seshasayee]

With mClock, osd_max_backfills and osd_recovery_max_active can be modified
at
runtime after setting osd_mclock_override_recovery_settings to true. See
the docs

for more information.

There's no change in the behavior when recovery/backfill  limits are
modified with
mClock enabled.

I suspect when you added a new osd, the recovery traffic you observed could
just
be related to the backfill operation trying to move data due to PGs mapped
to the
new osd.

-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

I suspect my initial spike in performance was pg's balancing between the
three osd of the one host.

host load is very low, under 1.  hdd iops on the three discs hover
around 80 +/- 5.  atop shows about 20% business.  Gig-ethernet shows about
20% utilized according to atop.  I find it extremely hard to believe .2
gig-e can swamp three hdds.

I give up.  Recovery performance is what it is.


Next week when this operation completes I have two more osd recovery
operations where the osds are located on two different hosts.  It'll be
interesting to

/C

On Mon, May 27, 2024 at 1:17 AM Sridhar Seshasayee 
wrote:

> With mClock, osd_max_backfills and osd_recovery_max_active can be modified
> at
> runtime after setting osd_mclock_override_recovery_settings to true. See
> the docs
> 
> for more information.
>
> There's no change in the behavior when recovery/backfill  limits are
> modified with
> mClock enabled.
>
> I suspect when you added a new osd, the recovery traffic you observed
> could just
> be related to the backfill operation trying to move data due to PGs mapped
> to the
> new osd.
>
> -Sridhar
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Anthony D'Atri]

> 
>   hdd iops on the three discs hover around 80 +/- 5.  

Each or total?  I wouldn’t expect much more than 80 per drive.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Mazzystr]

Each simultaneously.  These are SATA3 with 128mb cache drives.  The bus is
6 gb/s.  I expect usage to be in the 90+% range not the 50% range.

On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri  wrote:

>
>
>
>   hdd iops on the three discs hover around 80 +/- 5.
>
>
> Each or total?  I wouldn’t expect much more than 80 per drive.
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Lousy recovery for mclock and reef

[Anthony D'Atri]

> 
> Each simultaneously.  These are SATA3 with 128mb cache drives.

Turn off the cache.

>  The bus is 6 gb/s.  I expect usage to be in the 90+% range not the 50% range.

"usage" as measured how?

> 
> On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri  wrote:
> 
>> 
>> 
>> 
>>  hdd iops on the three discs hover around 80 +/- 5.
>> 
>> 
>> Each or total?  I wouldn’t expect much more than 80 per drive.
>> 
>> 
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io