[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

Thanks. It seems it is related to wpq implementation on how it is
organizing priorities!
I want to slow down the keys/s and I've set all the priorities to 1 for
recovery but it doesn't slow down!

On Thu, Dec 3, 2020 at 1:13 PM Anthony D'Atri 
wrote:

>
> >> If so why the client op priority is default 63 and recovery op is 3?
> This means that by default recovery op is more prioritize than client op!
> >
> > Exactly the opposite.  Client ops take priority over recovery ops.  And
> various other ops have priorities as described in the document I pointed
> you to.
>
>
> Now that I poke around, I do find some resources that claim that a *lower*
> osd_recovery_op_priority value *increases* the priority of recovery ops vs
> client ops, eg. this one:
>
> https://www.suse.com/support/kb/doc/?id=19693
>
> This is counter to everything I’ve always been told and observed.  Poking
> around in the sources my should-be-in-bed eyes don’t find a definitive
> answer, but I do see in OpRequest.cc :
>
>   if (req->get_priority() < tracker->cct->_conf->osd_client_op_priority) {
> // don't warn as quickly for low priority ops
> warn_interval_multiplier =
> tracker->cct->_conf->osd_recovery_op_warn_multiple;
>   }
>
> which seems to corroborate the idea that a lower value is a lower priority.
>
> If we can get consensus here I’ll add a bit to the docs to make the
> relationship more clear.
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Anthony D'Atri]

>> If so why the client op priority is default 63 and recovery op is 3? This 
>> means that by default recovery op is more prioritize than client op!
> 
> Exactly the opposite.  Client ops take priority over recovery ops.  And 
> various other ops have priorities as described in the document I pointed you 
> to.


Now that I poke around, I do find some resources that claim that a *lower* 
osd_recovery_op_priority value *increases* the priority of recovery ops vs 
client ops, eg. this one:

https://www.suse.com/support/kb/doc/?id=19693

This is counter to everything I’ve always been told and observed.  Poking 
around in the sources my should-be-in-bed eyes don’t find a definitive answer, 
but I do see in OpRequest.cc :

  if (req->get_priority() < tracker->cct->_conf->osd_client_op_priority) {
// don't warn as quickly for low priority ops 
warn_interval_multiplier = 
tracker->cct->_conf->osd_recovery_op_warn_multiple;
  }

which seems to corroborate the idea that a lower value is a lower priority.

If we can get consensus here I’ll add a bit to the docs to make the 
relationship more clear.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Anthony D'Atri]

> 
> 
> Sorry I got confused! Do you mean that both recovery_op_priority and 
> recovery_priority should be 63 to have a slow recovery?

recovery_priority is set per-pool to rank recovery relative to other pools.  
This is not related to your questions.

https://docs.ceph.com/en/latest/rados/operations/placement-groups/#prioritize-backfill-recovery-of-a-placement-group-s

> If so why the client op priority is default 63 and recovery op is 3? This 
> means that by default recovery op is more prioritize than client op!

Exactly the opposite.  Client ops take priority over recovery ops.  And various 
other ops have priorities as described in the document I pointed you to.

> In the link you share (Github one) It says that if priority is higher as an 
> integer it will mean that it prioritizes others!
> Can you please clarify this, please? I think this should be documented as 
> well in Ceph because in general, I think it means that a higher integer 
> number means higher priority than a lower integer number and so if we set 
> recovery op priority to 63 it will do recovery too fast and will impact the 
> overall cluster performance!

That’s why it is commonly suggested to set recovery_op_priority to 1 if you 
need to slow down recovery as well as the other values I sent you.

https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#operations


> Thanks.
> 
> On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri  
> wrote:
> FWIW 
> https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_reservation.rst
>  has some discussion of op priorities, though client ops aren’t mentioned 
> explicitly.  If you like, enter a documentation tracker and tag me and I’ll 
> look into adding that.
> 
> > On Dec 2, 2020, at 9:56 AM, Stefan Kooman  wrote:
> > 
> > On 12/2/20 5:36 PM, Seena Fallah wrote:
> >> If it uses PriorityQueue 
> >> 
> >>  Data Structure an element with high priority should be dequeued before an 
> >> element with low priority.
> > 
> > - osd client op priority
> > The priority set for client operations. It is relative to osd recovery op 
> > priority.
> > 
> > The op priority for client io is (default) 63. So with that regard a lower 
> > value should be lower priority. Same as the PriorityQueue you linked.
> > 
> > A simple example would be a great way to take out any confusion. I'll try 
> > to attend a "Docubetter" meeting to discuss this.
> > 
> > Gr. Stefan
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Stefan Kooman]

On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue 
 Data 
Structure an element with high priority should be dequeued before an 
element with low priority.


- osd client op priority
The priority set for client operations. It is relative to osd recovery 
op priority.


The op priority for client io is (default) 63. So with that regard a 
lower value should be lower priority. Same as the PriorityQueue you linked.


A simple example would be a great way to take out any confusion. I'll 
try to attend a "Docubetter" meeting to discuss this.


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

Sorry I got confused! Do you mean that both recovery_op_priority and
recovery_priority should be 63 to have a slow recovery?
If so why the client op priority is default 63 and recovery op is 3? This
means that by default recovery op is more prioritize than client op!
In the link you share (Github one) It says that if priority is higher as an
integer it will mean that it prioritizes others!
Can you please clarify this, please? I think this should be documented as
well in Ceph because in general, I think it means that a higher integer
number means higher priority than a lower integer number and so if we set
recovery op priority to 63 it will do recovery too fast and will impact the
overall cluster performance!

Thanks.

On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri 
wrote:

> FWIW
> https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_reservation.rst
> has some discussion of op priorities, though client ops aren’t mentioned
> explicitly.  If you like, enter a documentation tracker and tag me and I’ll
> look into adding that.
>
> > On Dec 2, 2020, at 9:56 AM, Stefan Kooman  wrote:
> >
> > On 12/2/20 5:36 PM, Seena Fallah wrote:
> >> If it uses PriorityQueue <
> https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.>
> Data Structure an element with high priority should be dequeued before an
> element with low priority.
> >
> > - osd client op priority
> > The priority set for client operations. It is relative to osd recovery
> op priority.
> >
> > The op priority for client io is (default) 63. So with that regard a
> lower value should be lower priority. Same as the PriorityQueue you linked.
> >
> > A simple example would be a great way to take out any confusion. I'll
> try to attend a "Docubetter" meeting to discuss this.
> >
> > Gr. Stefan
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Anthony D'Atri]

FWIW 
https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_reservation.rst
 has some discussion of op priorities, though client ops aren’t mentioned 
explicitly.  If you like, enter a documentation tracker and tag me and I’ll 
look into adding that.

> On Dec 2, 2020, at 9:56 AM, Stefan Kooman  wrote:
> 
> On 12/2/20 5:36 PM, Seena Fallah wrote:
>> If it uses PriorityQueue 
>> 
>>  Data Structure an element with high priority should be dequeued before an 
>> element with low priority.
> 
> - osd client op priority
> The priority set for client operations. It is relative to osd recovery op 
> priority.
> 
> The op priority for client io is (default) 63. So with that regard a lower 
> value should be lower priority. Same as the PriorityQueue you linked.
> 
> A simple example would be a great way to take out any confusion. I'll try to 
> attend a "Docubetter" meeting to discuss this.
> 
> Gr. Stefan
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Stefan Kooman]

On 12/2/20 3:04 PM, Seena Fallah wrote:
I don't think so! I want to slow down the recovery not speed up and it 
says I should reduce these values.


osd recovery op priority: This is the priority set for recovery 
operation. Lower the number, higher the recovery priority. Higher 
recovery priority might cause performance degradation until recovery 
completes.


"Lower the number, higher the recovery priority." With that reasoning 
"higher the number, lower the recovery priority" should also be true.


I would try "63" for both priorities.

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Stefan Kooman]

On 12/2/20 2:55 PM, Seena Fallah wrote:

This is what I used in recovery:
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 1


^^ Shouldn't this go to 63 instead of 1? At least if I read this post 
from SUSE correctly I think it should [1].



osd recovery priority = 1


^^ same for this.


osd recovery sleep ssd = 0.2


Gr. Stefan

[1]: https://www.suse.com/support/kb/doc/?id=19693
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Stefan Kooman]

On 12/2/20 2:46 PM, Seena Fallah wrote:

I did the same but it moved 200K keys/s!
You might also want to decrease the op priority (as in _increasing_ the 
number) of "osd_recovery_op_priority".


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Stefan Kooman]

On 12/1/20 12:37 AM, Seena Fallah wrote:

Hi all,

Is there any configuration to slow down keys/s in recovery mode?


Not just keys, but you can limit recovery / backfill like this:

ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Anthony D'Atri]

In certain cases (Luminous) it can actually be faster to destroy an OSD and 
recreate it than to let it backfill huge maps, but I think that’s been improved 
 by Nautilus.

You might also try setting

osd_op_queue_cut_off = high

to reduce the impact of recovery on client operations.  This became the default 
setting at Octopus; yours is likely set to low.



> I did the same but it moved 200K keys/s!
> 
> On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman  wrote:
> 
>> On 12/1/20 12:37 AM, Seena Fallah wrote:
>>> Hi all,
>>> 
>>> Is there any configuration to slow down keys/s in recovery mode?
>> 
>> Not just keys, but you can limit recovery / backfill like this:
>> 
>> ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
>> ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
>> 
>> Gr. Stefan
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

If it uses PriorityQueue

Data
Structure an element with high priority should be dequeued before an
element with low priority.

On Wed, Dec 2, 2020 at 7:32 PM Seena Fallah  wrote:

> "Higher recovery priority might cause performance degradation until
> recovery completes."
> But what about this statement? I found this that it means if I set
> priority to 63, I will lose the cluster performance for clients. Am I
> wrong? Does it mean the performance for recovery?
>
> I'm using nautilus 14.2.14.
> > [1]
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference
> Can you please point me to the correct section. I don't find the point of
> this doc.
>
> On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven  wrote:
>
>> Am 02.12.20 um 15:04 schrieb Seena Fallah:
>> > I don't think so! I want to slow down the recovery not speed up and it
>> says
>> > I should reduce these values.
>>
>>
>> I read the documentation the same. Low value = low weight, High value =
>> high weight. [1]
>>
>> Operations with higher weight get easier dispatched.
>>
>>
>> May I ask which exact issue are you seeing and which ceph version are you
>> using?
>>
>>
>> Peter
>>
>>
>> [1]
>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference
>>
>>
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

"Higher recovery priority might cause performance degradation until
recovery completes."
But what about this statement? I found this that it means if I set priority
to 63, I will lose the cluster performance for clients. Am I wrong? Does it
mean the performance for recovery?

I'm using nautilus 14.2.14.
> [1]
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference
Can you please point me to the correct section. I don't find the point of
this doc.

On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven  wrote:

> Am 02.12.20 um 15:04 schrieb Seena Fallah:
> > I don't think so! I want to slow down the recovery not speed up and it
> says
> > I should reduce these values.
>
>
> I read the documentation the same. Low value = low weight, High value =
> high weight. [1]
>
> Operations with higher weight get easier dispatched.
>
>
> May I ask which exact issue are you seeing and which ceph version are you
> using?
>
>
> Peter
>
>
> [1]
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Peter Lieven]

Am 02.12.20 um 15:04 schrieb Seena Fallah:
> I don't think so! I want to slow down the recovery not speed up and it says
> I should reduce these values.


I read the documentation the same. Low value = low weight, High value = high 
weight. [1]

Operations with higher weight get easier dispatched.


May I ask which exact issue are you seeing and which ceph version are you using?


Peter


[1] 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

I don't think so! I want to slow down the recovery not speed up and it says
I should reduce these values.

On Wed, Dec 2, 2020 at 5:31 PM Stefan Kooman  wrote:

> On 12/2/20 2:55 PM, Seena Fallah wrote:
> > This is what I used in recovery:
> > osd max backfills = 1
> > osd recovery max active = 1
> > osd recovery op priority = 1
>
> ^^ Shouldn't this go to 63 instead of 1? At least if I read this post
> from SUSE correctly I think it should [1].
>
> > osd recovery priority = 1
>
> ^^ same for this.
>
> > osd recovery sleep ssd = 0.2
>
> Gr. Stefan
>
> [1]: https://www.suse.com/support/kb/doc/?id=19693
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

This is what I used in recovery:
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 1
osd recovery priority = 1
osd recovery sleep ssd = 0.2

But it doesn't help much!

On Wed, Dec 2, 2020 at 5:23 PM Stefan Kooman  wrote:

> On 12/2/20 2:46 PM, Seena Fallah wrote:
> > I did the same but it moved 200K keys/s!
> You might also want to decrease the op priority (as in _increasing_ the
> number) of "osd_recovery_op_priority".
>
> Gr. Stefan
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow down keys/s in recovery

[Seena Fallah]

I did the same but it moved 200K keys/s!

On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman  wrote:

> On 12/1/20 12:37 AM, Seena Fallah wrote:
> > Hi all,
> >
> > Is there any configuration to slow down keys/s in recovery mode?
>
> Not just keys, but you can limit recovery / backfill like this:
>
> ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
> ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
>
> Gr. Stefan
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io