[ceph-users] Re: slow recovery with Quincy

[Ben]

thanks. I tried and it improved the situation by double the speed to 10MB/s
or something. Good catch for the fix!

It would be good to be at 50MB/s of recovery as far as cluster
infrastructure could support in my case. there may be other constraint on
resource utilization for recovery that I am not aware of?

胡 玮文  于2023年10月11日周三 00:18写道：

> Hi Ben,
>
> Please see this thread
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6
>  for
> possible workaround.
>
> 发自我的 iPad
>
> 在 2023年10月10日，22:26，Ben  写道：
>
> Dear cephers,
>
> with one osd down(200GB/9.1TB data), rebalance takes 3 hours still in
> progress. Client bandwidth can go as high as 200MB/s. With little client
> request throughput, recovery goes at couple MB/s. I wonder if there is
> configuration to polish for improvement. It runs with quincy 17.2.5,
> deployed by cephadm. The slowness can do harm in peak hours of usage.
>
> Best wishes,
>
> Ben
> -
>volumes: 1/1 healthy
>pools:   8 pools, 209 pgs
>objects: 93.04M objects, 4.8 TiB
>usage:   15 TiB used, 467 TiB / 482 TiB avail
>pgs: 1206837/279121971 objects degraded (0.432%)
> 208 active+clean
> 1   active+undersized+degraded+remapped+backfilling
>
>  io:
>client:   80 KiB/s rd, 420 KiB/s wr, 12 op/s rd, 29 op/s wr
>recovery: 6.2 MiB/s, 113 objects/s
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: slow recovery with Quincy

[胡 玮文]

Hi Ben,

Please see this thread 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6
 for possible workaround.

发自我的 iPad

在 2023年10月10日，22:26，Ben  写道：

Dear cephers,

with one osd down(200GB/9.1TB data), rebalance takes 3 hours still in
progress. Client bandwidth can go as high as 200MB/s. With little client
request throughput, recovery goes at couple MB/s. I wonder if there is
configuration to polish for improvement. It runs with quincy 17.2.5,
deployed by cephadm. The slowness can do harm in peak hours of usage.

Best wishes,

Ben
-
   volumes: 1/1 healthy
   pools:   8 pools, 209 pgs
   objects: 93.04M objects, 4.8 TiB
   usage:   15 TiB used, 467 TiB / 482 TiB avail
   pgs: 1206837/279121971 objects degraded (0.432%)
208 active+clean
1   active+undersized+degraded+remapped+backfilling

 io:
   client:   80 KiB/s rd, 420 KiB/s wr, 12 op/s rd, 29 op/s wr
   recovery: 6.2 MiB/s, 113 objects/s
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

Thanks, will keep an eye out for this version. Will report back to this thread 
about these options and the recovery time/number of objects per second for 
recovery.

Again, thank you'll for the information and answers!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sridhar Seshasayee]

Yes, the fix should be in the next quincy upstream version. The version I
posted was the downstream one.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

If I glance at the commits to the quincy branch, shouldn't the mentioned 
configurations be included in 17.2.7?

The requested command output:
[ceph: root@mgrhost1 /]# ceph version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)

[ceph: root@mgrhost1 /]# ceph config show-with-defaults osd.0 | grep osd_mclock
osd_mclock_cost_per_byte_usec 0.00 default
osd_mclock_cost_per_byte_usec_hdd 0.10 mon
osd_mclock_cost_per_byte_usec_ssd 0.10 mon
osd_mclock_cost_per_io_usec 0.00 default
osd_mclock_cost_per_io_usec_hdd 11400.00 default
osd_mclock_cost_per_io_usec_ssd 50.00 default
osd_mclock_force_run_benchmark_on_init false default
osd_mclock_iops_capacity_threshold_hdd 500.00 default
osd_mclock_iops_capacity_threshold_ssd 8.00 default
osd_mclock_max_capacity_iops_hdd 1.00 mon
osd_mclock_max_capacity_iops_ssd 0.00 override mon[1.00]
osd_mclock_override_recovery_settings false mon
osd_mclock_profile high_client_ops mon
osd_mclock_scheduler_anticipation_timeout 0.00 default
osd_mclock_scheduler_background_best_effort_lim 99 default
osd_mclock_scheduler_background_best_effort_res 500 default
osd_mclock_scheduler_background_best_effort_wgt 2 default
osd_mclock_scheduler_background_recovery_lim 2000 default
osd_mclock_scheduler_background_recovery_res 500 default
osd_mclock_scheduler_background_recovery_wgt 1 default
osd_mclock_scheduler_client_lim 99 default
osd_mclock_scheduler_client_res 1000 default
osd_mclock_scheduler_client_wgt 2 default
osd_mclock_skip_benchmark false default

[ceph: root@mgrhost1 /]# ceph config show osd.0 | grep osd_mclock
osd_mclock_cost_per_byte_usec_hdd 0.10 mon
osd_mclock_cost_per_byte_usec_ssd 0.10 mon
osd_mclock_max_capacity_iops_hdd 1.00 mon
osd_mclock_max_capacity_iops_ssd 0.00 override mon[1.00]
osd_mclock_override_recovery_settings false mon
osd_mclock_profile high_client_ops mon
osd_mclock_scheduler_background_best_effort_lim 99 default
osd_mclock_scheduler_background_best_effort_res 500 default
osd_mclock_scheduler_background_best_effort_wgt 2 default
osd_mclock_scheduler_background_recovery_lim 2000 default
osd_mclock_scheduler_background_recovery_res 500 default
osd_mclock_scheduler_background_recovery_wgt 1 default
osd_mclock_scheduler_client_lim 99 default
osd_mclock_scheduler_client_res 1000 default
osd_mclock_scheduler_client_wgt 2 default
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sridhar Seshasayee]

I'm on 17.2.6, but the option "osd_mclock_max_sequential_bandwidth_hdd"
> isn't available when I try to set it via "ceph config set osd.0
> osd_mclock_max_sequential_bandwidth_hdd 500Mi".
>
> Can you paste the output of
1. ceph version
2. ceph config show-with-defaults osd.0 | grep osd_mclock
3. ceph config show osd.0 | grep osd_mclock

I just want to verify if your Ceph version has the changes.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

I'm on 17.2.6, but the option "osd_mclock_max_sequential_bandwidth_hdd" isn't 
available when I try to set it via "ceph config set osd.0 
osd_mclock_max_sequential_bandwidth_hdd 500Mi".

I need to use large numbers for hdd, because it looks like the mclock scheduler 
isn't using the device class override value.

Best regards,
Sake


From: Sridhar Seshasayee 
Sent: Wednesday, May 24, 2023 11:34:02 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] Re: Slow recovery on Quincy

As someone in this thread noted, the cost related config options are
removed in the next version (ceph-17.2.6-45.el9cp).
The cost parameters may not work in all cases due to the inherent
differences in the underlying device types and other
external factors.

With the endeavor to achieve a more hands free operation, there are
significant improvements in the next version
that should help resolve issues described in this thread. I am listing the
significant ones below:

1. The mClock profile QoS parameters (reservation and limit) are now
simplified and are specified in terms of a fraction of the OSD's IOPS
capacity
rather than in terms of IOPS.
2. The default mclock profile is now changed to the 'balanced' profile
which gives equal priority to client and background OSD operations.
3. The allocations for different types of OSD ops within mClock
profiles are changed.
4. The earlier cost related config parameters are removed. The cost is
now determined by the OSD based on the underlying device
characteristics, i.e.,
 its 4 KiB random IOPS capacity and the device's max sequential bandwidth.
  -  The random IOPS capacity is determined using 'osd bench' as
before, but now based on the result, unrealistic values are not
considered and
  reasonable defaults are used if the measurement crosses a
threshold governed by *osd_mclock_iops_capacity_threshold_[hdd**|ssd].
*The
  default IOPS capacity may be overridden by users if not
accurate, The thresholds too are configurable. The max sequential
bandwidth is
  defined by *osd_mclock_max_sequential_bandwidth_[hdd|ssd*],
and are set to reasonable defaults. Again, these may be modified if
not accurate.
  Therefore, these changes account for inaccuracies and
provide good control to the user in terms of specifying accurate OSD
characteristics.
5. Degraded recoveries are given more priority than backfills.
Therefore, you may observe faster degraded recovery rates compared to
backfills with the 'balanced' and 'high_client_ops' profile. But
the backfills will still show healthy rates when compared to the slow
backfill rates
mentioned in this thread. For faster recovery and backfills, the
'high_recovery_ops' profile with modified QoS parameters would help.

Please see the latest upstream documentation for more details:
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/


The recommendation is to upgrade when feasible and provide your
feedback, questions and suggestions.

-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sridhar Seshasayee]

As someone in this thread noted, the cost related config options are
removed in the next version (ceph-17.2.6-45.el9cp).
The cost parameters may not work in all cases due to the inherent
differences in the underlying device types and other
external factors.

With the endeavor to achieve a more hands free operation, there are
significant improvements in the next version
that should help resolve issues described in this thread. I am listing the
significant ones below:

1. The mClock profile QoS parameters (reservation and limit) are now
simplified and are specified in terms of a fraction of the OSD's IOPS
capacity
rather than in terms of IOPS.
2. The default mclock profile is now changed to the 'balanced' profile
which gives equal priority to client and background OSD operations.
3. The allocations for different types of OSD ops within mClock
profiles are changed.
4. The earlier cost related config parameters are removed. The cost is
now determined by the OSD based on the underlying device
characteristics, i.e.,
 its 4 KiB random IOPS capacity and the device's max sequential bandwidth.
  -  The random IOPS capacity is determined using 'osd bench' as
before, but now based on the result, unrealistic values are not
considered and
  reasonable defaults are used if the measurement crosses a
threshold governed by *osd_mclock_iops_capacity_threshold_[hdd**|ssd].
*The
  default IOPS capacity may be overridden by users if not
accurate, The thresholds too are configurable. The max sequential
bandwidth is
  defined by *osd_mclock_max_sequential_bandwidth_[hdd|ssd*],
and are set to reasonable defaults. Again, these may be modified if
not accurate.
  Therefore, these changes account for inaccuracies and
provide good control to the user in terms of specifying accurate OSD
characteristics.
5. Degraded recoveries are given more priority than backfills.
Therefore, you may observe faster degraded recovery rates compared to
backfills with the 'balanced' and 'high_client_ops' profile. But
the backfills will still show healthy rates when compared to the slow
backfill rates
mentioned in this thread. For faster recovery and backfills, the
'high_recovery_ops' profile with modified QoS parameters would help.

Please see the latest upstream documentation for more details:
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/


The recommendation is to upgrade when feasible and provide your
feedback, questions and suggestions.

-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Paul Mezzanini]

I found that I could get much more aggressive with the 
osd_mclock_cost_per_byte_usec_hdd values if my  
osd_mclock_max_capacity_iops_hdd numbers were set to sane values.  If I used 
the (very wrong) default calculated values I would start getting slow ops.  
Setting all hdd in my environment to 300 seemed to calm that down.I'm 
running the mclock profile of high_client_ops at the moment.  I do get better 
recovery on balanced but I saw client latency go up enough where it wasn't 
worth the trade-off in my environment at this time. 

I'm still slowly walking the cost number down looking for the  sweet spot.  It 
seems to be very sensitive.  A change from .75 to .70 changed my IOPS/BW from 
480/1.48GB/s to 515/1.57GB/s.  It's easy to overshoot and hit some other 
bottleneck.  


---
Paul Mezzanini
Platform Engineer III
Research Computing

Rochester Institute of Technology

 “End users is a description, not a goal.”






From: Nigel Williams 
Sent: Tuesday, May 23, 2023 12:19 AM
To: David Orman
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Slow recovery on Quincy

We're on 17.2.5 and had the default value (5.2), but changing it didn't
seem to impact recovery speed:

root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
5.20
root@rdx-00:/# ceph config show osd.0 osd_op_queue
mclock_scheduler
root@rdx-00:/# ceph config set osd osd_mclock_cost_per_byte_usec_hdd 0.4
root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
0.40
root@rdx-00:/#

We have about 500TB of misplaced objects moving about so we are keen to
speed it up; was there another parameter that needed to be changed too?
thanks.


On Mon, 22 May 2023 at 23:24, David Orman  wrote:

> Someone who's got data regarding this should file a bug report, it sounds
> like a quick fix for defaults if this holds true.
>
> On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> > On 17/05/2023 03.07, 胡 玮文 wrote:
> >> Hi Sake,
> >>
> >> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd”
> to 0.1 (default is 2.6) and get about 15 times backfill speed, without
> significant affect client IO. This parameter seems calculated wrongly, from
> the description 5e-3 should be a reasonable value for HDD (corresponding to
> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to
> increase the recovery speed. So I suspect the original author just convert
> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
> >>
> >> But all this may be not important in the next version. I see the
> relevant code is rewritten, and this parameter is now removed.
> >>
> >> high_recovery_ops profile works very poorly for us. It increase the
> average latency of client IO from 50ms to about 1s.
> >>
> >> Weiwen Hu
> >>
> >
> > Thank you for this, that parameter indeed seems completely wrong
> > (assuming it means what it says on the tin). After changing that my
> > Quincy cluster is no recovering at a much more reasonable speed.
> >
> > - Hector
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Nigel Williams]

We're on 17.2.5 and had the default value (5.2), but changing it didn't
seem to impact recovery speed:

root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
5.20
root@rdx-00:/# ceph config show osd.0 osd_op_queue
mclock_scheduler
root@rdx-00:/# ceph config set osd osd_mclock_cost_per_byte_usec_hdd 0.4
root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
0.40
root@rdx-00:/#

We have about 500TB of misplaced objects moving about so we are keen to
speed it up; was there another parameter that needed to be changed too?
thanks.


On Mon, 22 May 2023 at 23:24, David Orman  wrote:

> Someone who's got data regarding this should file a bug report, it sounds
> like a quick fix for defaults if this holds true.
>
> On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> > On 17/05/2023 03.07, 胡 玮文 wrote:
> >> Hi Sake,
> >>
> >> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd”
> to 0.1 (default is 2.6) and get about 15 times backfill speed, without
> significant affect client IO. This parameter seems calculated wrongly, from
> the description 5e-3 should be a reasonable value for HDD (corresponding to
> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to
> increase the recovery speed. So I suspect the original author just convert
> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
> >>
> >> But all this may be not important in the next version. I see the
> relevant code is rewritten, and this parameter is now removed.
> >>
> >> high_recovery_ops profile works very poorly for us. It increase the
> average latency of client IO from 50ms to about 1s.
> >>
> >> Weiwen Hu
> >>
> >
> > Thank you for this, that parameter indeed seems completely wrong
> > (assuming it means what it says on the tin). After changing that my
> > Quincy cluster is no recovering at a much more reasonable speed.
> >
> > - Hector
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[David Orman]

Someone who's got data regarding this should file a bug report, it sounds like 
a quick fix for defaults if this holds true.

On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> On 17/05/2023 03.07, 胡 玮文 wrote:
>> Hi Sake,
>> 
>> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 
>> 0.1 (default is 2.6) and get about 15 times backfill speed, without 
>> significant affect client IO. This parameter seems calculated wrongly, from 
>> the description 5e-3 should be a reasonable value for HDD (corresponding to 
>> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to 
>> increase the recovery speed. So I suspect the original author just convert 
>> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
>> 
>> But all this may be not important in the next version. I see the relevant 
>> code is rewritten, and this parameter is now removed.
>> 
>> high_recovery_ops profile works very poorly for us. It increase the average 
>> latency of client IO from 50ms to about 1s.
>> 
>> Weiwen Hu
>> 
>
> Thank you for this, that parameter indeed seems completely wrong
> (assuming it means what it says on the tin). After changing that my
> Quincy cluster is no recovering at a much more reasonable speed.
>
> - Hector
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Hector Martin]

On 17/05/2023 03.07, 胡 玮文 wrote:
> Hi Sake,
> 
> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 
> 0.1 (default is 2.6) and get about 15 times backfill speed, without 
> significant affect client IO. This parameter seems calculated wrongly, from 
> the description 5e-3 should be a reasonable value for HDD (corresponding to 
> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to 
> increase the recovery speed. So I suspect the original author just convert 
> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
> 
> But all this may be not important in the next version. I see the relevant 
> code is rewritten, and this parameter is now removed.
> 
> high_recovery_ops profile works very poorly for us. It increase the average 
> latency of client IO from 50ms to about 1s.
> 
> Weiwen Hu
> 

Thank you for this, that parameter indeed seems completely wrong
(assuming it means what it says on the tin). After changing that my
Quincy cluster is no recovering at a much more reasonable speed.

- Hector
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Chris Palmer]

This is interesting, and it arrived minutes after I had replaced an HDD 
OSD (with NVME DB/WAL) in a small cluster. With the three profiles i was 
only seeing objects/second of around 6-8 (high_client_ops), 9-12 
(balanced), 12-15 (high_recovery_ops). There was only a very light 
client load.


With a bit of ad-hoc experimenting I ended up setting 
osd_mclock_cost_per_byte_usec_hdd to 0.4 which seemed to give about a 
7-8 times increase in objects/second for each profile, without unduly 
affecting client response if I used high_client_ops. Values lower than 
0.4 did speed up backfill, but disk saturation started to become an issue.


These are only very rough-and-ready figures, but they do suggest 
something rather wrong in the calculations.


Chris

On 16/05/2023 19:07, 胡 玮文 wrote:

Hi Sake,

We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 0.1 
(default is 2.6) and get about 15 times backfill speed, without significant 
affect client IO. This parameter seems calculated wrongly, from the description 
5e-3 should be a reasonable value for HDD (corresponding to 200MB/s). I noticed 
this default is originally 5.2, then changed to 2.6 to increase the recovery 
speed. So I suspect the original author just convert the unit wrongly, he may 
want 5.2e-3 but wrote 5.2 in code.

But all this may be not important in the next version. I see the relevant code 
is rewritten, and this parameter is now removed.

high_recovery_ops profile works very poorly for us. It increase the average 
latency of client IO from 50ms to about 1s.

Weiwen Hu

在 2023年5月16日，19:16，Sake Paulusma  写道：


We noticed extremely slow performance when remapping is necessary. We didn't do 
anything special other than assigning the correct device_class (to ssd). When 
checking ceph status, we notice the number of objects recovering is around 
17-25 (with watch -n 1 -c ceph status).

How can we increase the recovery process?

There isn't any client load, because we're going to migrate to this cluster in 
the future, so only a rsync once a while is being executed.

[ceph: root@pwsoel12998 /]# ceph status
  cluster:
id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b
health: HEALTH_WARN
noscrub,nodeep-scrub flag(s) set

  services:
mon: 5 daemons, quorum 
pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h)
mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt
mds: 4/4 daemons up, 2 standby
osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs
 flags noscrub,nodeep-scrub

  data:
volumes: 2/2 healthy
pools:   5 pools, 193 pgs
objects: 13.97M objects, 853 GiB
usage:   3.5 TiB used, 12 TiB / 16 TiB avail
pgs: 755092/55882956 objects misplaced (1.351%)
 183 active+clean
 10  active+remapped+backfilling

  io:
recovery: 2.3 MiB/s, 20 objects/s

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

Did an extra test with shutting down an osd host and force a recovery. Only 
using the iops setting I got 500 objects a second, but using also the 
bytes_per_usec setting, I got 1200 objects a second!

Maybe there should also be an investigation about this performance issue.

Best regards

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

Thanks for the input! Changing this value we indeed increased the recovery 
speed from 20 object per second to 500!

Now something strange:
1. We needed to change the class for our drives manually to ssd.
2. The setting "osd_mclock_max_capacity_iops_ssd" was set to 0. With osd bench 
descriped in the documentation, we configured the value 1 for the ssd 
parameter. Only nothing changed.
3. But when setting "osd_mclock_max_capacity_iops_hdd" to 1, the recovery 
speed also increased dramatically!

I don't understand this anymore :( Is the mclock scheduler ignoring the 
override of the device class?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[胡 玮文]

Hi Sake,

We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 0.1 
(default is 2.6) and get about 15 times backfill speed, without significant 
affect client IO. This parameter seems calculated wrongly, from the description 
5e-3 should be a reasonable value for HDD (corresponding to 200MB/s). I noticed 
this default is originally 5.2, then changed to 2.6 to increase the recovery 
speed. So I suspect the original author just convert the unit wrongly, he may 
want 5.2e-3 but wrote 5.2 in code.

But all this may be not important in the next version. I see the relevant code 
is rewritten, and this parameter is now removed.

high_recovery_ops profile works very poorly for us. It increase the average 
latency of client IO from 50ms to about 1s.

Weiwen Hu

在 2023年5月16日，19:16，Sake Paulusma  写道：


We noticed extremely slow performance when remapping is necessary. We didn't do 
anything special other than assigning the correct device_class (to ssd). When 
checking ceph status, we notice the number of objects recovering is around 
17-25 (with watch -n 1 -c ceph status).

How can we increase the recovery process?

There isn't any client load, because we're going to migrate to this cluster in 
the future, so only a rsync once a while is being executed.

[ceph: root@pwsoel12998 /]# ceph status
 cluster:
   id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b
   health: HEALTH_WARN
   noscrub,nodeep-scrub flag(s) set

 services:
   mon: 5 daemons, quorum 
pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h)
   mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt
   mds: 4/4 daemons up, 2 standby
   osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs
flags noscrub,nodeep-scrub

 data:
   volumes: 2/2 healthy
   pools:   5 pools, 193 pgs
   objects: 13.97M objects, 853 GiB
   usage:   3.5 TiB used, 12 TiB / 16 TiB avail
   pgs: 755092/55882956 objects misplaced (1.351%)
183 active+clean
10  active+remapped+backfilling

 io:
   recovery: 2.3 MiB/s, 20 objects/s

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

Just to add:
high_client_ops: around 8-13 objects per second
high_recovery_ops: around 17-25 objects per second

Both observed with "watch - n 1 - c ceph status"

Best regards

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sake Paulusma]

Hi,

The config shows "mclock_scheduler" and I already switched to the 
high_recovery_ops, this does increase the recovery ops, but only a little.

You mention there is a fix in 17.2.6+, but we're running on 17.2.6 (this 
cluster is created on this version). Any more ideas?

Best regards

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[Sridhar Seshasayee]

Hello,

This is a known issue in quincy which uses mClock scheduler. There is a fix
for this which should be
available in 17.2.6+ releases.

You can confirm the active scheduler type on any osd using:

 ceph config show osd.0 osd_op_queue

If the active scheduler is 'mclock_scheduler', you can try switching the
mClock profile to
'high_recovery_ops' on all OSDs to speed up the backfilling using:

ceph config set osd osd_mclock_profile high_recovery_ops

After the backfilling is complete, you can switch the mClock profile back
to the default value using:

ceph config rm osd osd_mclock_profile


On Tue, May 16, 2023 at 4:46 PM Sake Paulusma  wrote:

>
> We noticed extremely slow performance when remapping is necessary. We
> didn't do anything special other than assigning the correct device_class
> (to ssd). When checking ceph status, we notice the number of objects
> recovering is around 17-25 (with watch -n 1 -c ceph status).
>
> How can we increase the recovery process?
>
> There isn't any client load, because we're going to migrate to this
> cluster in the future, so only a rsync once a while is being executed.
>
> [ceph: root@pwsoel12998 /]# ceph status
>   cluster:
> id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b
> health: HEALTH_WARN
> noscrub,nodeep-scrub flag(s) set
>
>   services:
> mon: 5 daemons, quorum
> pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h)
> mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt
> mds: 4/4 daemons up, 2 standby
> osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs
>  flags noscrub,nodeep-scrub
>
>   data:
> volumes: 2/2 healthy
> pools:   5 pools, 193 pgs
> objects: 13.97M objects, 853 GiB
> usage:   3.5 TiB used, 12 TiB / 16 TiB avail
> pgs: 755092/55882956 objects misplaced (1.351%)
>  183 active+clean
>  10  active+remapped+backfilling
>
>   io:
> recovery: 2.3 MiB/s, 20 objects/s
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 

Sridhar Seshasayee

Partner Engineer

Red Hat 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io