thanks. I tried and it improved the situation by double the speed to 10MB/s
or something. Good catch for the fix!
It would be good to be at 50MB/s of recovery as far as cluster
infrastructure could support in my case. there may be other constraint on
resource utilization for recovery that I am
Hi Ben,
Please see this thread
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6
for possible workaround.
发自我的 iPad
在 2023年10月10日，22:26，Ben 写道：
Dear cephers,
with one osd down(200GB/9.1TB data), rebalance
Thanks, will keep an eye out for this version. Will report back to this thread
about these options and the recovery time/number of objects per second for
recovery.
Again, thank you'll for the information and answers!
___
ceph-users mailing list --
Yes, the fix should be in the next quincy upstream version. The version I
posted was the downstream one.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
If I glance at the commits to the quincy branch, shouldn't the mentioned
configurations be included in 17.2.7?
The requested command output:
[ceph: root@mgrhost1 /]# ceph version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
[ceph: root@mgrhost1 /]# ceph config
I'm on 17.2.6, but the option "osd_mclock_max_sequential_bandwidth_hdd"
> isn't available when I try to set it via "ceph config set osd.0
> osd_mclock_max_sequential_bandwidth_hdd 500Mi".
>
> Can you paste the output of
1. ceph version
2. ceph config show-with-defaults osd.0 | grep osd_mclock
3.
ing the device class override value.
Best regards,
Sake
From: Sridhar Seshasayee
Sent: Wednesday, May 24, 2023 11:34:02 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Slow recovery on Quincy
As someone in this thread noted, the cost related config options are
As someone in this thread noted, the cost related config options are
removed in the next version (ceph-17.2.6-45.el9cp).
The cost parameters may not work in all cases due to the inherent
differences in the underlying device types and other
external factors.
With the endeavor to achieve a more
users is a description, not a goal.”
From: Nigel Williams
Sent: Tuesday, May 23, 2023 12:19 AM
To: David Orman
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Slow recovery on Quincy
We're on 17.2.5 and had the default value (5.2), but changing
We're on 17.2.5 and had the default value (5.2), but changing it didn't
seem to impact recovery speed:
root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
5.20
root@rdx-00:/# ceph config show osd.0 osd_op_queue
mclock_scheduler
root@rdx-00:/# ceph config set osd
Someone who's got data regarding this should file a bug report, it sounds like
a quick fix for defaults if this holds true.
On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> On 17/05/2023 03.07, 胡 玮文 wrote:
>> Hi Sake,
>>
>> We are experiencing the same. I set
On 17/05/2023 03.07, 胡 玮文 wrote:
> Hi Sake,
>
> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to
> 0.1 (default is 2.6) and get about 15 times backfill speed, without
> significant affect client IO. This parameter seems calculated wrongly, from
> the description 5e-3
This is interesting, and it arrived minutes after I had replaced an HDD
OSD (with NVME DB/WAL) in a small cluster. With the three profiles i was
only seeing objects/second of around 6-8 (high_client_ops), 9-12
(balanced), 12-15 (high_recovery_ops). There was only a very light
client load.
Did an extra test with shutting down an osd host and force a recovery. Only
using the iops setting I got 500 objects a second, but using also the
bytes_per_usec setting, I got 1200 objects a second!
Maybe there should also be an investigation about this performance issue.
Best regards
Thanks for the input! Changing this value we indeed increased the recovery
speed from 20 object per second to 500!
Now something strange:
1. We needed to change the class for our drives manually to ssd.
2. The setting "osd_mclock_max_capacity_iops_ssd" was set to 0. With osd bench
descriped in
Hi Sake,
We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 0.1
(default is 2.6) and get about 15 times backfill speed, without significant
affect client IO. This parameter seems calculated wrongly, from the description
5e-3 should be a reasonable value for HDD
Just to add:
high_client_ops: around 8-13 objects per second
high_recovery_ops: around 17-25 objects per second
Both observed with "watch - n 1 - c ceph status"
Best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an
Hi,
The config shows "mclock_scheduler" and I already switched to the
high_recovery_ops, this does increase the recovery ops, but only a little.
You mention there is a fix in 17.2.6+, but we're running on 17.2.6 (this
cluster is created on this version). Any more ideas?
Best regards
Hello,
This is a known issue in quincy which uses mClock scheduler. There is a fix
for this which should be
available in 17.2.6+ releases.
You can confirm the active scheduler type on any osd using:
ceph config show osd.0 osd_op_queue
If the active scheduler is 'mclock_scheduler', you can try
19 matches
Mail list logo
