> first of all, I'd still recommend to use the orchestrator to deploy OSDs.
> Building
> OSDs manually and then adopt them is redundant. Or do you have issues with
> the drivegroups?
I am having to do it this way because I couldn't find any doco on how to
specify a separate DB/WAL device when deploying OSDs using the orchestrator. If
there is such a command I agree it would be a better choice.
Ideally I would use the commands to simply move the DB of my existing
orchestrator deployed ODSs to the SSD, but when I tried that command it broke
my OSD and I had to delete it and leave he cluster in a degraded state until it
had recovered. I find it very stressful when I get out of my depth with
problems like that, so I gave up on that idea and am doing the remove, redploy,
adopt method, which is working, but VERY slow.
> I don't have *the* solution but you could try to disable the mclock scheduler
> [1] which is the default since Quincy. Maybe that will speed up things? There
> have been reports in the list about some unwanted or at least unexpected
> behavior.
I did try this to try and speed up my rebalances, but it didn't seem to make
much difference. I haven't tried it to see what difference it makes to
scrubbing.
> As for the "not (deep-)scrubbed in time" messages, there seems to be
> progress (in your ceph status), but depending on the drive utilization you
> could increase the number of scrubs per OSD (osd_max_scrubs).
There are lot of scrubs running, this morning after my rebalance finally
completed it has 22 scrubbing, 6 deep scrubbing (across 28 OSDs). This has
fallen from the number it was running yesterday when the rebalance was still
happening (38/9).
I believe if I kick the cluster by taking a host into maintenance and back the
numbers will jump up again. The trouble is I don't know how to tell if a scrub
is actually achieving something, stuck or restarting over and over.
My current ceph pg dump is:
https://pastebin.com/AQhNKSBN
and if I run it again a few minutes later:
https://pastebin.com/yfREzJ4s
I see evidence of scrubs not working because some of my OSD logs look like this:
2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 scrub starts
2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b scrub starts
2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 deep-scrub starts
2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b scrub starts
2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 scrub starts
2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b scrub starts
2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 deep-scrub starts
2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b deep-scrub starts
2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d deep-scrub starts
2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 scrub starts
2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b scrub starts
2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 scrub starts
2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b deep-scrub starts
2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.17 scrub starts
2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1d9 scrub starts
2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
5.9b scrub starts
2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] :
5.65 scrub starts
2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster) log [DBG] :
6.2d scrub starts
2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] :
5.1ac scrub starts
There is no scrub OK type messages.
I notice these logs were from yesterday, and currently there is nothing being
logged for these OSDs. If I was to "kick the cluster" these scrub logs would
probably start showing up again.
When this is happening the PGs mentioned seem to stay in a "scrub queued" state.
Any light you could shine my way would be appreciated.
Thanks,
Martin
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io