[ceph-users] Re: Scrubbing?

[Josh Baergen]

Ah, yeah, you hit https://tracker.ceph.com/issues/63389 during the upgrade.

Josh

On Tue, Jan 30, 2024 at 3:17 AM Jan Marek  wrote:
>
> Hello again,
>
> I'm sorry, I forgot attach file... :-(
>
> Sincerely
> Jan
>
> Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
> > Hello Sridhar,
> >
> > at Saturday I've finished upgrade proces to 18.2.1.
> >
> > Cluster is now in HEALTH_OK state and performs well.
> >
> > According to my colleagues there are lower latences and good
> > throughput.
> >
> > On OSD nodes there is relative low I/O activity.
> >
> > I still have mClock profile "high_client_ops".
> >
> > When I was stucked in the upgrade process, I had in logs so many
> > records, see attached file. Since upgrade is complete, this
> > messages went away... Can be this reason of poor
> > performance?
> >
> > Sincerely
> > Jan Marek
> >
> > Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> > > Hello Sridhar,
> > >
> > > Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > > > Hello Jan,
> > > >
> > > > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > > > my needs and, although I had set mClock profile to
> > > > "high_client_ops" (because I have a plenty of time to rebalancing
> > > > and scrubbing), my clients went to problems.
> > > >
> > > > As far as the question around mClock is concerned, there are further
> > > > improvements in the works to handle QoS between client ops and
> > > > background scrub ops. This should help address the issue you are
> > > > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > > > for more information.
> > > > Also, it would be helpful to know the Ceph version you are currently 
> > > > using.
> > >
> > > thanks for your reply.
> > >
> > > I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> > > see my previous posts about stuck in upgrade to reef).
> > >
> > > Maybe this was cause of my problem...
> > >
> > > Now I've tried give rest to the cluster to do some "background"
> > > tasks (and it seems, that this was correct, because on my hosts
> > > there is around 50-100MBps read and cca 10-50MBps write traffic -
> > > cca 1/4-1/2 of previous load).
> > >
> > > At Saturday I will change some settings on networking and I will
> > > try to start upgrade process, maybe with --limit=1, to be "soft"
> > > for cluster and for our clients...
> > >
> > > > -Sridhar
> > >
> > > Sincerely
> > > Jan Marek
> > > --
> > > Ing. Jan Marek
> > > University of South Bohemia
> > > Academic Computer Centre
> > > Phone: +420389032080
> > > http://www.gnu.org/philosophy/no-word-attachments.cs.html
> >
> >
> >
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> > --
> > Ing. Jan Marek
> > University of South Bohemia
> > Academic Computer Centre
> > Phone: +420389032080
> > http://www.gnu.org/philosophy/no-word-attachments.cs.html
>
>
>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

[Jan Marek]

Hello again,

I'm sorry, I forgot attach file... :-(

Sincerely
Jan

Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
> Hello Sridhar,
> 
> at Saturday I've finished upgrade proces to 18.2.1.
> 
> Cluster is now in HEALTH_OK state and performs well.
> 
> According to my colleagues there are lower latences and good
> throughput.
> 
> On OSD nodes there is relative low I/O activity.
> 
> I still have mClock profile "high_client_ops".
> 
> When I was stucked in the upgrade process, I had in logs so many
> records, see attached file. Since upgrade is complete, this
> messages went away... Can be this reason of poor
> performance?
> 
> Sincerely
> Jan Marek
> 
> Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> > Hello Sridhar,
> > 
> > Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > > Hello Jan,
> > > 
> > > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > > my needs and, although I had set mClock profile to
> > > "high_client_ops" (because I have a plenty of time to rebalancing
> > > and scrubbing), my clients went to problems.
> > > 
> > > As far as the question around mClock is concerned, there are further
> > > improvements in the works to handle QoS between client ops and
> > > background scrub ops. This should help address the issue you are
> > > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > > for more information.
> > > Also, it would be helpful to know the Ceph version you are currently 
> > > using.
> > 
> > thanks for your reply.
> > 
> > I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> > see my previous posts about stuck in upgrade to reef).
> > 
> > Maybe this was cause of my problem...
> > 
> > Now I've tried give rest to the cluster to do some "background"
> > tasks (and it seems, that this was correct, because on my hosts
> > there is around 50-100MBps read and cca 10-50MBps write traffic -
> > cca 1/4-1/2 of previous load).
> > 
> > At Saturday I will change some settings on networking and I will
> > try to start upgrade process, maybe with --limit=1, to be "soft"
> > for cluster and for our clients...
> > 
> > > -Sridhar
> > 
> > Sincerely
> > Jan Marek
> > -- 
> > Ing. Jan Marek
> > University of South Bohemia
> > Academic Computer Centre
> > Phone: +420389032080
> > http://www.gnu.org/philosophy/no-word-attachments.cs.html
> 
> 
> 
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> -- 
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html



> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
2024-01-27T18:46:25.152544+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773745 with expected crc
2024-01-27T18:46:25.154758+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773745 with expected crc
2024-01-27T18:49:00.248561+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773746 with expected crc
2024-01-27T18:49:02.294944+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773748 with expected crc
2024-01-27T18:49:03.281543+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773749 with expected crc
2024-01-27T18:49:03.281894+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773749 with expected crc
2024-01-27T18:49:06.611229+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773752 with expected crc
2024-01-27T18:49:06.612924+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773752 with expected crc
2024-01-27T18:49:07.634638+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773753 with expected crc
2024-01-27T18:49:08.635094+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773754 with expected crc
2024-01-27T18:49:11.698700+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773757 with expected crc
2024-01-27T18:49:12.935850+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773758 with expected crc
2024-01-27T18:49:18.050429+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773761 with expected crc
2024-01-27T18:49:20.072799+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773763 with expected crc

[ceph-users] Re: Scrubbing?

[Jan Marek]

Hello Sridhar,

at Saturday I've finished upgrade proces to 18.2.1.

Cluster is now in HEALTH_OK state and performs well.

According to my colleagues there are lower latences and good
throughput.

On OSD nodes there is relative low I/O activity.

I still have mClock profile "high_client_ops".

When I was stucked in the upgrade process, I had in logs so many
records, see attached file. Since upgrade is complete, this
messages went away... Can be this reason of poor
performance?

Sincerely
Jan Marek

Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> Hello Sridhar,
> 
> Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > Hello Jan,
> > 
> > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > my needs and, although I had set mClock profile to
> > "high_client_ops" (because I have a plenty of time to rebalancing
> > and scrubbing), my clients went to problems.
> > 
> > As far as the question around mClock is concerned, there are further
> > improvements in the works to handle QoS between client ops and
> > background scrub ops. This should help address the issue you are
> > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > for more information.
> > Also, it would be helpful to know the Ceph version you are currently using.
> 
> thanks for your reply.
> 
> I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> see my previous posts about stuck in upgrade to reef).
> 
> Maybe this was cause of my problem...
> 
> Now I've tried give rest to the cluster to do some "background"
> tasks (and it seems, that this was correct, because on my hosts
> there is around 50-100MBps read and cca 10-50MBps write traffic -
> cca 1/4-1/2 of previous load).
> 
> At Saturday I will change some settings on networking and I will
> try to start upgrade process, maybe with --limit=1, to be "soft"
> for cluster and for our clients...
> 
> > -Sridhar
> 
> Sincerely
> Jan Marek
> -- 
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html



> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html


signature.asc
Description: PGP signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

[Jan Marek]

Hello Sridhar,

Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> Hello Jan,
> 
> Meaning of my previous post was, that CEPH cluster didn't fulfill
> my needs and, although I had set mClock profile to
> "high_client_ops" (because I have a plenty of time to rebalancing
> and scrubbing), my clients went to problems.
> 
> As far as the question around mClock is concerned, there are further
> improvements in the works to handle QoS between client ops and
> background scrub ops. This should help address the issue you are
> currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> for more information.
> Also, it would be helpful to know the Ceph version you are currently using.

thanks for your reply.

I've just in process upgrade between 17.2.6 and 18.2.1 (you can
see my previous posts about stuck in upgrade to reef).

Maybe this was cause of my problem...

Now I've tried give rest to the cluster to do some "background"
tasks (and it seems, that this was correct, because on my hosts
there is around 50-100MBps read and cca 10-50MBps write traffic -
cca 1/4-1/2 of previous load).

At Saturday I will change some settings on networking and I will
try to start upgrade process, maybe with --limit=1, to be "soft"
for cluster and for our clients...

> -Sridhar

Sincerely
Jan Marek
-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html


signature.asc
Description: PGP signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

[Sridhar Seshasayee]

Hello Jan,


> Meaning of my previous post was, that CEPH cluster didn't fulfill
> my needs and, although I had set mClock profile to
> "high_client_ops" (because I have a plenty of time to rebalancing
> and scrubbing), my clients went to problems.
>

As far as the question around mClock is concerned, there are further
improvements in the works to handle QoS between client ops and
background scrub ops. This should help address the issue you are
currently facing. See PR: https://github.com/ceph/ceph/pull/51171
for more information.

Also, it would be helpful to know the Ceph version you are currently using.

-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

[Jan Marek]

Hello Peter,

your irony is perfect, it is worth to notice.

Meaning of my previous post was, that CEPH cluster didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.

And there was question, if scheduler manage CEPH cluster
background (and clients) operation in this way to stil be usable
for clients.

I've tried to send feedback to developers.

Thanks for understanding.

Sincerely
Jan Marek

Dne St, led 24, 2024 at 11:18:20 CET napsal(a) Peter Grandi:
> > [...] After a few days, I have on our OSD nodes around 90MB/s
> > read and 70MB/s write while 'ceph -s' have client io as
> > 2,5MB/s read and 50MB/s write. [...]
> 
> This is one of my pet-peeves: that a storage system must have
> capacity (principally IOPS) to handle both a maintenance
> workload and a user workload, and since the former often
> involves whole-storage or whole-metadata operations it can be
> quite heavy, especially in the case of Ceph where rebalancing
> and scrubbing and checking should be fairly frequent to detect
> and correct inconsistencies.
> 
> > Is this activity OK? [...]
> 
> Indeed. Some "clever" people "save money" by "rightsizing" their
> storage so it cannot run at the same time the maintenance and
> the user workload, and so turn off the maintenance workload,
> because they "feel lucky" I guess, but I do not recommend that.
> :-). I have seen more than one Ceph cluster that did not have
> the capacity even to run *just* the maintenance workload.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html


signature.asc
Description: PGP signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

[Peter Grandi]

> [...] After a few days, I have on our OSD nodes around 90MB/s
> read and 70MB/s write while 'ceph -s' have client io as
> 2,5MB/s read and 50MB/s write. [...]

This is one of my pet-peeves: that a storage system must have
capacity (principally IOPS) to handle both a maintenance
workload and a user workload, and since the former often
involves whole-storage or whole-metadata operations it can be
quite heavy, especially in the case of Ceph where rebalancing
and scrubbing and checking should be fairly frequent to detect
and correct inconsistencies.

> Is this activity OK? [...]

Indeed. Some "clever" people "save money" by "rightsizing" their
storage so it cannot run at the same time the maintenance and
the user workload, and so turn off the maintenance workload,
because they "feel lucky" I guess, but I do not recommend that.
:-). I have seen more than one Ceph cluster that did not have
the capacity even to run *just* the maintenance workload.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: scrubbing+deep+repair PGs since Upgrade

[Marcus Müller]

Hi Stefan,

thanks for the fast reply. I did some research and have the following output:

~ $ rados list-inconsistent-pg {pool-name1}
[]
 ~ $ rados list-inconsistent-pg {pool-name2}
[]
 ~ $ rados list-inconsistent-pg {pool-name3}
[]

—

 ~ $ rados list-inconsistent-obj 7.989
{"epoch":3006349,"inconsistents":[]}

~ $ rados list-inconsistent-obj 7.28f
{"epoch":3006337,"inconsistents":[]}

 ~ $ rados list-inconsistent-obj 7.603
{"epoch":3006329,"inconsistents":[]}


 ~ $ ceph config dump |grep osd_scrub_auto_repair 

Is empty 

 $ ceph daemon mon.ceph4 config get osd_scrub_auto_repair
{
"osd_scrub_auto_repair": "true"
}

What does this tell me know? Setting can be changed to false of course, but as 
list-inconsistent-obj shows something, I would like to find the reason for that 
first.

Regards
Marcus


> Am 27.06.2022 um 08:56 schrieb Stefan Kooman :
> 
> On 6/27/22 08:48, Marcus Müller wrote:
>> Hi all,
>> we recently upgraded from Ceph Luminous (12.x) to Ceph Octopus (15.x) (of 
>> course with Mimic and Nautilus in between). Since this upgrade we see we 
>> constant number of active+clean+scrubbing+deep+repair PGs. We never had this 
>> in the past, now every time (like 10 or 20 PGs at the same time with the 
>> +repair flag).
>> Does anyone know how to debug this more in detail ?
> 
> ceph daemon mon.$mon-id config get osd_scrub_auto_repair
> 
> ^^ This is disabled by default (Octopus 15.2.16), but you might have this 
> setting changed to true?
> 
> ceph config dump |grep osd_scrub_auto_repair to check if it's a global 
> setting.
> 
> Do the following commands return any info?
> 
> rados list-inconsistent-pg
> rados list-inconsistent-obj
> 
> Gr. Stefan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

Thank you Dennis! We have made most of these changes and are waiting to see 
what happens. 

Thank you,
Ray 

-Original Message-
From: Denis Polom  
Sent: Saturday, March 12, 2022 1:40 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Scrubbing

Hi,

I had similar problem on my larce cluster.

What I found and helped me to solve it:

Due to bad drives and replacing drives too often due to scrub error there was 
always some recovery operations going on.

I did set this:

osd_scrub_during_recovery true

and it basically solved my issue.

If not then you can try change the interval.

I did it also from default once per week to two weeks:

osd_deep_scrub_interval 1209600

and if you want or need to speed it up to get rid of not scrubbed in time PGs 
take a look into

osd_max_scrubs

default is 1 and if I need to speed it up I set it to 3 and I didn't recognize 
any performance impact.


dp

On 3/11/22 17:32, Ray Cunningham wrote:
> That's what I thought. We looked at the cluster storage nodes and found them 
> all to be less than .2 normalized maximum load.
>
> Our 'normal' BW for client IO according to ceph -s is around 60MB/s-100MB/s. 
> I don't usually look at the IOPs so I don't have that number right now. We 
> have seen GB/s numbers during repairs, so the cluster can get up there when 
> the workload requires.
>
> We discovered that this system never got the auto repair setting configured 
> to true and since we turned that on, we have been repairing PGs for the past 
> 24 hours. So, maybe we've been bottlenecked by those?
>
> Thank you,
> Ray
>   
>
> -Original Message-
> From: norman.kern 
> Sent: Thursday, March 10, 2022 9:27
> To: Ray Cunningham 
> Cc: ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: Scrubbing
>
> Ray,
>
> You can use node-exporter+prom+grafana  to collect the load of CPUs 
> statistics. You can use uptime command to get the current statistics.
>
> On 3/10/22 10:51 PM, Ray Cunningham wrote:
>> From:
>>
>> osd_scrub_load_threshold
>> The normalized maximum load. Ceph will not scrub when the system load (as 
>> defined by getloadavg() / number of online CPUs) is higher than this number. 
>> Default is 0.5.
>>
>> Does anyone know how I can run getloadavg() / number of online CPUs so I can 
>> see what our load is? Is that a ceph command, or an OS command?
>>
>> Thank you,
>> Ray
>>
>>
>> -Original Message-
>> From: Ray Cunningham
>> Sent: Thursday, March 10, 2022 7:59 AM
>> To: norman.kern 
>> Cc: ceph-users@ceph.io
>> Subject: RE: [ceph-users] Scrubbing
>>
>>
>> We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so 
>> we are using bluestore. The system is running Nautilus 14.2.19 at the 
>> moment, with an upgrade scheduled this month. I can't give you a complete 
>> ceph config dump as this is an offline customer system, but I can get 
>> answers for specific questions.
>>
>> Off the top of my head, we have set:
>>
>> osd_max_scrubs 20
>> osd_scrub_auto_repair true
>> osd _scrub_load_threashold 0.6
>> We do not limit srub hours.
>>
>> Thank you,
>> Ray
>>
>>
>>
>>
>> -Original Message-
>> From: norman.kern 
>> Sent: Wednesday, March 9, 2022 7:28 PM
>> To: Ray Cunningham 
>> Cc: ceph-users@ceph.io
>> Subject: Re: [ceph-users] Scrubbing
>>
>> Ray,
>>
>> Can you  provide more information about your cluster(hardware and software 
>> configs)?
>>
>> On 3/10/22 7:40 AM, Ray Cunningham wrote:
>>> make any difference. Do
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
>> email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Denis Polom]

Hi,

I had similar problem on my larce cluster.

What I found and helped me to solve it:

Due to bad drives and replacing drives too often due to scrub error 
there was always some recovery operations going on.


I did set this:

osd_scrub_during_recovery true

and it basically solved my issue.

If not then you can try change the interval.

I did it also from default once per week to two weeks:

osd_deep_scrub_interval 1209600

and if you want or need to speed it up to get rid of not scrubbed in 
time PGs take a look into


osd_max_scrubs

default is 1 and if I need to speed it up I set it to 3 and I didn't 
recognize any performance impact.



dp

On 3/11/22 17:32, Ray Cunningham wrote:

That's what I thought. We looked at the cluster storage nodes and found them 
all to be less than .2 normalized maximum load.

Our 'normal' BW for client IO according to ceph -s is around 60MB/s-100MB/s. I 
don't usually look at the IOPs so I don't have that number right now. We have 
seen GB/s numbers during repairs, so the cluster can get up there when the 
workload requires.

We discovered that this system never got the auto repair setting configured to 
true and since we turned that on, we have been repairing PGs for the past 24 
hours. So, maybe we've been bottlenecked by those?

Thank you,
Ray
  


-Original Message-
From: norman.kern 
Sent: Thursday, March 10, 2022 9:27
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Re: Scrubbing

Ray,

You can use node-exporter+prom+grafana  to collect the load of CPUs statistics. 
You can use uptime command to get the current statistics.

On 3/10/22 10:51 PM, Ray Cunningham wrote:

From:

osd_scrub_load_threshold
The normalized maximum load. Ceph will not scrub when the system load (as 
defined by getloadavg() / number of online CPUs) is higher than this number. 
Default is 0.5.

Does anyone know how I can run getloadavg() / number of online CPUs so I can 
see what our load is? Is that a ceph command, or an OS command?

Thank you,
Ray


-Original Message-
From: Ray Cunningham
Sent: Thursday, March 10, 2022 7:59 AM
To: norman.kern 
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing


We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
are using bluestore. The system is running Nautilus 14.2.19 at the moment, with 
an upgrade scheduled this month. I can't give you a complete ceph config dump 
as this is an offline customer system, but I can get answers for specific 
questions.

Off the top of my head, we have set:

osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours.

Thank you,
Ray




-Original Message-
From: norman.kern 
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Ray,

Can you  provide more information about your cluster(hardware and software 
configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:

make any difference. Do

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

That's what I thought. We looked at the cluster storage nodes and found them 
all to be less than .2 normalized maximum load. 

Our 'normal' BW for client IO according to ceph -s is around 60MB/s-100MB/s. I 
don't usually look at the IOPs so I don't have that number right now. We have 
seen GB/s numbers during repairs, so the cluster can get up there when the 
workload requires. 

We discovered that this system never got the auto repair setting configured to 
true and since we turned that on, we have been repairing PGs for the past 24 
hours. So, maybe we've been bottlenecked by those? 

Thank you,
Ray 
 

-Original Message-
From: norman.kern  
Sent: Thursday, March 10, 2022 9:27 
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Re: Scrubbing

Ray,

You can use node-exporter+prom+grafana  to collect the load of CPUs statistics. 
You can use uptime command to get the current statistics.

On 3/10/22 10:51 PM, Ray Cunningham wrote:
> From:
>
> osd_scrub_load_threshold
> The normalized maximum load. Ceph will not scrub when the system load (as 
> defined by getloadavg() / number of online CPUs) is higher than this number. 
> Default is 0.5.
>
> Does anyone know how I can run getloadavg() / number of online CPUs so I can 
> see what our load is? Is that a ceph command, or an OS command?
>
> Thank you,
> Ray
>
>
> -Original Message-
> From: Ray Cunningham
> Sent: Thursday, March 10, 2022 7:59 AM
> To: norman.kern 
> Cc: ceph-users@ceph.io
> Subject: RE: [ceph-users] Scrubbing
>
>
> We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
> are using bluestore. The system is running Nautilus 14.2.19 at the moment, 
> with an upgrade scheduled this month. I can't give you a complete ceph config 
> dump as this is an offline customer system, but I can get answers for 
> specific questions.
>
> Off the top of my head, we have set:
>
> osd_max_scrubs 20
> osd_scrub_auto_repair true
> osd _scrub_load_threashold 0.6
> We do not limit srub hours.
>
> Thank you,
> Ray
>
>
>
>
> -Original Message-
> From: norman.kern 
> Sent: Wednesday, March 9, 2022 7:28 PM
> To: Ray Cunningham 
> Cc: ceph-users@ceph.io
> Subject: Re: [ceph-users] Scrubbing
>
> Ray,
>
> Can you  provide more information about your cluster(hardware and software 
> configs)?
>
> On 3/10/22 7:40 AM, Ray Cunningham wrote:
>>make any difference. Do
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[norman.kern]

Ray,

Do you known the IOPS/BW of the cluster?  The 16TB HDD is more suitable
for cold data, If the clients' bw/iops is too big, you can never  finish
the scrub.

And if you adjust the priority, it will have a great impact to the clients.

On 3/10/22 9:59 PM, Ray Cunningham wrote:

We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
are using bluestore. The system is running Nautilus 14.2.19 at the moment, with 
an upgrade scheduled this month. I can't give you a complete ceph config dump 
as this is an offline customer system, but I can get answers for specific 
questions.

Off the top of my head, we have set:

osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours.

Thank you,
Ray




-Original Message-
From: norman.kern 
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Ray,

Can you  provide more information about your cluster(hardware and software 
configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:

   make any difference. Do

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[norman.kern]

Ray,

You can use node-exporter+prom+grafana  to collect the load of CPUs
statistics. You can use uptime command to get the current statistics.

On 3/10/22 10:51 PM, Ray Cunningham wrote:

From:

osd_scrub_load_threshold
The normalized maximum load. Ceph will not scrub when the system load (as 
defined by getloadavg() / number of online CPUs) is higher than this number. 
Default is 0.5.

Does anyone know how I can run getloadavg() / number of online CPUs so I can 
see what our load is? Is that a ceph command, or an OS command?

Thank you,
Ray


-Original Message-
From: Ray Cunningham
Sent: Thursday, March 10, 2022 7:59 AM
To: norman.kern 
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing


We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
are using bluestore. The system is running Nautilus 14.2.19 at the moment, with 
an upgrade scheduled this month. I can't give you a complete ceph config dump 
as this is an offline customer system, but I can get answers for specific 
questions.

Off the top of my head, we have set:

osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours.

Thank you,
Ray




-Original Message-
From: norman.kern 
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Ray,

Can you  provide more information about your cluster(hardware and software 
configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:

   make any difference. Do

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

Well that was incorrect. Someone changed it back to 1. I have now set our max 
scrubs to 2. We’ll see if that makes a difference.

Thank you,
Ray

From: Ray Cunningham
Sent: Thursday, March 10, 2022 8:00 AM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing

We have that set to 20 at the moment.

Thank you,
Ray

From: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>
Sent: Wednesday, March 9, 2022 7:35 PM
To: Ray Cunningham 
mailto:ray.cunning...@keepertech.com>>
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Have you tried to increase osd max scrubs to 2?
Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2022. Mar 10., at 6:42, Ray Cunningham 
mailto:ray.cunning...@keepertech.com>> wrote:
Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


Hi Everyone,

We have a 900 OSD cluster and our pg scrubs aren't keeping up. We are always 
behind and have tried to tweak some of the scrub config settings to allow a 
higher priority and faster scrubbing, but it doesn't seem to make any 
difference. Does anyone have any suggestions for increasing scrub throughput?

Thank you,
Ray Cunningham

Systems Engineering and Services Manager
keepertechnology
(571) 223-7242


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

From: 

osd_scrub_load_threshold
The normalized maximum load. Ceph will not scrub when the system load (as 
defined by getloadavg() / number of online CPUs) is higher than this number. 
Default is 0.5.

Does anyone know how I can run getloadavg() / number of online CPUs so I can 
see what our load is? Is that a ceph command, or an OS command? 

Thank you,
Ray 
 

-Original Message-
From: Ray Cunningham 
Sent: Thursday, March 10, 2022 7:59 AM
To: norman.kern 
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing


We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
are using bluestore. The system is running Nautilus 14.2.19 at the moment, with 
an upgrade scheduled this month. I can't give you a complete ceph config dump 
as this is an offline customer system, but I can get answers for specific 
questions. 

Off the top of my head, we have set:

osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours. 

Thank you,
Ray 
 

 

-Original Message-
From: norman.kern  
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Ray,

Can you  provide more information about your cluster(hardware and software 
configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:
>   make any difference. Do
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

We have that set to 20 at the moment.

Thank you,
Ray Cunningham

Systems Engineering and Services Manager
keepertechnology
(571) 223-7242


From: Szabo, Istvan (Agoda) 
Sent: Wednesday, March 9, 2022 7:35 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Have you tried to increase osd max scrubs to 2?
Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---


On 2022. Mar 10., at 6:42, Ray Cunningham 
mailto:ray.cunning...@keepertech.com>> wrote:
Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


Hi Everyone,

We have a 900 OSD cluster and our pg scrubs aren't keeping up. We are always 
behind and have tried to tweak some of the scrub config settings to allow a 
higher priority and faster scrubbing, but it doesn't seem to make any 
difference. Does anyone have any suggestions for increasing scrub throughput?

Thank you,
Ray Cunningham

Systems Engineering and Services Manager
keepertechnology
(571) 223-7242


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[Ray Cunningham]

We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we 
are using bluestore. The system is running Nautilus 14.2.19 at the moment, with 
an upgrade scheduled this month. I can't give you a complete ceph config dump 
as this is an offline customer system, but I can get answers for specific 
questions. 

Off the top of my head, we have set:

osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours. 

Thank you,
Ray 
 

 

-Original Message-
From: norman.kern  
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunningham 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Scrubbing

Ray,

Can you  provide more information about your cluster(hardware and software 
configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:
>   make any difference. Do
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing

[norman.kern]

Ray,

Can you  provide more information about your cluster(hardware and
software configs)?

On 3/10/22 7:40 AM, Ray Cunningham wrote:

  make any difference. Do

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing - osd down

[Igor Fedotov]

Miroslav,

On 12/11/2020 4:57 PM, Miroslav Boháč wrote:

Hi Igor,

thank you. Yes you are right.

It seems that the background removal is completed.
you can inspect "numpg_removing" performance counter to make sure it's 
been completed.


The correct way to fix it is "ceph-kvstore-tool bluestore-kv 
 compact" to all OSD (one by one)?


Right.

You can want to try online compaction: ceph tell osd.X compact

I'm still not sure the latter is effective in fixing the issue though. 
Will appreciate if you share your experience on that.



Regards,
Miroslav

pá 11. 12. 2020 v 14:19 odesílatel Igor Fedotov > napsal:


Hi Miroslav,

haven't you performed massive data removal (PG migration) recently?

If so you might want to apply manual DB compaction to your OSDs.

The positive effect might be just temporary if background removals
are
still in progress though..


See https://tracker.ceph.com/issues/47044
 and the references for more
info on the issue.


Thanks,

Igor




On 12/11/2020 12:50 PM, Miroslav Boháč wrote:
> Hi,
>
> I have a problem with crashing OSD daemons in our Ceph 15.2.6
cluster . The
> problem was temporarily resolved by disabling scrub and
deep-scrub. All PGs
> are active+clean. After a few days I tried to enable scrubbing
again, but
> the problem persists. OSD with high latencies, PG laggy, osd not
> responding, OSD was marked down and the cluster is not usable.
Problem
> appeared after failover when 10 OSD was marked as out.
>
> I will appreciate any help or advice, how to resolve this problem.
>
> Regards,
> Miroslav
>
>
>
>
>
> 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG]
18.11
> deep-scrub starts
>
> 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG]
18.11
> deep-scrub ok
>
> 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG]
43.5 scrub
> starts
>
> 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG]
43.5 scrub
> ok
>
> 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377)
118746 :
> cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s
>
> 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster
[DBG] 1.13b
> deep-scrub starts
>
> 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377)
118747 :
> cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s
>
> 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster
[DBG] 1.a2
> deep-scrub ok
>
> 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377)
118749 :
> cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s
>
> 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377)
118750 :
> cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s
>
> 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377)
118751 :
> cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s
>
>
>
>
>
> 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 :
cluster
> [DBG] osd.4 reported failed by osd.24
>
> 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 :
cluster
> [DBG] osd.39 reported failed by osd.24
>
> 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 :
cluster
> [WRN] Health check failed: 0 slow ops, oldest one blocked for 32
sec, osd.8
> has slow ops (SLOW_OPS)
>
> 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 :
cluster
> [DBG] osd.4 reported failed by osd.34
>
> 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 :
cluster
> [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
> different host after 24.000267 >= grace 22.361677)
>
> 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 :
cluster
> [DBG] osd.39 reported failed by osd.6
>
> 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 :

[ceph-users] Re: Scrubbing - osd down

[Miroslav Boháč]

Hi Igor,

thank you. Yes you are right.

It seems that the background removal is completed.

The correct way to fix it is "ceph-kvstore-tool bluestore-kv 
compact" to all OSD (one by one)?

Regards,
Miroslav

pá 11. 12. 2020 v 14:19 odesílatel Igor Fedotov  napsal:

> Hi Miroslav,
>
> haven't you performed massive data removal (PG migration) recently?
>
> If so you might want to apply manual DB compaction to your OSDs.
>
> The positive effect might be just temporary if background removals are
> still in progress though..
>
>
> See https://tracker.ceph.com/issues/47044 and the references for more
> info on the issue.
>
>
> Thanks,
>
> Igor
>
>
>
>
> On 12/11/2020 12:50 PM, Miroslav Boháč wrote:
> > Hi,
> >
> > I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster .
> The
> > problem was temporarily resolved by disabling scrub and deep-scrub. All
> PGs
> > are active+clean. After a few days I tried to enable scrubbing again, but
> > the problem persists. OSD with high latencies, PG laggy, osd not
> > responding, OSD was marked down and the cluster is not usable. Problem
> > appeared after failover when 10 OSD was marked as out.
> >
> > I will appreciate any help or advice, how to resolve this problem.
> >
> > Regards,
> > Miroslav
> >
> >
> >
> >
> >
> > 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11
> > deep-scrub starts
> >
> > 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11
> > deep-scrub ok
> >
> > 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5
> scrub
> > starts
> >
> > 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5
> scrub
> > ok
> >
> > 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 :
> > cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s
> >
> > 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b
> > deep-scrub starts
> >
> > 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 :
> > cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s
> >
> > 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2
> > deep-scrub ok
> >
> > 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 :
> > cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s
> >
> > 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 :
> > cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s
> >
> > 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 :
> > cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s
> >
> >
> >
> >
> >
> > 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster
> > [DBG] osd.4 reported failed by osd.24
> >
> > 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster
> > [DBG] osd.39 reported failed by osd.24
> >
> > 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster
> > [WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec,
> osd.8
> > has slow ops (SLOW_OPS)
> >
> > 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster
> > [DBG] osd.4 reported failed by osd.34
> >
> > 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster
> > [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
> > different host after 24.000267 >= grace 22.361677)
> >
> > 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster
> > [DBG] osd.39 reported failed by osd.6
> >
> > 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster
> > [DBG] osd.39 reported failed by osd.8
> >
> > 2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster
> > [WRN] Health check failed: 1 osds down (OSD_DOWN)
> >
> > 2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster
> > [DBG] osdmap e12819: 40 total, 39 up, 30 in
> >
> > 2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster
> > [DBG] osd.39 reported failed by osd.21
> >
> > 2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 :
> > cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean,
> 1
> > active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3
> > TiB data, 7.8 TiB 

[ceph-users] Re: Scrubbing - osd down

[Igor Fedotov]

Hi Miroslav,

haven't you performed massive data removal (PG migration) recently?

If so you might want to apply manual DB compaction to your OSDs.

The positive effect might be just temporary if background removals are 
still in progress though..



See https://tracker.ceph.com/issues/47044 and the references for more 
info on the issue.



Thanks,

Igor




On 12/11/2020 12:50 PM, Miroslav Boháč wrote:

Hi,

I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster . The
problem was temporarily resolved by disabling scrub and deep-scrub. All PGs
are active+clean. After a few days I tried to enable scrubbing again, but
the problem persists. OSD with high latencies, PG laggy, osd not
responding, OSD was marked down and the cluster is not usable. Problem
appeared after failover when 10 OSD was marked as out.

I will appreciate any help or advice, how to resolve this problem.

Regards,
Miroslav





2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11
deep-scrub starts

2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11
deep-scrub ok

2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5 scrub
starts

2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5 scrub
ok

2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 :
cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s

2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b
deep-scrub starts

2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 :
cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s

2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2
deep-scrub ok

2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 :
cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s

2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 :
cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s

2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 :
cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s





2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster
[DBG] osd.4 reported failed by osd.24

2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster
[DBG] osd.39 reported failed by osd.24

2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster
[WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.8
has slow ops (SLOW_OPS)

2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster
[DBG] osd.4 reported failed by osd.34

2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster
[INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
different host after 24.000267 >= grace 22.361677)

2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster
[DBG] osd.39 reported failed by osd.6

2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster
[DBG] osd.39 reported failed by osd.8

2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster
[WRN] Health check failed: 1 osds down (OSD_DOWN)

2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster
[DBG] osdmap e12819: 40 total, 39 up, 30 in

2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster
[DBG] osd.39 reported failed by osd.21

2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 :
cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean, 1
active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3
TiB data, 7.8 TiB used, 14 TiB / 22 TiB avail; 44 KiB/s rd, 2.5 MiB/s wr,
107 op/s

2020-12-10T21:40:17.378069+0100 mon.pve1-prg2a (mon.0) 125955 : cluster
[DBG] osd.39 reported failed by osd.26

2020-12-10T21:40:17.424429+0100 mon.pve1-prg2a (mon.0) 125956 : cluster
[DBG] osd.39 reported failed by osd.18

2020-12-10T21:40:17.829447+0100 mon.pve1-prg2a (mon.0) 125957 : cluster
[DBG] osd.39 reported failed by osd.36

2020-12-10T21:40:17.847373+0100 mon.pve1-prg2a (mon.0) 125958 : cluster
[DBG] osd.39 reported failed by osd.1

2020-12-10T21:40:17.858371+0100 mon.pve1-prg2a (mon.0) 125959 : cluster
[DBG] osd.39 reported failed by osd.17

2020-12-10T21:40:17.915755+0100 mon.pve1-prg2a (mon.0) 125960 :