[ceph-users] Re: Scrubbing?

Ah, yeah, you hit https://tracker.ceph.com/issues/63389 during the upgrade. Josh On Tue, Jan 30, 2024 at 3:17 AM Jan Marek wrote: > > Hello again, > > I'm sorry, I forgot attach file... :-( > > Sincerely > Jan > > Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek: > > Hello Sridhar, > >

[ceph-users] Re: Scrubbing?

Hello again, I'm sorry, I forgot attach file... :-( Sincerely Jan Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek: > Hello Sridhar, > > at Saturday I've finished upgrade proces to 18.2.1. > > Cluster is now in HEALTH_OK state and performs well. > > According to my colleagues there

[ceph-users] Re: Scrubbing?

Hello Sridhar, at Saturday I've finished upgrade proces to 18.2.1. Cluster is now in HEALTH_OK state and performs well. According to my colleagues there are lower latences and good throughput. On OSD nodes there is relative low I/O activity. I still have mClock profile "high_client_ops".

[ceph-users] Re: Scrubbing?

Hello Sridhar, Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee: > Hello Jan, > > Meaning of my previous post was, that CEPH cluster didn't fulfill > my needs and, although I had set mClock profile to > "high_client_ops" (because I have a plenty of time to rebalancing > and

[ceph-users] Re: Scrubbing?

Hello Jan, > Meaning of my previous post was, that CEPH cluster didn't fulfill > my needs and, although I had set mClock profile to > "high_client_ops" (because I have a plenty of time to rebalancing > and scrubbing), my clients went to problems. > As far as the question around mClock is

[ceph-users] Re: Scrubbing?

Hello Peter, your irony is perfect, it is worth to notice. Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems. And

[ceph-users] Re: Scrubbing?

> [...] After a few days, I have on our OSD nodes around 90MB/s > read and 70MB/s write while 'ceph -s' have client io as > 2,5MB/s read and 50MB/s write. [...] This is one of my pet-peeves: that a storage system must have capacity (principally IOPS) to handle both a maintenance workload and a

[ceph-users] Re: scrubbing+deep+repair PGs since Upgrade

Hi Stefan, thanks for the fast reply. I did some research and have the following output: ~ $ rados list-inconsistent-pg {pool-name1} [] ~ $ rados list-inconsistent-pg {pool-name2} [] ~ $ rados list-inconsistent-pg {pool-name3} [] — ~ $ rados list-inconsistent-obj 7.989

[ceph-users] Re: Scrubbing

Thank you Dennis! We have made most of these changes and are waiting to see what happens. Thank you, Ray -Original Message- From: Denis Polom Sent: Saturday, March 12, 2022 1:40 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Scrubbing Hi, I had similar problem on my larce

[ceph-users] Re: Scrubbing

have been repairing PGs for the past 24 hours. So, maybe we've been bottlenecked by those? Thank you, Ray -Original Message- From: norman.kern Sent: Thursday, March 10, 2022 9:27 To: Ray Cunningham Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Scrubbing Ray, You can use

[ceph-users] Re: Scrubbing

? Thank you, Ray   -Original Message- From: norman.kern Sent: Thursday, March 10, 2022 9:27 To: Ray Cunningham Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Scrubbing Ray, You can use node-exporter+prom+grafana  to collect the load of CPUs statistics. You can use uptime command

[ceph-users] Re: Scrubbing

Ray, Do you known the IOPS/BW of the cluster?  The 16TB HDD is more suitable for cold data, If the clients' bw/iops is too big, you can never  finish the scrub. And if you adjust the priority, it will have a great impact to the clients. On 3/10/22 9:59 PM, Ray Cunningham wrote: We have 16

[ceph-users] Re: Scrubbing

Ray, You can use node-exporter+prom+grafana  to collect the load of CPUs statistics. You can use uptime command to get the current statistics. On 3/10/22 10:51 PM, Ray Cunningham wrote: From: osd_scrub_load_threshold The normalized maximum load. Ceph will not scrub when the system load (as

[ceph-users] Re: Scrubbing

Well that was incorrect. Someone changed it back to 1. I have now set our max scrubs to 2. We’ll see if that makes a difference. Thank you, Ray From: Ray Cunningham Sent: Thursday, March 10, 2022 8:00 AM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Scrubbing We

[ceph-users] Re: Scrubbing

From: osd_scrub_load_threshold The normalized maximum load. Ceph will not scrub when the system load (as defined by getloadavg() / number of online CPUs) is higher than this number. Default is 0.5. Does anyone know how I can run getloadavg() / number of online CPUs so I can see what our

[ceph-users] Re: Scrubbing

We have that set to 20 at the moment. Thank you, Ray Cunningham Systems Engineering and Services Manager keepertechnology (571) 223-7242 From: Szabo, Istvan (Agoda) Sent: Wednesday, March 9, 2022 7:35 PM To: Ray Cunningham Cc: ceph-users@ceph.io Subject: Re:

[ceph-users] Re: Scrubbing

We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we are using bluestore. The system is running Nautilus 14.2.19 at the moment, with an upgrade scheduled this month. I can't give you a complete ceph config dump as this is an offline customer system, but I can get

[ceph-users] Re: Scrubbing

Ray, Can you  provide more information about your cluster(hardware and software configs)? On 3/10/22 7:40 AM, Ray Cunningham wrote: make any difference. Do ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to

[ceph-users] Re: Scrubbing - osd down

Miroslav, On 12/11/2020 4:57 PM, Miroslav Boháč wrote: Hi Igor, thank you. Yes you are right. It seems that the background removal is completed. you can inspect "numpg_removing" performance counter to make sure it's been completed. The correct way to fix it is "ceph-kvstore-tool

[ceph-users] Re: Scrubbing - osd down

Hi Igor, thank you. Yes you are right. It seems that the background removal is completed. The correct way to fix it is "ceph-kvstore-tool bluestore-kv compact" to all OSD (one by one)? Regards, Miroslav pá 11. 12. 2020 v 14:19 odesílatel Igor Fedotov napsal: > Hi Miroslav, > > haven't you

[ceph-users] Re: Scrubbing - osd down

Hi Miroslav, haven't you performed massive data removal (PG migration) recently? If so you might want to apply manual DB compaction to your OSDs. The positive effect might be just temporary if background removals are still in progress though.. See https://tracker.ceph.com/issues/47044 and