Re: [ceph-users] pgs not deep-scrubbed for 86400

2017-07-25 Thread Roger Brown 
Got it, thanks!

On Tue, Jul 25, 2017 at 8:27 AM Малков Петр Викторович 
wrote:

> Hello, Roger!
>
>
>
> ceph pg dump
>
> ceph pg deep-scrub 
>
>
>
> or just kick them all
>
>
>
> ceph pg dump | grep -i active+clean | awk '{print $1}' | while read i; do
> ceph pg deep-scrub ${i}; done
>
>
>
> --
>
> Petr Malkov
>
>
>
> -
>
> Message: 57
>
> Date: Wed, 19 Jul 2017 16:38:20 +
>
> From: Roger Brown 
>
> To: ceph-users 
>
> Subject: [ceph-users] pgs not deep-scrubbed for 86400
>
> Message-ID:
>
><
> cafs+eljefq5zgctdd+cexhh0dkqby+mq0d3t2esdblsucph...@mail.gmail.com>
>
> Content-Type: text/plain; charset="utf-8"
>
>
>
> I just upgraded from Luminous 12.1.0 to 12.1.1 and was greeted with this
>
> new "pgs not deep-scrubbed for" warning. Should this resolve itself, or
>
> should I get scrubbing?
>
>
>
> $ ceph health detail
>
> HEALTH_WARN 4 pgs not deep-scrubbed for 86400; 15 pgs not scrubbed for
> 86400
>
> PG_NOT_DEEP_SCRUBBED 4 pgs not deep-scrubbed for 86400
>
> pg 85.0 not deep-scrubbed since 2017-07-12 09:08:04.611006
>
> pg 85.6 not deep-scrubbed since 2017-07-12 07:06:09.706367
>
> pg 91.0 not deep-scrubbed since 2017-07-12 06:49:35.241460
>
> pg 100.2 not deep-scrubbed since 2017-07-11 22:50:05.534546
>
> PG_NOT_SCRUBBED 15 pgs not scrubbed for 86400
>
> pg 101.7 not scrubbed since 2017-07-18 08:24:03.332885
>
> pg 97.1 not scrubbed since 2017-07-18 06:24:20.317393
>
> pg 96.0 not scrubbed since 2017-07-18 04:32:11.542037
>
> pg 98.2 not scrubbed since 2017-07-18 02:19:48.638410
>
> pg 85.0 not scrubbed since 2017-07-18 03:12:32.014656
>
> pg 83.3 not scrubbed since 2017-07-18 07:55:48.96
>
> pg 83.4 not scrubbed since 2017-07-18 10:14:33.595142
>
> pg 91.2 not scrubbed since 2017-07-18 06:35:24.338351
>
> pg 90.0 not scrubbed since 2017-07-18 05:48:33.604003
>
> pg 89.3 not scrubbed since 2017-07-18 08:21:03.806888
>
> pg 92.7 not scrubbed since 2017-07-18 09:20:27.170389
>
> pg 91.7 not scrubbed since 2017-07-18 10:22:41.240209
>
> pg 93.2 not scrubbed since 2017-07-18 09:24:26.186475
>
> pg 89.6 not scrubbed since 2017-07-18 05:37:45.016260
>
> pg 100.3 not scrubbed since 2017-07-18 09:53:10.459155
>
> -- next part --
>
> An HTML attachment was scrubbed...
>
> URL: <
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20170719/12ed3df3/attachment-0001.htm
> >
>
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pgs not deep-scrubbed for 86400

2017-07-25 Thread Малков Петр Викторович 
Hello, Roger!

ceph pg dump
ceph pg deep-scrub 

or just kick them all

ceph pg dump | grep -i active+clean | awk '{print $1}' | while read i; do ceph 
pg deep-scrub ${i}; done

--
Petr Malkov

-

Message: 57

Date: Wed, 19 Jul 2017 16:38:20 +

From: Roger Brown 

To: ceph-users 

Subject: [ceph-users] pgs not deep-scrubbed for 86400

Message-ID:

   


Content-Type: text/plain; charset="utf-8"



I just upgraded from Luminous 12.1.0 to 12.1.1 and was greeted with this

new "pgs not deep-scrubbed for" warning. Should this resolve itself, or

should I get scrubbing?



$ ceph health detail

HEALTH_WARN 4 pgs not deep-scrubbed for 86400; 15 pgs not scrubbed for 86400

PG_NOT_DEEP_SCRUBBED 4 pgs not deep-scrubbed for 86400

pg 85.0 not deep-scrubbed since 2017-07-12 09:08:04.611006

pg 85.6 not deep-scrubbed since 2017-07-12 07:06:09.706367

pg 91.0 not deep-scrubbed since 2017-07-12 06:49:35.241460

pg 100.2 not deep-scrubbed since 2017-07-11 22:50:05.534546

PG_NOT_SCRUBBED 15 pgs not scrubbed for 86400

pg 101.7 not scrubbed since 2017-07-18 08:24:03.332885

pg 97.1 not scrubbed since 2017-07-18 06:24:20.317393

pg 96.0 not scrubbed since 2017-07-18 04:32:11.542037

pg 98.2 not scrubbed since 2017-07-18 02:19:48.638410

pg 85.0 not scrubbed since 2017-07-18 03:12:32.014656

pg 83.3 not scrubbed since 2017-07-18 07:55:48.96

pg 83.4 not scrubbed since 2017-07-18 10:14:33.595142

pg 91.2 not scrubbed since 2017-07-18 06:35:24.338351

pg 90.0 not scrubbed since 2017-07-18 05:48:33.604003

pg 89.3 not scrubbed since 2017-07-18 08:21:03.806888

pg 92.7 not scrubbed since 2017-07-18 09:20:27.170389

pg 91.7 not scrubbed since 2017-07-18 10:22:41.240209

pg 93.2 not scrubbed since 2017-07-18 09:24:26.186475

pg 89.6 not scrubbed since 2017-07-18 05:37:45.016260

pg 100.3 not scrubbed since 2017-07-18 09:53:10.459155

-- next part --

An HTML attachment was scrubbed...

URL: 




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pgs not deep-scrubbed for 86400

2017-07-19 Thread Brad Hubbard 
This code shows how that all works (part of some new health reporting code).

https://github.com/ceph/ceph/blob/master/src/mon/PGMap.cc#L3188-L3203

So the last_deep_scrub_stamp of the pg is compared to deep_cutoff which is the
time now minus mon_warn_not_deep_scrubbed + osd_deep_scrub_interval.

By default mon_warn_not_deep_scrubbed = 0
OPTION(mon_warn_not_deep_scrubbed, OPT_INT, 0)

By default osd_deep_scrub_interval = 1 week
OPTION(osd_deep_scrub_interval, OPT_FLOAT, 60*60*24*7) // once a week

So by default if a pg has not been deep scrubbed in more than a week you will
get this warning. I believe the value reported, 86400, is incorrect in the case
of a deep scrub as it should be 604800 and this looks like a copy and paste
error. I'll submit a patch to rectify that.

The calculations for scrub (as opposed to deep scrub) are very similar except
the value involved is 86400, or one day (mon_scrub_interval).

I'll leave it as an exercise for you to do the math in your cases using the
values you have set locally.

HTH.

On Thu, Jul 20, 2017 at 5:10 AM, Gencer W. Genç  wrote:
> Exactly have this issue (or not?) at the moment. Mine says “906 pgs not
> scrubbed for 86400”. But it is decrementing slowly (very slowly).
>
>  
>
> I cannot find any documentation for exact “pgs not srubbed for” phrase on
> the web but only this.
>
>  
>
> Log like this:
>
>  
>
> 2017-07-19 15:05:10.125041 [INF]  3.5e scrub ok
>
> 2017-07-19 15:05:10.123522 [INF]  3.5e scrub starts
>
> 2017-07-19 15:05:14.613124 [WRN]  Health check update: 914 pgs not scrubbed
> for 86400 (PG_NOT_SCRUBBED)
>
> 2017-07-19 15:05:07.433748 [INF]  1.c4 scrub ok
>
> ...
>
>  
>
> Should this be scared us?
>
>  
>
> Gencer.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pgs not deep-scrubbed for 86400

2017-07-19 Thread Gencer W . Genç 
Exactly have this issue (or not?) at the moment. Mine says "906 pgs not
scrubbed for 86400". But it is decrementing slowly (very slowly).

 

I cannot find any documentation for exact "pgs not srubbed for" phrase on
the web but only this.

 

Log like this:

 

2017-07-19 15:05:10.125041 [INF]  3.5e scrub ok 

2017-07-19 15:05:10.123522 [INF]  3.5e scrub starts 

2017-07-19 15:05:14.613124 [WRN]  Health check update: 914 pgs not scrubbed
for 86400 (PG_NOT_SCRUBBED) 

2017-07-19 15:05:07.433748 [INF]  1.c4 scrub ok

...

 

Should this be scared us?

 

Gencer.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com