[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

 22  up  osd.4
>>> > >>>>>  7hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2
>>> MiB
>>> > >>>>> 5.1 GiB  5.5 TiB  39.02  1.12   30  up  osd.7
>>> > >>>>> 11hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10
>>> MiB
>>> > >>>>> 5.1 GiB  5.4 TiB  39.97  1.15   27  up  osd.11
>>> > >>>>> 14hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.7 TiB   10
>>> MiB
>>> > >>>>> 5.1 GiB  5.6 TiB  38.24  1.10   27  up  osd.14
>>> > >>>>> 17hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4
>>> MiB
>>> > >>>>> 4.1 GiB  6.0 TiB  33.09  0.95   23  up  osd.17
>>> > >>>>> 20hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.6
>>> MiB
>>> > >>>>> 3.8 GiB  6.2 TiB  31.55  0.90   20      up  osd.20
>>> > >>>>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0
>>> MiB
>>> > >>>>> 3.3 GiB  6.5 TiB  28.32  0.81   23  up  osd.23
>>> > >>>>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8
>>> MiB
>>> > >>>>> 3.8 GiB  6.1 TiB  32.12  0.92   26  up  osd.26
>>> > >>>>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10
>>> MiB
>>> > >>>>> 5.1 GiB  5.4 TiB  39.73  1.14   24  up  osd.29
>>> > >>>>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8
>>> MiB
>>> > >>>>> 3.7 GiB  6.2 TiB  31.56  0.91   22  up  osd.31
>>> > >>>>> 34hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.2
>>> MiB
>>> > >>>>> 4.6 GiB  5.7 TiB  36.29  1.04   23  up  osd.34
>>> > >>>>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2
>>> MiB
>>> > >>>>> 4.5 GiB  5.8 TiB  35.51  1.02   20  up  osd.37
>>> > >>>>> 40hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  9.3
>>> MiB
>>> > >>>>> 4.9 GiB  5.6 TiB  38.16  1.09   25  up  osd.40
>>> > >>>>> 43hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.6 TiB  8.5
>>> MiB
>>> > >>>>> 4.8 GiB  5.7 TiB  37.19  1.07   29  up  osd.43
>>> > >>>>> 46hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4
>>> MiB
>>> > >>>>> 4.4 GiB  5.9 TiB  34.85  1.00   23  up  osd.46
>>> > >>>>>  TOTAL  433 TiB  151 TiB67 TiB  364
>>> MiB
>>> > >>>>> 210 GiB  282 TiB  34.86
>>> > >>>>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> Michel
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <
>>> > >>>>> w...@wesdillingham.com> wrote:
>>> > >>>>>
>>> > >>>>>> I now concur you should increase the pg_num as a first step for
>>> this
>>> > >>>>>> cluster. Disable the pg autoscaler for and increase the volumes
>>> pool to
>>> > >>>>>> pg_num 256. Then likely re-asses and make the next power of 2
>>> jump to 512
>>> > >>>>>> and probably beyond.
>>> > >>>>>>
>>> > >>>>>> Keep in mind this is not going to fix your short term deep-scrub
>>> > >>>>>> issue in fact it will increase the number of not scrubbed in
>>> time PGs until
>>> > >>>>>> the pg_num change is complete.  This is because OSDs dont scrub
>>> when they
>>> > >>>>>> are backfilling.
>>> > >>>>>>
>>> > >>>>>> I would sit on 256 for a couple weeks and let scrubs happen then
>>> > >>>>>> continue past 256.
>>> > >>>>>>
>>> > >>>>>> with the ultimate target of ar

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

B  31.01  0.89   19  up  osd.16
>> > >>>>> 19hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4
>> MiB
>> > >>>>> 4.0 GiB  6.1 TiB  32.77  0.94   21  up  osd.19
>> > >>>>> 21hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.5
>> MiB
>> > >>>>> 3.7 GiB  6.2 TiB  31.58  0.91   26  up  osd.21
>> > >>>>> 24hdd9.02330   1.0  9.0 TiB  2.6 TiB   855 GiB  4.7
>> MiB
>> > >>>>> 3.3 GiB  6.4 TiB  28.61  0.82   19  up  osd.24
>> > >>>>> 27hdd9.02330   1.0  9.0 TiB  3.7 TiB   1.9 TiB   10
>> MiB
>> > >>>>> 5.2 GiB  5.3 TiB  40.84  1.17   24  up  osd.27
>> > >>>>> 30hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.4 TiB  7.5
>> MiB
>> > >>>>> 4.5 GiB  5.9 TiB  35.16  1.01   22  up  osd.30
>> > >>>>> 33hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.6
>> MiB
>> > >>>>> 4.3 GiB  5.9 TiB  34.59  0.99   23  up  osd.33
>> > >>>>> 36hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB   10
>> MiB
>> > >>>>> 5.0 GiB  5.6 TiB  38.17  1.09   25  up  osd.36
>> > >>>>> 39hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  8.5
>> MiB
>> > >>>>> 5.1 GiB  5.6 TiB  37.79  1.08   31  up  osd.39
>> > >>>>> 42hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10
>> MiB
>> > >>>>> 5.2 GiB  5.4 TiB  39.68  1.14   23  up  osd.42
>> > >>>>> 45hdd9.02330   1.0  9.0 TiB  2.7 TiB   964 GiB  5.1
>> MiB
>> > >>>>> 3.5 GiB  6.3 TiB  29.78  0.85   21  up  osd.45
>> > >>>>> -5 144.37280 -  144 TiB   50 TiB22 TiB  121
>> MiB
>> > >>>>>  70 GiB   94 TiB  34.86  1.00-  host ceph-osd3
>> > >>>>>  0hdd9.02330   1.0  9.0 TiB  2.7 TiB   934 GiB  4.9
>> MiB
>> > >>>>> 3.4 GiB  6.4 TiB  29.47  0.85   21  up  osd.0
>> > >>>>>  4hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5
>> MiB
>> > >>>>> 4.1 GiB  6.1 TiB  32.73  0.94   22  up  osd.4
>> > >>>>>  7hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2
>> MiB
>> > >>>>> 5.1 GiB  5.5 TiB  39.02  1.12   30  up  osd.7
>> > >>>>> 11hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10
>> MiB
>> > >>>>> 5.1 GiB  5.4 TiB  39.97  1.15   27  up  osd.11
>> > >>>>> 14hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.7 TiB   10
>> MiB
>> > >>>>> 5.1 GiB  5.6 TiB  38.24  1.10   27  up  osd.14
>> > >>>>> 17hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4
>> MiB
>> > >>>>> 4.1 GiB  6.0 TiB  33.09  0.95   23  up  osd.17
>> > >>>>> 20hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.6
>> MiB
>> > >>>>> 3.8 GiB  6.2 TiB  31.55  0.90   20  up  osd.20
>> > >>>>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0
>> MiB
>> > >>>>> 3.3 GiB  6.5 TiB  28.32  0.81   23  up  osd.23
>> > >>>>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8
>> MiB
>> > >>>>> 3.8 GiB  6.1 TiB  32.12  0.92   26  up  osd.26
>> > >>>>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10
>> MiB
>> > >>>>> 5.1 GiB  5.4 TiB  39.73  1.14   24  up  osd.29
>> > >>>>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8
>> MiB
>> > >>>>> 3.7 GiB  6.2 TiB  31.56  0.91   22  up  osd.31
>> > >>>>> 34hdd9.02330   1.00000  9.0 TiB  3.3 TiB   1.5 TiB  8.2
>> MiB
>> > >>>>> 4.6 GiB  5.7 TiB  36.29  1.04   23  up  osd.34
>> > >>>>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2
>> MiB
>> > >>>>> 4.5 GiB  5.8 TiB  35.51  1.02   20  up  osd.37
>> > >&g

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

6 TiB  37.79  1.08   31  up  osd.39
> > >>>>> 42hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB
> > >>>>> 5.2 GiB  5.4 TiB  39.68  1.14   23  up  osd.42
> > >>>>> 45hdd9.02330   1.0  9.0 TiB  2.7 TiB   964 GiB  5.1 MiB
> > >>>>> 3.5 GiB  6.3 TiB  29.78  0.85   21  up  osd.45
> > >>>>> -5 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB
> > >>>>>  70 GiB   94 TiB  34.86  1.00-  host ceph-osd3
> > >>>>>  0hdd9.02330   1.0  9.0 TiB  2.7 TiB   934 GiB  4.9 MiB
> > >>>>> 3.4 GiB  6.4 TiB  29.47  0.85   21  up  osd.0
> > >>>>>  4hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5 MiB
> > >>>>> 4.1 GiB  6.1 TiB  32.73  0.94   22  up  osd.4
> > >>>>>  7hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB
> > >>>>> 5.1 GiB  5.5 TiB  39.02  1.12   30  up  osd.7
> > >>>>> 11hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10 MiB
> > >>>>> 5.1 GiB  5.4 TiB  39.97  1.15   27  up  osd.11
> > >>>>> 14hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.7 TiB   10 MiB
> > >>>>> 5.1 GiB  5.6 TiB  38.24  1.10   27  up  osd.14
> > >>>>> 17hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4 MiB
> > >>>>> 4.1 GiB  6.0 TiB  33.09  0.95   23  up  osd.17
> > >>>>> 20hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.6 MiB
> > >>>>> 3.8 GiB  6.2 TiB  31.55  0.90   20  up  osd.20
> > >>>>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0 MiB
> > >>>>> 3.3 GiB  6.5 TiB  28.32  0.81   23  up  osd.23
> > >>>>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8 MiB
> > >>>>> 3.8 GiB  6.1 TiB  32.12  0.92   26  up  osd.26
> > >>>>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB
> > >>>>> 5.1 GiB  5.4 TiB  39.73  1.14   24  up  osd.29
> > >>>>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8 MiB
> > >>>>> 3.7 GiB  6.2 TiB  31.56  0.91   22  up  osd.31
> > >>>>> 34hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.2 MiB
> > >>>>> 4.6 GiB  5.7 TiB  36.29  1.04   23  up  osd.34
> > >>>>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2 MiB
> > >>>>> 4.5 GiB  5.8 TiB  35.51  1.02   20  up  osd.37
> > >>>>> 40hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  9.3 MiB
> > >>>>> 4.9 GiB  5.6 TiB  38.16  1.09   25  up  osd.40
> > >>>>> 43hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.6 TiB  8.5 MiB
> > >>>>> 4.8 GiB  5.7 TiB  37.19  1.07   29  up  osd.43
> > >>>>> 46hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4 MiB
> > >>>>> 4.4 GiB  5.9 TiB  34.85  1.00   23  up  osd.46
> > >>>>>  TOTAL  433 TiB  151 TiB67 TiB  364 MiB
> > >>>>> 210 GiB  282 TiB  34.86
> > >>>>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
> > >>>>>
> > >>>>>
> > >>>>> Michel
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <
> > >>>>> w...@wesdillingham.com> wrote:
> > >>>>>
> > >>>>>> I now concur you should increase the pg_num as a first step for
> this
> > >>>>>> cluster. Disable the pg autoscaler for and increase the volumes
> pool to
> > >>>>>> pg_num 256. Then likely re-asses and make the next power of 2
> jump to 512
> > >>>>>> and probably beyond.
> > >>>>>>
> > >>>>>> Keep in mind this is not going to fix your short term deep-scrub
> > >>>>>> issue in fact it will increase the number of not scrubbed in time
> PGs until
> > >>>>>> the pg_num change is complete.  This is because OSDs dont scrub
> when they
> > >>>>>> are backfilling.
> &

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Janne Johansson]

   1.7 TiB   10 MiB
> >>>>> 5.1 GiB  5.6 TiB  38.24  1.10   27  up  osd.14
> >>>>> 17hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4 MiB
> >>>>> 4.1 GiB  6.0 TiB  33.09  0.95   23  up  osd.17
> >>>>> 20hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.6 MiB
> >>>>> 3.8 GiB  6.2 TiB  31.55  0.90   20  up  osd.20
> >>>>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0 MiB
> >>>>> 3.3 GiB  6.5 TiB  28.32  0.81   23  up  osd.23
> >>>>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8 MiB
> >>>>> 3.8 GiB  6.1 TiB  32.12  0.92   26  up  osd.26
> >>>>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB
> >>>>> 5.1 GiB  5.4 TiB  39.73  1.14   24  up  osd.29
> >>>>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8 MiB
> >>>>> 3.7 GiB  6.2 TiB  31.56  0.91   22  up  osd.31
> >>>>> 34hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.2 MiB
> >>>>> 4.6 GiB  5.7 TiB  36.29  1.04   23  up  osd.34
> >>>>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2 MiB
> >>>>> 4.5 GiB  5.8 TiB  35.51  1.02   20  up  osd.37
> >>>>> 40hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  9.3 MiB
> >>>>> 4.9 GiB  5.6 TiB  38.16  1.09   25  up  osd.40
> >>>>> 43hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.6 TiB  8.5 MiB
> >>>>> 4.8 GiB  5.7 TiB  37.19  1.07   29  up  osd.43
> >>>>> 46hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4 MiB
> >>>>> 4.4 GiB  5.9 TiB  34.85  1.00   23  up  osd.46
> >>>>>  TOTAL  433 TiB  151 TiB67 TiB  364 MiB
> >>>>> 210 GiB  282 TiB  34.86
> >>>>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
> >>>>>
> >>>>>
> >>>>> Michel
> >>>>>
> >>>>>
> >>>>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <
> >>>>> w...@wesdillingham.com> wrote:
> >>>>>
> >>>>>> I now concur you should increase the pg_num as a first step for this
> >>>>>> cluster. Disable the pg autoscaler for and increase the volumes pool to
> >>>>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 
> >>>>>> 512
> >>>>>> and probably beyond.
> >>>>>>
> >>>>>> Keep in mind this is not going to fix your short term deep-scrub
> >>>>>> issue in fact it will increase the number of not scrubbed in time PGs 
> >>>>>> until
> >>>>>> the pg_num change is complete.  This is because OSDs dont scrub when 
> >>>>>> they
> >>>>>> are backfilling.
> >>>>>>
> >>>>>> I would sit on 256 for a couple weeks and let scrubs happen then
> >>>>>> continue past 256.
> >>>>>>
> >>>>>> with the ultimate target of around 100-200 PGs per OSD which "ceph
> >>>>>> osd df tree" will show you in the PGs column.
> >>>>>>
> >>>>>> Respectfully,
> >>>>>>
> >>>>>> *Wes Dillingham*
> >>>>>> w...@wesdillingham.com
> >>>>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita 
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Dear team,
> >>>>>>>
> >>>>>>> below is the output of ceph df command and the ceph version I am
> >>>>>>> running
> >>>>>>>
> >>>>>>>  ceph df
> >>>>>>> --- RAW STORAGE ---
> >>>>>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
> >>>>>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
> >>>>>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
> >>>>>>>
> >>>>>>> --- POOLS --

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4 MiB
>>>>> 4.4 GiB  5.9 TiB  34.85  1.00   23  up  osd.46
>>>>>  TOTAL  433 TiB  151 TiB67 TiB  364 MiB
>>>>> 210 GiB  282 TiB  34.86
>>>>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
>>>>>
>>>>>
>>>>> Michel
>>>>>
>>>>>
>>>>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <
>>>>> w...@wesdillingham.com> wrote:
>>>>>
>>>>>> I now concur you should increase the pg_num as a first step for this
>>>>>> cluster. Disable the pg autoscaler for and increase the volumes pool to
>>>>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512
>>>>>> and probably beyond.
>>>>>>
>>>>>> Keep in mind this is not going to fix your short term deep-scrub
>>>>>> issue in fact it will increase the number of not scrubbed in time PGs 
>>>>>> until
>>>>>> the pg_num change is complete.  This is because OSDs dont scrub when they
>>>>>> are backfilling.
>>>>>>
>>>>>> I would sit on 256 for a couple weeks and let scrubs happen then
>>>>>> continue past 256.
>>>>>>
>>>>>> with the ultimate target of around 100-200 PGs per OSD which "ceph
>>>>>> osd df tree" will show you in the PGs column.
>>>>>>
>>>>>> Respectfully,
>>>>>>
>>>>>> *Wes Dillingham*
>>>>>> w...@wesdillingham.com
>>>>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita 
>>>>>> wrote:
>>>>>>
>>>>>>> Dear team,
>>>>>>>
>>>>>>> below is the output of ceph df command and the ceph version I am
>>>>>>> running
>>>>>>>
>>>>>>>  ceph df
>>>>>>> --- RAW STORAGE ---
>>>>>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>>>>>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>>>>
>>>>>>> --- POOLS ---
>>>>>>> POOL   ID  PGS   STORED  OBJECTS USED  %USED
>>>>>>> MAX AVAIL
>>>>>>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0
>>>>>>>  73 TiB
>>>>>>> .rgw.root   2   32  3.7 KiB8   96 KiB  0
>>>>>>>  73 TiB
>>>>>>> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0
>>>>>>>  73 TiB
>>>>>>> default.rgw.control 4   32  0 B8  0 B  0
>>>>>>>  73 TiB
>>>>>>> default.rgw.meta5   32382 B2   24 KiB  0
>>>>>>>  73 TiB
>>>>>>> volumes 6  128   21 TiB5.68M   62 TiB  22.09
>>>>>>>  73 TiB
>>>>>>> images  7   32  878 GiB  112.50k  2.6 TiB   1.17
>>>>>>>  73 TiB
>>>>>>> backups 8   32  0 B0  0 B  0
>>>>>>>  73 TiB
>>>>>>> vms 9   32  881 GiB  174.30k  2.5 TiB   1.13
>>>>>>>  73 TiB
>>>>>>> testbench  10   32  0 B0  0 B  0
>>>>>>>  73 TiB
>>>>>>> root@ceph-mon1:~# ceph --version
>>>>>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894)
>>>>>>> pacific
>>>>>>> (stable)
>>>>>>> root@ceph-mon1:~#
>>>>>>>
>>>>>>> please advise accordingly
>>>>>>>
>>>>>>> Michel
>>>>>>>
>>>>>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>>>>>>>
>>>>>>> > You will have to look at the output of "ceph df" and make a
>>>>>>> decision to
>>>>>>> > balance "objects per PG" and "GB per PG". Increase h

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

>>>>> *Wes Dillingham*
>>>>> w...@wesdillingham.com
>>>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>>>
>>>>>
>>>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita 
>>>>> wrote:
>>>>>
>>>>>> Dear team,
>>>>>>
>>>>>> below is the output of ceph df command and the ceph version I am
>>>>>> running
>>>>>>
>>>>>>  ceph df
>>>>>> --- RAW STORAGE ---
>>>>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>>>>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>>>
>>>>>> --- POOLS ---
>>>>>> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX
>>>>>> AVAIL
>>>>>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0
>>>>>>  73 TiB
>>>>>> .rgw.root   2   32  3.7 KiB8   96 KiB  0
>>>>>>  73 TiB
>>>>>> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0
>>>>>>  73 TiB
>>>>>> default.rgw.control 4   32  0 B8  0 B  0
>>>>>>  73 TiB
>>>>>> default.rgw.meta5   32382 B2   24 KiB  0
>>>>>>  73 TiB
>>>>>> volumes 6  128   21 TiB5.68M   62 TiB  22.09
>>>>>>  73 TiB
>>>>>> images  7   32  878 GiB  112.50k  2.6 TiB   1.17
>>>>>>  73 TiB
>>>>>> backups 8   32  0 B0  0 B  0
>>>>>>  73 TiB
>>>>>> vms 9   32  881 GiB  174.30k  2.5 TiB   1.13
>>>>>>  73 TiB
>>>>>> testbench  10   32  0 B0  0 B  0
>>>>>>  73 TiB
>>>>>> root@ceph-mon1:~# ceph --version
>>>>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894)
>>>>>> pacific
>>>>>> (stable)
>>>>>> root@ceph-mon1:~#
>>>>>>
>>>>>> please advise accordingly
>>>>>>
>>>>>> Michel
>>>>>>
>>>>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>>>>>>
>>>>>> > You will have to look at the output of "ceph df" and make a
>>>>>> decision to
>>>>>> > balance "objects per PG" and "GB per PG". Increase he PG count for
>>>>>> the
>>>>>> > pools with the worst of these two numbers most such that it
>>>>>> balances out as
>>>>>> > much as possible. If you have pools that see significantly more
>>>>>> user-IO
>>>>>> > than others, prioritise these.
>>>>>> >
>>>>>> > You will have to find out for your specific cluster, we can only
>>>>>> give
>>>>>> > general guidelines. Make changes, run benchmarks, re-evaluate. Take
>>>>>> the
>>>>>> > time for it. The better you know your cluster and your users, the
>>>>>> better
>>>>>> > the end result will be.
>>>>>> >
>>>>>> > Best regards,
>>>>>> > =
>>>>>> > Frank Schilder
>>>>>> > AIT Risø Campus
>>>>>> > Bygning 109, rum S14
>>>>>> >
>>>>>> > 
>>>>>> > From: Michel Niyoyita 
>>>>>> > Sent: Monday, January 29, 2024 2:04 PM
>>>>>> > To: Janne Johansson
>>>>>> > Cc: Frank Schilder; E Taka; ceph-users
>>>>>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>>>>>> >
>>>>>> > This is how it is set , if you suggest to make some changes please
>>>>>> advises.
>>>>>> >
>>>>>> > Thank you.
>>>>>> >
>>>>>> >
>>>>>> > ceph osd pool ls detail
>>>>>> > pool 1 'device_health_metrics' replicated size 3 min_size 2
>>>>>> crush_rule 0
>>>>>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on
>>>>>> last_change 1407
>>>>>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1
>>>>>> application
>>>>>> > mgr_devicehealth
>>>>>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0
>>>>>> object_hash
>>>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393
>>>>>> flags
>>>>>> > hashpspool stripe_width 0 application rgw
>>>>>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
>>>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
>>>>>> last_change
>>>>>> > 1394 flags hashpspool stripe_width 0 application rgw
>>>>>> > pool 4 'default.rgw.control' replicated size 3 min_size 2
>>>>>> crush_rule 0
>>>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
>>>>>> last_change
>>>>>> > 1395 flags hashpspool stripe_width 0 application rgw
>>>>>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
>>>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
>>>>>> last_change
>>>>>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4
>>>>>> application rgw
>>>>>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0
>>>>>> object_hash
>>>>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change
>>>>>> 108802 lfor
>>>>>> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0
>>>>>> application rbd
>>>>>> > removed_snaps_queue
>>>>>> >
>>>>>> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
>>>>>> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0
>>>>>> object_hash
>>>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609
>>>>>> flags
>>>>>> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd
>>>>>> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0
>>>>>> object_hash
>>>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399
>>>>>> flags
>>>>>> > hashpspool stripe_width 0 application rbd
>>>>>> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
>>>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783
>>>>>> lfor
>>>>>> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0
>>>>>> application rbd
>>>>>> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
>>>>>> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0
>>>>>> object_hash
>>>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931
>>>>>> lfor
>>>>>> > 0/20931/20929 flags hashpspool stripe_width 0
>>>>>> >
>>>>>> >
>>>>>> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita >>>>> >>>>> > mico...@gmail.com>> wrote:
>>>>>> > Thank you Janne ,
>>>>>> >
>>>>>> > no need of setting some flags like ceph osd set nodeep-scrub  ???
>>>>>> >
>>>>>> > Thank you
>>>>>> >
>>>>>> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <
>>>>>> icepic...@gmail.com
>>>>>> > <mailto:icepic...@gmail.com>> wrote:
>>>>>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <
>>>>>> mico...@gmail.com
>>>>>> > <mailto:mico...@gmail.com>>:
>>>>>> > >
>>>>>> > > Thank you Frank ,
>>>>>> > >
>>>>>> > > All disks are HDDs . Would like to know if I can increase the
>>>>>> number of
>>>>>> > PGs
>>>>>> > > live in production without a negative impact to the cluster. if
>>>>>> yes which
>>>>>> > > commands to use .
>>>>>> >
>>>>>> > Yes. "ceph osd pool set  pg_num >>>>> before>"
>>>>>> > where the number usually should be a power of two that leads to a
>>>>>> > number of PGs per OSD between 100-200.
>>>>>> >
>>>>>> > --
>>>>>> > May the most significant bit of your life be positive.
>>>>>> >
>>>>>> ___
>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>>>>
>>>>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

.9 MiB  3.4
>>> GiB  6.4 TiB  29.47  0.85   21  up  osd.0
>>>  4hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5 MiB  4.1
>>> GiB  6.1 TiB  32.73  0.94   22  up  osd.4
>>>  7hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB  5.1
>>> GiB  5.5 TiB  39.02  1.12   30  up  osd.7
>>> 11hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10 MiB  5.1
>>> GiB  5.4 TiB  39.97  1.15   27  up  osd.11
>>> 14hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.7 TiB   10 MiB  5.1
>>> GiB  5.6 TiB  38.24  1.10   27  up  osd.14
>>> 17hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4 MiB  4.1
>>> GiB  6.0 TiB  33.09  0.95   23  up  osd.17
>>> 20hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.6 MiB  3.8
>>> GiB  6.2 TiB  31.55  0.90   20  up  osd.20
>>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0 MiB  3.3
>>> GiB  6.5 TiB  28.32  0.81   23  up  osd.23
>>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8 MiB  3.8
>>> GiB  6.1 TiB  32.12  0.92   26  up  osd.26
>>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB  5.1
>>> GiB  5.4 TiB  39.73  1.14   24  up  osd.29
>>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8 MiB  3.7
>>> GiB  6.2 TiB  31.56  0.91   22  up  osd.31
>>> 34hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.2 MiB  4.6
>>> GiB  5.7 TiB  36.29  1.04   23  up  osd.34
>>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2 MiB  4.5
>>> GiB  5.8 TiB  35.51  1.02   20  up  osd.37
>>> 40hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  9.3 MiB  4.9
>>> GiB  5.6 TiB  38.16  1.09   25  up  osd.40
>>> 43hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.6 TiB  8.5 MiB  4.8
>>> GiB  5.7 TiB  37.19  1.07   29  up  osd.43
>>> 46hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4 MiB  4.4
>>> GiB  5.9 TiB  34.85  1.00   23  up  osd.46
>>>  TOTAL  433 TiB  151 TiB67 TiB  364 MiB  210
>>> GiB  282 TiB  34.86
>>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
>>>
>>>
>>> Michel
>>>
>>>
>>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham 
>>> wrote:
>>>
>>>> I now concur you should increase the pg_num as a first step for this
>>>> cluster. Disable the pg autoscaler for and increase the volumes pool to
>>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512
>>>> and probably beyond.
>>>>
>>>> Keep in mind this is not going to fix your short term deep-scrub issue
>>>> in fact it will increase the number of not scrubbed in time PGs until the
>>>> pg_num change is complete.  This is because OSDs dont scrub when they are
>>>> backfilling.
>>>>
>>>> I would sit on 256 for a couple weeks and let scrubs happen then
>>>> continue past 256.
>>>>
>>>> with the ultimate target of around 100-200 PGs per OSD which "ceph osd
>>>> df tree" will show you in the PGs column.
>>>>
>>>> Respectfully,
>>>>
>>>> *Wes Dillingham*
>>>> w...@wesdillingham.com
>>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>>
>>>>
>>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita 
>>>> wrote:
>>>>
>>>>> Dear team,
>>>>>
>>>>> below is the output of ceph df command and the ceph version I am
>>>>> running
>>>>>
>>>>>  ceph df
>>>>> --- RAW STORAGE ---
>>>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>>>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>>
>>>>> --- POOLS ---
>>>>> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX
>>>>> AVAIL
>>>>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0
>>>>>  73 TiB
>>>>> .rgw.root   2   32  3.7 KiB8   96 KiB      0
>>>>>  73 TiB
>>>>> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0
>>

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

p  osd.20
>> 23hdd9.02330   1.0  9.0 TiB  2.6 TiB   828 GiB  4.0 MiB  3.3
>> GiB  6.5 TiB  28.32  0.81   23  up  osd.23
>> 26hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  5.8 MiB  3.8
>> GiB  6.1 TiB  32.12  0.92   26  up  osd.26
>> 29hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB  5.1
>> GiB  5.4 TiB  39.73  1.14   24  up  osd.29
>> 31hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.8 MiB  3.7
>> GiB  6.2 TiB  31.56  0.91   22  up  osd.31
>> 34hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.2 MiB  4.6
>> GiB  5.7 TiB  36.29  1.04   23  up  osd.34
>> 37hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  8.2 MiB  4.5
>> GiB  5.8 TiB  35.51  1.02   20  up  osd.37
>> 40hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  9.3 MiB  4.9
>> GiB  5.6 TiB  38.16  1.09   25  up  osd.40
>> 43hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.6 TiB  8.5 MiB  4.8
>> GiB  5.7 TiB  37.19  1.07   29  up  osd.43
>> 46hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.4 MiB  4.4
>> GiB  5.9 TiB  34.85  1.00   23  up  osd.46
>>  TOTAL  433 TiB  151 TiB67 TiB  364 MiB  210
>> GiB  282 TiB  34.86
>> MIN/MAX VAR: 0.81/1.28  STDDEV: 3.95
>>
>>
>> Michel
>>
>>
>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham 
>> wrote:
>>
>>> I now concur you should increase the pg_num as a first step for this
>>> cluster. Disable the pg autoscaler for and increase the volumes pool to
>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512
>>> and probably beyond.
>>>
>>> Keep in mind this is not going to fix your short term deep-scrub issue
>>> in fact it will increase the number of not scrubbed in time PGs until the
>>> pg_num change is complete.  This is because OSDs dont scrub when they are
>>> backfilling.
>>>
>>> I would sit on 256 for a couple weeks and let scrubs happen then
>>> continue past 256.
>>>
>>> with the ultimate target of around 100-200 PGs per OSD which "ceph osd
>>> df tree" will show you in the PGs column.
>>>
>>> Respectfully,
>>>
>>> *Wes Dillingham*
>>> w...@wesdillingham.com
>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>
>>>
>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita 
>>> wrote:
>>>
>>>> Dear team,
>>>>
>>>> below is the output of ceph df command and the ceph version I am running
>>>>
>>>>  ceph df
>>>> --- RAW STORAGE ---
>>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>>
>>>> --- POOLS ---
>>>> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX
>>>> AVAIL
>>>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73
>>>> TiB
>>>> .rgw.root   2   32  3.7 KiB8   96 KiB  0 73
>>>> TiB
>>>> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73
>>>> TiB
>>>> default.rgw.control 4   32  0 B8  0 B  0 73
>>>> TiB
>>>> default.rgw.meta5   32382 B2   24 KiB  0 73
>>>> TiB
>>>> volumes 6  128   21 TiB5.68M   62 TiB  22.09 73
>>>> TiB
>>>> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73
>>>> TiB
>>>> backups 8   32  0 B0  0 B  0 73
>>>> TiB
>>>> vms             9   32  881 GiB  174.30k  2.5 TiB   1.13 73
>>>> TiB
>>>> testbench  10   32  0 B0  0 B  0 73
>>>> TiB
>>>> root@ceph-mon1:~# ceph --version
>>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
>>>> (stable)
>>>> root@ceph-mon1:~#
>>>>
>>>> please advise accordingly
>>>>
>>>> Michel
>>>>
>>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>>>>
>>>> > You will have to look at the output of "ceph df" and make a deci

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

am,
>>>
>>> below is the output of ceph df command and the ceph version I am running
>>>
>>>  ceph df
>>> --- RAW STORAGE ---
>>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>>>
>>> --- POOLS ---
>>> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX
>>> AVAIL
>>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73
>>> TiB
>>> .rgw.root   2   32  3.7 KiB8   96 KiB  0 73
>>> TiB
>>> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73
>>> TiB
>>> default.rgw.control 4   32  0 B8  0 B  0 73
>>> TiB
>>> default.rgw.meta5   32382 B2   24 KiB  0 73
>>> TiB
>>> volumes 6  128   21 TiB5.68M   62 TiB  22.09 73
>>> TiB
>>> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73
>>> TiB
>>> backups 8   32  0 B0  0 B  0 73
>>> TiB
>>> vms 9   32  881 GiB  174.30k  2.5 TiB   1.13 73
>>> TiB
>>> testbench  10   32  0 B0  0 B  0 73
>>> TiB
>>> root@ceph-mon1:~# ceph --version
>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
>>> (stable)
>>> root@ceph-mon1:~#
>>>
>>> please advise accordingly
>>>
>>> Michel
>>>
>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>>>
>>> > You will have to look at the output of "ceph df" and make a decision to
>>> > balance "objects per PG" and "GB per PG". Increase he PG count for the
>>> > pools with the worst of these two numbers most such that it balances
>>> out as
>>> > much as possible. If you have pools that see significantly more user-IO
>>> > than others, prioritise these.
>>> >
>>> > You will have to find out for your specific cluster, we can only give
>>> > general guidelines. Make changes, run benchmarks, re-evaluate. Take the
>>> > time for it. The better you know your cluster and your users, the
>>> better
>>> > the end result will be.
>>> >
>>> > Best regards,
>>> > =
>>> > Frank Schilder
>>> > AIT Risø Campus
>>> > Bygning 109, rum S14
>>> >
>>> > 
>>> > From: Michel Niyoyita 
>>> > Sent: Monday, January 29, 2024 2:04 PM
>>> > To: Janne Johansson
>>> > Cc: Frank Schilder; E Taka; ceph-users
>>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>>> >
>>> > This is how it is set , if you suggest to make some changes please
>>> advises.
>>> >
>>> > Thank you.
>>> >
>>> >
>>> > ceph osd pool ls detail
>>> > pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule
>>> 0
>>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change
>>> 1407
>>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
>>> > mgr_devicehealth
>>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0
>>> object_hash
>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
>>> > hashpspool stripe_width 0 application rgw
>>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>>> > 1394 flags hashpspool stripe_width 0 application rgw
>>> > pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>>> > 1395 flags hashpspool stripe_width 0 application rgw
>>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application
>>> rgw
>>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_chan

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

ision to
>> > balance "objects per PG" and "GB per PG". Increase he PG count for the
>> > pools with the worst of these two numbers most such that it balances
>> out as
>> > much as possible. If you have pools that see significantly more user-IO
>> > than others, prioritise these.
>> >
>> > You will have to find out for your specific cluster, we can only give
>> > general guidelines. Make changes, run benchmarks, re-evaluate. Take the
>> > time for it. The better you know your cluster and your users, the better
>> > the end result will be.
>> >
>> > Best regards,
>> > =
>> > Frank Schilder
>> > AIT Risø Campus
>> > Bygning 109, rum S14
>> >
>> > 
>> > From: Michel Niyoyita 
>> > Sent: Monday, January 29, 2024 2:04 PM
>> > To: Janne Johansson
>> > Cc: Frank Schilder; E Taka; ceph-users
>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>> >
>> > This is how it is set , if you suggest to make some changes please
>> advises.
>> >
>> > Thank you.
>> >
>> >
>> > ceph osd pool ls detail
>> > pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change
>> 1407
>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
>> > mgr_devicehealth
>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
>> > hashpspool stripe_width 0 application rgw
>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>> > 1394 flags hashpspool stripe_width 0 application rgw
>> > pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>> > 1395 flags hashpspool stripe_width 0 application rgw
>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802
>> lfor
>> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application
>> rbd
>> > removed_snaps_queue
>> >
>> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
>> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags
>> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd
>> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash
>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags
>> > hashpspool stripe_width 0 application rbd
>> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor
>> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application
>> rbd
>> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
>> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0
>> object_hash
>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor
>> > 0/20931/20929 flags hashpspool stripe_width 0
>> >
>> >
>> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita > > > mico...@gmail.com>> wrote:
>> > Thank you Janne ,
>> >
>> > no need of setting some flags like ceph osd set nodeep-scrub  ???
>> >
>> > Thank you
>> >
>> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson > > <mailto:icepic...@gmail.com>> wrote:
>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita > > <mailto:mico...@gmail.com>>:
>> > >
>> > > Thank you Frank ,
>> > >
>> > > All disks are HDDs . Would like to know if I can increase the number
>> of
>> > PGs
>> > > live in production without a negative impact to the cluster. if yes
>> which
>> > > commands to use .
>> >
>> > Yes. "ceph osd pool set  pg_num "
>> > where the number usually should be a power of two that leads to a
>> > number of PGs per OSD between 100-200.
>> >
>> > --
>> > May the most significant bit of your life be positive.
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

I now concur you should increase the pg_num as a first step for this
cluster. Disable the pg autoscaler for and increase the volumes pool to
pg_num 256. Then likely re-asses and make the next power of 2 jump to 512
and probably beyond.

Keep in mind this is not going to fix your short term deep-scrub issue in
fact it will increase the number of not scrubbed in time PGs until the
pg_num change is complete.  This is because OSDs dont scrub when they are
backfilling.

I would sit on 256 for a couple weeks and let scrubs happen then continue
past 256.

with the ultimate target of around 100-200 PGs per OSD which "ceph osd df
tree" will show you in the PGs column.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>


On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita  wrote:

> Dear team,
>
> below is the output of ceph df command and the ceph version I am running
>
>  ceph df
> --- RAW STORAGE ---
> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>
> --- POOLS ---
> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73 TiB
> .rgw.root   2   32  3.7 KiB8   96 KiB  0 73 TiB
> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73 TiB
> default.rgw.control 4   32  0 B8  0 B  0 73 TiB
> default.rgw.meta5   32382 B2   24 KiB  0 73 TiB
> volumes 6  128   21 TiB5.68M   62 TiB  22.09 73 TiB
> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73 TiB
> backups 8   32  0 B0  0 B  0 73 TiB
> vms 9   32  881 GiB  174.30k  2.5 TiB   1.13 73 TiB
> testbench  10   32  0 B0  0 B  0 73 TiB
> root@ceph-mon1:~# ceph --version
> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
> (stable)
> root@ceph-mon1:~#
>
> please advise accordingly
>
> Michel
>
> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>
> > You will have to look at the output of "ceph df" and make a decision to
> > balance "objects per PG" and "GB per PG". Increase he PG count for the
> > pools with the worst of these two numbers most such that it balances out
> as
> > much as possible. If you have pools that see significantly more user-IO
> > than others, prioritise these.
> >
> > You will have to find out for your specific cluster, we can only give
> > general guidelines. Make changes, run benchmarks, re-evaluate. Take the
> > time for it. The better you know your cluster and your users, the better
> > the end result will be.
> >
> > Best regards,
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Michel Niyoyita 
> > Sent: Monday, January 29, 2024 2:04 PM
> > To: Janne Johansson
> > Cc: Frank Schilder; E Taka; ceph-users
> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
> >
> > This is how it is set , if you suggest to make some changes please
> advises.
> >
> > Thank you.
> >
> >
> > ceph osd pool ls detail
> > pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change
> 1407
> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
> > mgr_devicehealth
> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
> > hashpspool stripe_width 0 application rgw
> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1394 flags hashpspool stripe_width 0 application rgw
> > pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1395 flags hashpspool stripe_width 0 application rgw
> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 128 pgp

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Dear team,

below is the output of ceph df command and the ceph version I am running

 ceph df
--- RAW STORAGE ---
CLASS SIZEAVAIL USED  RAW USED  %RAW USED
hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82

--- POOLS ---
POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73 TiB
.rgw.root   2   32  3.7 KiB8   96 KiB  0 73 TiB
default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73 TiB
default.rgw.control 4   32  0 B8  0 B  0 73 TiB
default.rgw.meta5   32382 B2   24 KiB  0 73 TiB
volumes 6  128   21 TiB5.68M   62 TiB  22.09 73 TiB
images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73 TiB
backups 8   32  0 B0  0 B  0 73 TiB
vms 9   32  881 GiB  174.30k  2.5 TiB   1.13 73 TiB
testbench  10   32  0 B0  0 B  0 73 TiB
root@ceph-mon1:~# ceph --version
ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
(stable)
root@ceph-mon1:~#

please advise accordingly

Michel

On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:

> You will have to look at the output of "ceph df" and make a decision to
> balance "objects per PG" and "GB per PG". Increase he PG count for the
> pools with the worst of these two numbers most such that it balances out as
> much as possible. If you have pools that see significantly more user-IO
> than others, prioritise these.
>
> You will have to find out for your specific cluster, we can only give
> general guidelines. Make changes, run benchmarks, re-evaluate. Take the
> time for it. The better you know your cluster and your users, the better
> the end result will be.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Michel Niyoyita 
> Sent: Monday, January 29, 2024 2:04 PM
> To: Janne Johansson
> Cc: Frank Schilder; E Taka; ceph-users
> Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>
> This is how it is set , if you suggest to make some changes please advises.
>
> Thank you.
>
>
> ceph osd pool ls detail
> pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407
> flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
> mgr_devicehealth
> pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
> hashpspool stripe_width 0 application rgw
> pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1394 flags hashpspool stripe_width 0 application rgw
> pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1395 flags hashpspool stripe_width 0 application rgw
> pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
> pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor
> 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> removed_snaps_queue
> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
> pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags
> hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags
> hashpspool stripe_width 0 application rbd
> pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor
> 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
> pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor
> 0/20931/20929 flags hashpspool stripe_width 0
>
>
> On Mon, Jan 29, 

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

You will have to look at the output of "ceph df" and make a decision to balance 
"objects per PG" and "GB per PG". Increase he PG count for the pools with the 
worst of these two numbers most such that it balances out as much as possible. 
If you have pools that see significantly more user-IO than others, prioritise 
these. 

You will have to find out for your specific cluster, we can only give general 
guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. 
The better you know your cluster and your users, the better the end result will 
be.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Michel Niyoyita 
Sent: Monday, January 29, 2024 2:04 PM
To: Janne Johansson
Cc: Frank Schilder; E Taka; ceph-users
Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time

This is how it is set , if you suggest to make some changes please advises.

Thank you.


ceph osd pool ls detail
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 
object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 
flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application 
mgr_devicehealth
pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags 
hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags 
hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 
flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags 
hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue 
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags 
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool 
stripe_width 0 application rbd
pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags 
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 
0/20931/20929 flags hashpspool stripe_width 0


On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita 
mailto:mico...@gmail.com>> wrote:
Thank you Janne ,

no need of setting some flags like ceph osd set nodeep-scrub  ???

Thank you

On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson 
mailto:icepic...@gmail.com>> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita 
mailto:mico...@gmail.com>>:
>
> Thank you Frank ,
>
> All disks are HDDs . Would like to know if I can increase the number of PGs
> live in production without a negative impact to the cluster. if yes which
> commands to use .

Yes. "ceph osd pool set  pg_num "
where the number usually should be a power of two that leads to a
number of PGs per OSD between 100-200.

--
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

Setting osd_max_scrubs = 2 for HDD OSDs was a mistake I made. The result was 
that PGs needed a bit more than twice as long to deep-scrub. Net effect: high 
scrub load, much less user IO and, last but not least, the "not deep-scrubbed 
in time" problem got worse, because (2+eps)/2 > 1.

For spinners a consideration looking at the actually available drive 
performance is required, plus a few things more, like PG count, distribution 
etc.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Wesley Dillingham 
Sent: Monday, January 29, 2024 7:14 PM
To: Michel Niyoyita
Cc: Josh Baergen; E Taka; ceph-users
Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time

Respond back with "ceph versions" output

If your sole goal is to eliminate the not scrubbed in time errors you can
increase the aggressiveness of scrubbing by setting:
osd_max_scrubs = 2

The default in pacific is 1.

if you are going to start tinkering manually with the pg_num you will want
to turn off the pg autoscaler on the pools you are touching.
reducing the size of your PGs may make sense and help with scrubbing but if
the pool has a lot of data it will take a long long time to finish.





Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>


On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita  wrote:

> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
> wrote:
>
> > Make sure you're on a fairly recent version of Ceph before doing this,
> > though.
> >
> > Josh
> >
> > On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> > wrote:
> > >
> > > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita  >:
> > > >
> > > > Thank you Frank ,
> > > >
> > > > All disks are HDDs . Would like to know if I can increase the number
> > of PGs
> > > > live in production without a negative impact to the cluster. if yes
> > which
> > > > commands to use .
> > >
> > > Yes. "ceph osd pool set  pg_num "
> > > where the number usually should be a power of two that leads to a
> > > number of PGs per OSD between 100-200.
> > >
> > > --
> > > May the most significant bit of your life be positive.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

Respond back with "ceph versions" output

If your sole goal is to eliminate the not scrubbed in time errors you can
increase the aggressiveness of scrubbing by setting:
osd_max_scrubs = 2

The default in pacific is 1.

if you are going to start tinkering manually with the pg_num you will want
to turn off the pg autoscaler on the pools you are touching.
reducing the size of your PGs may make sense and help with scrubbing but if
the pool has a lot of data it will take a long long time to finish.





Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita  wrote:

> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
> wrote:
>
> > Make sure you're on a fairly recent version of Ceph before doing this,
> > though.
> >
> > Josh
> >
> > On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> > wrote:
> > >
> > > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita  >:
> > > >
> > > > Thank you Frank ,
> > > >
> > > > All disks are HDDs . Would like to know if I can increase the number
> > of PGs
> > > > live in production without a negative impact to the cluster. if yes
> > which
> > > > commands to use .
> > >
> > > Yes. "ceph osd pool set  pg_num "
> > > where the number usually should be a power of two that leads to a
> > > number of PGs per OSD between 100-200.
> > >
> > > --
> > > May the most significant bit of your life be positive.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Josh Baergen]

You need to be running at least 16.2.11 on the OSDs so that you have
the fix for https://tracker.ceph.com/issues/55631.

On Mon, Jan 29, 2024 at 8:07 AM Michel Niyoyita  wrote:
>
> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using 
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen  
> wrote:
>>
>> Make sure you're on a fairly recent version of Ceph before doing this, 
>> though.
>>
>> Josh
>>
>> On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson  wrote:
>> >
>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>> > >
>> > > Thank you Frank ,
>> > >
>> > > All disks are HDDs . Would like to know if I can increase the number of 
>> > > PGs
>> > > live in production without a negative impact to the cluster. if yes which
>> > > commands to use .
>> >
>> > Yes. "ceph osd pool set  pg_num "
>> > where the number usually should be a power of two that leads to a
>> > number of PGs per OSD between 100-200.
>> >
>> > --
>> > May the most significant bit of your life be positive.
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
ceph-ansible.

Michel

On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
wrote:

> Make sure you're on a fairly recent version of Ceph before doing this,
> though.
>
> Josh
>
> On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> wrote:
> >
> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> > >
> > > Thank you Frank ,
> > >
> > > All disks are HDDs . Would like to know if I can increase the number
> of PGs
> > > live in production without a negative impact to the cluster. if yes
> which
> > > commands to use .
> >
> > Yes. "ceph osd pool set  pg_num "
> > where the number usually should be a power of two that leads to a
> > number of PGs per OSD between 100-200.
> >
> > --
> > May the most significant bit of your life be positive.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Josh Baergen]

Make sure you're on a fairly recent version of Ceph before doing this, though.

Josh

On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson  wrote:
>
> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> >
> > Thank you Frank ,
> >
> > All disks are HDDs . Would like to know if I can increase the number of PGs
> > live in production without a negative impact to the cluster. if yes which
> > commands to use .
>
> Yes. "ceph osd pool set  pg_num "
> where the number usually should be a power of two that leads to a
> number of PGs per OSD between 100-200.
>
> --
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

This is how it is set , if you suggest to make some changes please advises.

Thank you.


ceph osd pool ls detail
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407
flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
mgr_devicehealth
pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1394 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1395 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor
0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags
hashpspool stripe_width 0 application rbd
pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor
0/20931/20929 flags hashpspool stripe_width 0


On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita  wrote:

> Thank you Janne ,
>
> no need of setting some flags like ceph osd set nodeep-scrub  ???
>
> Thank you
>
> On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson 
> wrote:
>
>> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>> >
>> > Thank you Frank ,
>> >
>> > All disks are HDDs . Would like to know if I can increase the number of
>> PGs
>> > live in production without a negative impact to the cluster. if yes
>> which
>> > commands to use .
>>
>> Yes. "ceph osd pool set  pg_num "
>> where the number usually should be a power of two that leads to a
>> number of PGs per OSD between 100-200.
>>
>> --
>> May the most significant bit of your life be positive.
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Thank you Janne ,

no need of setting some flags like ceph osd set nodeep-scrub  ???

Thank you

On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson  wrote:

> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> >
> > Thank you Frank ,
> >
> > All disks are HDDs . Would like to know if I can increase the number of
> PGs
> > live in production without a negative impact to the cluster. if yes which
> > commands to use .
>
> Yes. "ceph osd pool set  pg_num "
> where the number usually should be a power of two that leads to a
> number of PGs per OSD between 100-200.
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Janne Johansson]

Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>
> Thank you Frank ,
>
> All disks are HDDs . Would like to know if I can increase the number of PGs
> live in production without a negative impact to the cluster. if yes which
> commands to use .

Yes. "ceph osd pool set  pg_num "
where the number usually should be a power of two that leads to a
number of PGs per OSD between 100-200.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Thank you Frank ,

All disks are HDDs . Would like to know if I can increase the number of PGs
live in production without a negative impact to the cluster. if yes which
commands to use .

Thank you very much for your prompt reply.

Michel

On Mon, Jan 29, 2024 at 10:59 AM Frank Schilder  wrote:

> Hi Michel,
>
> are your OSDs HDD or SSD? If they are HDD, its possible that they can't
> handle the deep-scrub load with default settings. In that case, have a look
> at this post
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5MPU6ODTXUSKT7RVPERBJF/
> for some basic tuning info and a script to check your scrub stamp
> distribution.
>
> You should also get rid of slow/failing ones. Look at smartctl output and
> throw out disks with remapped sectors, uncorrectable r/w errors or
> unusually many corrected read-write-errors (assuming you have disks with
> ECC).
>
> A basic calculation for deep-scrub is as follows: max number of PGs that
> can be scrubbed at the same time: A=#OSDs/replication factor (rounded
> down). Take the B=deep-scrub times from the OSD logs (grep for deep-scrub)
> in minutes. Your pool can deep-scrub at a max A*24*(60/B) PGs per day. For
> reasonable operations you should not do more than 50% of that. With that
> you can calculate how many days it needs to deep-scrub your PGs.
>
> Usual reasons for slow deep-scrub progress is too few PGs. With
> replication factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca
> 200/OSD) but use only 385. You should consider increasing the PG count for
> pools with lots of data. This should already relax the situation somewhat.
> Then do the calc above and tune deep-scrub times per pool such that they
> match with disk performance.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Michel Niyoyita 
> Sent: Monday, January 29, 2024 7:42 AM
> To: E Taka
> Cc: ceph-users
> Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>
> Now they are increasing , Friday I tried to deep-scrubbing manually and
> they have been successfully done , but Monday morning I found that they are
> increasing to 37 , is it the best to deep-scrubbing manually while we are
> using the cluster? if not what is the best to do in order to address that .
>
> Best Regards.
>
> Michel
>
>  ceph -s
>   cluster:
> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
> health: HEALTH_WARN
> 37 pgs not deep-scrubbed in time
>
>   services:
> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
> osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
> rgw: 6 daemons active (6 hosts, 1 zones)
>
>   data:
> pools:   10 pools, 385 pgs
> objects: 6.00M objects, 23 TiB
> usage:   151 TiB used, 282 TiB / 433 TiB avail
> pgs: 381 active+clean
>  4   active+clean+scrubbing+deep
>
>   io:
> client:   265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr
>
> On Sun, Jan 28, 2024 at 6:14 PM E Taka <0eta...@gmail.com> wrote:
>
> > 22 is more often there than the others. Other operations may be blocked
> > because of a deep-scrub is not finished yet. I would remove OSD 22, just
> to
> > be sure about this: ceph orch osd rm osd.22
> >
> > If this does not help, just add it again.
> >
> > Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
> > mico...@gmail.com>:
> >
> >> It seems that are different OSDs as shown here . how have you managed to
> >> sort this out?
> >>
> >> ceph pg dump | grep -F 6.78
> >> dumped all
> >> 6.78   44268   0 0  00
> >> 1786796401180   0  10099 10099
> >>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
> >> 107547:225274427  [12,36,37]  12  [12,36,37]  12
> >> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
> >> 2024-01-11T16:07:54.875746+0200  0
> >> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
> >> dumped all
> >> 6.60   9   0 0  00
> >> 179484338742  716  36  10097 10097
> >>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
> >> 107547:287193139   [32,5,29]  32   [32,5,29]  32
> >> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
> >> 2024-01-13T19:44:26.922000+0200  0
> >&g

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

Hi Michel,

are your OSDs HDD or SSD? If they are HDD, its possible that they can't handle 
the deep-scrub load with default settings. In that case, have a look at this 
post 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5MPU6ODTXUSKT7RVPERBJF/
 for some basic tuning info and a script to check your scrub stamp distribution.

You should also get rid of slow/failing ones. Look at smartctl output and throw 
out disks with remapped sectors, uncorrectable r/w errors or unusually many 
corrected read-write-errors (assuming you have disks with ECC).

A basic calculation for deep-scrub is as follows: max number of PGs that can be 
scrubbed at the same time: A=#OSDs/replication factor (rounded down). Take the 
B=deep-scrub times from the OSD logs (grep for deep-scrub) in minutes. Your 
pool can deep-scrub at a max A*24*(60/B) PGs per day. For reasonable operations 
you should not do more than 50% of that. With that you can calculate how many 
days it needs to deep-scrub your PGs.

Usual reasons for slow deep-scrub progress is too few PGs. With replication 
factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca 200/OSD) but use only 
385. You should consider increasing the PG count for pools with lots of data. 
This should already relax the situation somewhat. Then do the calc above and 
tune deep-scrub times per pool such that they match with disk performance.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Michel Niyoyita 
Sent: Monday, January 29, 2024 7:42 AM
To: E Taka
Cc: ceph-users
Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time

Now they are increasing , Friday I tried to deep-scrubbing manually and
they have been successfully done , but Monday morning I found that they are
increasing to 37 , is it the best to deep-scrubbing manually while we are
using the cluster? if not what is the best to do in order to address that .

Best Regards.

Michel

 ceph -s
  cluster:
id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
health: HEALTH_WARN
37 pgs not deep-scrubbed in time

  services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
rgw: 6 daemons active (6 hosts, 1 zones)

  data:
pools:   10 pools, 385 pgs
objects: 6.00M objects, 23 TiB
usage:   151 TiB used, 282 TiB / 433 TiB avail
pgs: 381 active+clean
 4   active+clean+scrubbing+deep

  io:
client:   265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr

On Sun, Jan 28, 2024 at 6:14 PM E Taka <0eta...@gmail.com> wrote:

> 22 is more often there than the others. Other operations may be blocked
> because of a deep-scrub is not finished yet. I would remove OSD 22, just to
> be sure about this: ceph orch osd rm osd.22
>
> If this does not help, just add it again.
>
> Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
> mico...@gmail.com>:
>
>> It seems that are different OSDs as shown here . how have you managed to
>> sort this out?
>>
>> ceph pg dump | grep -F 6.78
>> dumped all
>> 6.78   44268   0 0  00
>> 1786796401180   0  10099 10099
>>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
>> 107547:225274427  [12,36,37]  12  [12,36,37]  12
>> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
>> 2024-01-11T16:07:54.875746+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
>> dumped all
>> 6.60   9   0 0  00
>> 179484338742  716  36  10097 10097
>>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
>> 107547:287193139   [32,5,29]  32   [32,5,29]  32
>> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
>> 2024-01-13T19:44:26.922000+0200  0
>> 6.3a   44807   0 0  00
>> 1809690056940   0  10093 10093
>>  active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
>> 107547:238170093  [22,13,11]  22  [22,13,11]  22
>> 106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
>> 2024-01-15T05:31:36.606478+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
>> 6.5c   44277   0 0  00
>> 1787649782300   0  10051 10051
>>  active+clean  2024-01-26T03:55:23.339584+0200  107547'126480090
>> 107547:264432655  [22,37,30]  22  [22,37,30]  22
>> 107205'12

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Now they are increasing , Friday I tried to deep-scrubbing manually and
they have been successfully done , but Monday morning I found that they are
increasing to 37 , is it the best to deep-scrubbing manually while we are
using the cluster? if not what is the best to do in order to address that .

Best Regards.

Michel

 ceph -s
  cluster:
id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
health: HEALTH_WARN
37 pgs not deep-scrubbed in time

  services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
rgw: 6 daemons active (6 hosts, 1 zones)

  data:
pools:   10 pools, 385 pgs
objects: 6.00M objects, 23 TiB
usage:   151 TiB used, 282 TiB / 433 TiB avail
pgs: 381 active+clean
 4   active+clean+scrubbing+deep

  io:
client:   265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr

On Sun, Jan 28, 2024 at 6:14 PM E Taka <0eta...@gmail.com> wrote:

> 22 is more often there than the others. Other operations may be blocked
> because of a deep-scrub is not finished yet. I would remove OSD 22, just to
> be sure about this: ceph orch osd rm osd.22
>
> If this does not help, just add it again.
>
> Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
> mico...@gmail.com>:
>
>> It seems that are different OSDs as shown here . how have you managed to
>> sort this out?
>>
>> ceph pg dump | grep -F 6.78
>> dumped all
>> 6.78   44268   0 0  00
>> 1786796401180   0  10099 10099
>>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
>> 107547:225274427  [12,36,37]  12  [12,36,37]  12
>> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
>> 2024-01-11T16:07:54.875746+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
>> dumped all
>> 6.60   9   0 0  00
>> 179484338742  716  36  10097 10097
>>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
>> 107547:287193139   [32,5,29]  32   [32,5,29]  32
>> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
>> 2024-01-13T19:44:26.922000+0200  0
>> 6.3a   44807   0 0  00
>> 1809690056940   0  10093 10093
>>  active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
>> 107547:238170093  [22,13,11]  22  [22,13,11]  22
>> 106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
>> 2024-01-15T05:31:36.606478+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
>> 6.5c   44277   0 0  00
>> 1787649782300   0  10051 10051
>>  active+clean  2024-01-26T03:55:23.339584+0200  107547'126480090
>> 107547:264432655  [22,37,30]  22  [22,37,30]  22
>> 107205'125858697  2024-01-24T22:32:10.365869+0200  101941'120957992
>> 2024-01-13T09:07:24.780936+0200  0
>> dumped all
>> root@ceph-osd3:~# ceph pg dump | grep -F 4.12
>> dumped all
>> 4.12   0   0 0  00
>>  00   0  0 0
>>  active+clean  2024-01-24T08:36:48.284388+0200   0'0
>>  107546:152711   [22,19,7]  22   [22,19,7]  22
>>  0'0  2024-01-24T08:36:48.284307+0200   0'0
>> 2024-01-13T09:09:22.176240+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 10.d
>> dumped all
>> 10.d   0   0 0  00
>>  00   0  0 0
>>  active+clean  2024-01-24T04:04:33.641541+0200   0'0
>>  107546:142651   [14,28,1]  14   [14,28,1]  14
>>  0'0  2024-01-24T04:04:33.641451+0200   0'0
>> 2024-01-12T08:04:02.078062+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 5.f
>> dumped all
>> 5.f0   0 0  00
>>  00   0  0 0
>>  active+clean  2024-01-25T08:19:04.148941+0200   0'0
>>  107546:161331  [11,24,35]  11  [11,24,35]  11
>>  0'0  2024-01-25T08:19:04.148837+0200   0'0
>> 2024-01-12T06:06:00.970665+0200  0
>>
>>
>> On Fri, Jan 26, 2024 at 8:58 AM E Taka <0eta...@gmail.com> wrote:
>>
>>> We had the same problem. It turned out that one disk was slowly dying.
>>> It was easy to identify by the commands (in your case):
>>>
>>> ceph pg dump | grep -F 6.78
>>> ceph pg dump | grep -F 6.60
>>> …
>>>
>>> This command shows the OSDs of a PG in square brackets. If is there
>>> always the same number, then you've found the OSD which causes the slow
>>> scrubs.
>>>
>>> Am Fr., 26. Jan. 

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[E Taka]

22 is more often there than the others. Other operations may be blocked
because of a deep-scrub is not finished yet. I would remove OSD 22, just to
be sure about this: ceph orch osd rm osd.22

If this does not help, just add it again.

Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
mico...@gmail.com>:

> It seems that are different OSDs as shown here . how have you managed to
> sort this out?
>
> ceph pg dump | grep -F 6.78
> dumped all
> 6.78   44268   0 0  00
> 1786796401180   0  10099 10099
>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
> 107547:225274427  [12,36,37]  12  [12,36,37]  12
> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
> 2024-01-11T16:07:54.875746+0200  0
> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
> dumped all
> 6.60   9   0 0  00
> 179484338742  716  36  10097 10097
>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
> 107547:287193139   [32,5,29]  32   [32,5,29]  32
> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
> 2024-01-13T19:44:26.922000+0200  0
> 6.3a   44807   0 0  00
> 1809690056940   0  10093 10093
>  active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
> 107547:238170093  [22,13,11]  22  [22,13,11]  22
> 106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
> 2024-01-15T05:31:36.606478+0200  0
> root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
> 6.5c   44277   0 0  00
> 1787649782300   0  10051 10051
>  active+clean  2024-01-26T03:55:23.339584+0200  107547'126480090
> 107547:264432655  [22,37,30]  22  [22,37,30]  22
> 107205'125858697  2024-01-24T22:32:10.365869+0200  101941'120957992
> 2024-01-13T09:07:24.780936+0200  0
> dumped all
> root@ceph-osd3:~# ceph pg dump | grep -F 4.12
> dumped all
> 4.12   0   0 0  00
>  00   0  0 0
>  active+clean  2024-01-24T08:36:48.284388+0200   0'0
>  107546:152711   [22,19,7]  22   [22,19,7]  22
>  0'0  2024-01-24T08:36:48.284307+0200   0'0
> 2024-01-13T09:09:22.176240+0200  0
> root@ceph-osd3:~# ceph pg dump | grep -F 10.d
> dumped all
> 10.d   0   0 0  00
>  00   0  0 0
>  active+clean  2024-01-24T04:04:33.641541+0200   0'0
>  107546:142651   [14,28,1]  14   [14,28,1]  14
>  0'0  2024-01-24T04:04:33.641451+0200   0'0
> 2024-01-12T08:04:02.078062+0200  0
> root@ceph-osd3:~# ceph pg dump | grep -F 5.f
> dumped all
> 5.f0   0 0  00
>  00   0  0 0
>  active+clean  2024-01-25T08:19:04.148941+0200   0'0
>  107546:161331  [11,24,35]  11  [11,24,35]  11
>  0'0  2024-01-25T08:19:04.148837+0200   0'0
> 2024-01-12T06:06:00.970665+0200  0
>
>
> On Fri, Jan 26, 2024 at 8:58 AM E Taka <0eta...@gmail.com> wrote:
>
>> We had the same problem. It turned out that one disk was slowly dying. It
>> was easy to identify by the commands (in your case):
>>
>> ceph pg dump | grep -F 6.78
>> ceph pg dump | grep -F 6.60
>> …
>>
>> This command shows the OSDs of a PG in square brackets. If is there
>> always the same number, then you've found the OSD which causes the slow
>> scrubs.
>>
>> Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita <
>> mico...@gmail.com>:
>>
>>> Hello team,
>>>
>>> I have a cluster in production composed by  3 osds servers with 20 disks
>>> each deployed using ceph-ansibleand ubuntu OS , and the version is
>>> pacific
>>> . These days is in WARN state caused by pgs which are not deep-scrubbed
>>> in
>>> time . I tried to deep-scrubbed some pg manually but seems that the
>>> cluster
>>> can be slow, would like your assistance in order that my cluster can be
>>> in
>>> HEALTH_OK state as before without any interuption of service . The
>>> cluster
>>> is used as openstack backend storage.
>>>
>>> Best Regards
>>>
>>> Michel
>>>
>>>
>>>  ceph -s
>>>   cluster:
>>> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
>>> health: HEALTH_WARN
>>> 6 pgs not deep-scrubbed in time
>>>
>>>   services:
>>> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
>>> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
>>> osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
>>> rgw: 6 daemons active (6 hosts, 1 zones)
>>>
>>>   data:
>>> pools:   10 pools, 385 

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

It seems that are different OSDs as shown here . how have you managed to
sort this out?

ceph pg dump | grep -F 6.78
dumped all
6.78   44268   0 0  00
1786796401180   0  10099 10099
 active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
107547:225274427  [12,36,37]  12  [12,36,37]  12
106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
2024-01-11T16:07:54.875746+0200  0
root@ceph-osd3:~# ceph pg dump | grep -F 6.60
dumped all
6.60   9   0 0  00
179484338742  716  36  10097 10097
 active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
107547:287193139   [32,5,29]  32   [32,5,29]  32
107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
2024-01-13T19:44:26.922000+0200  0
6.3a   44807   0 0  00
1809690056940   0  10093 10093
 active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
107547:238170093  [22,13,11]  22  [22,13,11]  22
106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
2024-01-15T05:31:36.606478+0200  0
root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
6.5c   44277   0 0  00
1787649782300   0  10051 10051
 active+clean  2024-01-26T03:55:23.339584+0200  107547'126480090
107547:264432655  [22,37,30]  22  [22,37,30]  22
107205'125858697  2024-01-24T22:32:10.365869+0200  101941'120957992
2024-01-13T09:07:24.780936+0200  0
dumped all
root@ceph-osd3:~# ceph pg dump | grep -F 4.12
dumped all
4.12   0   0 0  00
   00   0  0 0
 active+clean  2024-01-24T08:36:48.284388+0200   0'0
 107546:152711   [22,19,7]  22   [22,19,7]  22
 0'0  2024-01-24T08:36:48.284307+0200   0'0
2024-01-13T09:09:22.176240+0200  0
root@ceph-osd3:~# ceph pg dump | grep -F 10.d
dumped all
10.d   0   0 0  00
   00   0  0 0
 active+clean  2024-01-24T04:04:33.641541+0200   0'0
 107546:142651   [14,28,1]  14   [14,28,1]  14
 0'0  2024-01-24T04:04:33.641451+0200   0'0
2024-01-12T08:04:02.078062+0200  0
root@ceph-osd3:~# ceph pg dump | grep -F 5.f
dumped all
5.f0   0 0  00
   00   0  0 0
 active+clean  2024-01-25T08:19:04.148941+0200   0'0
 107546:161331  [11,24,35]  11  [11,24,35]  11
 0'0  2024-01-25T08:19:04.148837+0200   0'0
2024-01-12T06:06:00.970665+0200  0


On Fri, Jan 26, 2024 at 8:58 AM E Taka <0eta...@gmail.com> wrote:

> We had the same problem. It turned out that one disk was slowly dying. It
> was easy to identify by the commands (in your case):
>
> ceph pg dump | grep -F 6.78
> ceph pg dump | grep -F 6.60
> …
>
> This command shows the OSDs of a PG in square brackets. If is there always
> the same number, then you've found the OSD which causes the slow scrubs.
>
> Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita <
> mico...@gmail.com>:
>
>> Hello team,
>>
>> I have a cluster in production composed by  3 osds servers with 20 disks
>> each deployed using ceph-ansibleand ubuntu OS , and the version is pacific
>> . These days is in WARN state caused by pgs which are not deep-scrubbed in
>> time . I tried to deep-scrubbed some pg manually but seems that the
>> cluster
>> can be slow, would like your assistance in order that my cluster can be in
>> HEALTH_OK state as before without any interuption of service . The cluster
>> is used as openstack backend storage.
>>
>> Best Regards
>>
>> Michel
>>
>>
>>  ceph -s
>>   cluster:
>> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
>> health: HEALTH_WARN
>> 6 pgs not deep-scrubbed in time
>>
>>   services:
>> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
>> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
>> osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
>> rgw: 6 daemons active (6 hosts, 1 zones)
>>
>>   data:
>> pools:   10 pools, 385 pgs
>> objects: 5.97M objects, 23 TiB
>> usage:   151 TiB used, 282 TiB / 433 TiB avail
>> pgs: 381 active+clean
>>  4   active+clean+scrubbing+deep
>>
>>   io:
>> client:   59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
>>
>> root@ceph-osd3:~# ceph health detail
>> HEALTH_WARN 6 pgs not deep-scrubbed in time
>> [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time
>> pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200
>> pg 6.60 not 

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[E Taka]

We had the same problem. It turned out that one disk was slowly dying. It
was easy to identify by the commands (in your case):

ceph pg dump | grep -F 6.78
ceph pg dump | grep -F 6.60
…

This command shows the OSDs of a PG in square brackets. If is there always
the same number, then you've found the OSD which causes the slow scrubs.

Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita <
mico...@gmail.com>:

> Hello team,
>
> I have a cluster in production composed by  3 osds servers with 20 disks
> each deployed using ceph-ansibleand ubuntu OS , and the version is pacific
> . These days is in WARN state caused by pgs which are not deep-scrubbed in
> time . I tried to deep-scrubbed some pg manually but seems that the cluster
> can be slow, would like your assistance in order that my cluster can be in
> HEALTH_OK state as before without any interuption of service . The cluster
> is used as openstack backend storage.
>
> Best Regards
>
> Michel
>
>
>  ceph -s
>   cluster:
> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
> health: HEALTH_WARN
> 6 pgs not deep-scrubbed in time
>
>   services:
> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
> osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
> rgw: 6 daemons active (6 hosts, 1 zones)
>
>   data:
> pools:   10 pools, 385 pgs
> objects: 5.97M objects, 23 TiB
> usage:   151 TiB used, 282 TiB / 433 TiB avail
> pgs: 381 active+clean
>  4   active+clean+scrubbing+deep
>
>   io:
> client:   59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
>
> root@ceph-osd3:~# ceph health detail
> HEALTH_WARN 6 pgs not deep-scrubbed in time
> [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time
> pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200
> pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200
> pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200
> pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200
> pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200
> pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io