Re: [ceph-users] Ceph is moving data ONLY to near-full OSDs [BUG]

2019-10-28 Thread Philippe D'Anjou 
 I was following the pg autoscaler recommendations and I did not get a 
recommendation to raise the PGs there.
I'll try that, I am raising it already. But still seems weird why it would move 
data onto almost full OSDs, see the data distribution, it's horrible, ranges 
from 60 to almost 90% of full. PGs are not equally distributed otherwise it'd 
be a PG size issue.

Thanks

Am Sonntag, 27. Oktober 2019, 20:33:11 OEZ hat Wido den Hollander 
 Folgendes geschrieben:  
 
 

On 10/26/19 8:01 AM, Philippe D'Anjou wrote:
> V14.2.4
> So, this is not new, this happens every time there is a rebalance, now
> because of raising PGs. PG balancer is disabled because I thought it was
> the reason but apparently it's not, but it ain't helping either.
> 
> Ceph is totally borged, it's only moving data on nearfull OSDs causing
> issues. See this after PG raise.
> 
>     health: HEALTH_WARN
>     3 nearfull osd(s)
>     2 pool(s) nearfull
> 
> 
> 08:44 am
> 
> 92   hdd  9.09470  1.0 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.79 1.25  39 up
> 71   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.09 1.27  38 up
> 21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.5
> TiB 84.05 1.27  36 up
>  
>  
> 08:54 am
> 
> 92   hdd  9.09470  1.0 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.81 1.25  39 up
> 71   hdd  9.09470  1.0 9.1 TiB  7.7 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.14 1.27  38 up
> 21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.4
> TiB 84.10 1.27  36 up
>    
>    
>    
>  14   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  76 KiB  17 GiB
> 2.6 TiB 71.33 1.07  32 up
>  19   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.81 1.04  30 up
>  22   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  92 KiB  17 GiB
> 2.8 TiB 68.90 1.04  32 up
>  25   hdd  9.09470  1.0 9.1 TiB  6.2 TiB  6.2 TiB 219 KiB  17 GiB
> 2.9 TiB 68.11 1.03  31 up
>  30   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  20 KiB  17 GiB
> 2.6 TiB 71.41 1.08  33 up
>  33   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  40 KiB  17 GiB
> 2.6 TiB 71.30 1.07  32 up
>  34   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.33 1.07  30 up
>  35   hdd  9.09470  1.0 9.1 TiB  6.6 TiB  6.6 TiB 124 KiB  17 GiB
> 2.5 TiB 72.61 1.09  32 up
>  12   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  32 up
>  16   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.4 TiB  96 KiB  17 GiB
> 2.6 TiB 71.08 1.07  29 up
>  17   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  60 KiB  17 GiB
> 2.6 TiB 71.41 1.08  31 up
>  20   hdd  9.09470  1.0 9.1 TiB  6.2 TiB  6.2 TiB  92 KiB  17 GiB
> 2.9 TiB 68.57 1.03  28 up
>  23   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.37 1.08  29 up
>  26   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  84 KiB  17 GiB
> 2.7 TiB 70.02 1.06  30 up
>  28   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  28 KiB  17 GiB
> 2.7 TiB 70.11 1.06  30 up
>  31   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  56 KiB  17 GiB
> 2.6 TiB 71.26 1.07  32 up
>  13   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  31 up
>  15   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  44 KiB  17 GiB
> 2.6 TiB 71.35 1.08  29 up
>  18   hdd  9.09470  1.0 9.1 TiB  5.8 TiB  5.8 TiB  76 KiB  16 GiB
> 3.3 TiB 63.70 0.96  26 up
>  21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB
> 1.4 TiB 84.10 1.27  36 up
>  24   hdd  9.09470  1.0 9.1 TiB  5.8 TiB  5.8 TiB  64 KiB  15 GiB
> 3.3 TiB 63.67 0.96  28 up
>  27   hdd  9.09470  1.0 9.1 TiB  6.0 TiB  6.0 TiB  48 KiB  17 GiB
> 3.1 TiB 66.03 1.00  28 up
>  29   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.3 TiB  28 KiB  18 GiB
> 2.7 TiB 69.93 1.05  34 up
>  32   hdd  9.09470  1.0 9.1 TiB  6.0 TiB  6.0 TiB  20 KiB  17 GiB
> 3.1 TiB 66.20 1.00  28 up
>  37   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  18 GiB
> 2.7 TiB 70.59 1.06  31 up
>  39   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  19 GiB
> 2.7 TiB 70.50 1.06  29 up
>  41   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.79 1.04  30 up
>  43   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  48 KiB  17 GiB
> 2.8 TiB 68.84 1.04  28 up
>  45   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  80 KiB  18 GiB
> 2.4 TiB 74.02 1.12  33 up
>  46   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  36 KiB  18 GiB
> 2.4 TiB 73.88 1.11  30 up
>  48   hdd  9.09470  1.0 9.1 TiB  6.6 TiB  6.6 TiB 101 KiB  17 GiB
> 2.5 TiB 72.57 1.09  31 up
>  50   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  96 KiB  17 GiB
> 2.8 TiB 68.86 1.04  31 up
>  36   hdd  9.09470  1.0 9.1 TiB 

Re: [ceph-users] Ceph is moving data ONLY to near-full OSDs [BUG]

2019-10-27 Thread Wido den Hollander 


On 10/26/19 8:01 AM, Philippe D'Anjou wrote:
> V14.2.4
> So, this is not new, this happens every time there is a rebalance, now
> because of raising PGs. PG balancer is disabled because I thought it was
> the reason but apparently it's not, but it ain't helping either.
> 
> Ceph is totally borged, it's only moving data on nearfull OSDs causing
> issues. See this after PG raise.
> 
>     health: HEALTH_WARN
>     3 nearfull osd(s)
>     2 pool(s) nearfull
> 
> 
> 08:44 am
> 
> 92   hdd  9.09470  1.0 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.79 1.25  39 up
> 71   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.09 1.27  38 up
> 21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.5
> TiB 84.05 1.27  36 up
>  
>  
> 08:54 am
> 
> 92   hdd  9.09470  1.0 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.81 1.25  39 up
> 71   hdd  9.09470  1.0 9.1 TiB  7.7 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.14 1.27  38 up
> 21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.4
> TiB 84.10 1.27  36 up
>    
>    
>    
>  14   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  76 KiB  17 GiB
> 2.6 TiB 71.33 1.07  32 up
>  19   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.81 1.04  30 up
>  22   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  92 KiB  17 GiB
> 2.8 TiB 68.90 1.04  32 up
>  25   hdd  9.09470  1.0 9.1 TiB  6.2 TiB  6.2 TiB 219 KiB  17 GiB
> 2.9 TiB 68.11 1.03  31 up
>  30   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  20 KiB  17 GiB
> 2.6 TiB 71.41 1.08  33 up
>  33   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  40 KiB  17 GiB
> 2.6 TiB 71.30 1.07  32 up
>  34   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.33 1.07  30 up
>  35   hdd  9.09470  1.0 9.1 TiB  6.6 TiB  6.6 TiB 124 KiB  17 GiB
> 2.5 TiB 72.61 1.09  32 up
>  12   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  32 up
>  16   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.4 TiB  96 KiB  17 GiB
> 2.6 TiB 71.08 1.07  29 up
>  17   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  60 KiB  17 GiB
> 2.6 TiB 71.41 1.08  31 up
>  20   hdd  9.09470  1.0 9.1 TiB  6.2 TiB  6.2 TiB  92 KiB  17 GiB
> 2.9 TiB 68.57 1.03  28 up
>  23   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.37 1.08  29 up
>  26   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  84 KiB  17 GiB
> 2.7 TiB 70.02 1.06  30 up
>  28   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  28 KiB  17 GiB
> 2.7 TiB 70.11 1.06  30 up
>  31   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  56 KiB  17 GiB
> 2.6 TiB 71.26 1.07  32 up
>  13   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  31 up
>  15   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  44 KiB  17 GiB
> 2.6 TiB 71.35 1.08  29 up
>  18   hdd  9.09470  1.0 9.1 TiB  5.8 TiB  5.8 TiB  76 KiB  16 GiB
> 3.3 TiB 63.70 0.96  26 up
>  21   hdd  9.09470  1.0 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB
> 1.4 TiB 84.10 1.27  36 up
>  24   hdd  9.09470  1.0 9.1 TiB  5.8 TiB  5.8 TiB  64 KiB  15 GiB
> 3.3 TiB 63.67 0.96  28 up
>  27   hdd  9.09470  1.0 9.1 TiB  6.0 TiB  6.0 TiB  48 KiB  17 GiB
> 3.1 TiB 66.03 1.00  28 up
>  29   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.3 TiB  28 KiB  18 GiB
> 2.7 TiB 69.93 1.05  34 up
>  32   hdd  9.09470  1.0 9.1 TiB  6.0 TiB  6.0 TiB  20 KiB  17 GiB
> 3.1 TiB 66.20 1.00  28 up
>  37   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  18 GiB
> 2.7 TiB 70.59 1.06  31 up
>  39   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  19 GiB
> 2.7 TiB 70.50 1.06  29 up
>  41   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.79 1.04  30 up
>  43   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  48 KiB  17 GiB
> 2.8 TiB 68.84 1.04  28 up
>  45   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  80 KiB  18 GiB
> 2.4 TiB 74.02 1.12  33 up
>  46   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  36 KiB  18 GiB
> 2.4 TiB 73.88 1.11  30 up
>  48   hdd  9.09470  1.0 9.1 TiB  6.6 TiB  6.6 TiB 101 KiB  17 GiB
> 2.5 TiB 72.57 1.09  31 up
>  50   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB  96 KiB  17 GiB
> 2.8 TiB 68.86 1.04  31 up
>  36   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB  60 KiB  18 GiB
> 2.4 TiB 73.82 1.11  32 up
>  38   hdd  9.09470  1.0 9.1 TiB  6.4 TiB  6.4 TiB  76 KiB  17 GiB
> 2.7 TiB 70.62 1.06  33 up
>  40   hdd  9.09470  1.0 9.1 TiB  6.7 TiB  6.7 TiB   8 KiB  18 GiB
> 2.4 TiB 73.89 1.11  33 up
>  42   hdd  9.09470  1.0 9.1 TiB  6.3 TiB  6.2 TiB 120 KiB  17 GiB
> 2.8 TiB 68.82 1.04  28 up
>  44   hdd  9.09470  1.0 9.1 TiB  6.5 TiB  6.5 TiB  68 KiB  17 GiB
> 2.6 TiB 71.39 1.08  29