Re: [ceph-users] Issues with data distribution on Nautilus / weird filling behavior

2019-10-15 Thread Philippe D'Anjou 
 
Still seeing this issue. I had equal distribution but now copying data on 
cephfs and now 1 OSD is acting up again.
... 52   hdd  9.09470  1.0 9.1 TiB 5.4 TiB 5.4 TiB  84 KiB  14 GiB 3.7 TiB 
59.32 1.00  26 up 
 53   hdd  8.0  1.0 9.1 TiB 7.9 TiB 7.9 TiB  64 KiB  19 GiB 1.2 TiB 
86.83 1.46  38 up 
 54   hdd  9.09470  1.0 9.1 TiB 5.0 TiB 5.0 TiB 136 KiB  13 GiB 4.1 TiB 
54.80 0.92  24 up 
...
Now I again have to manually reweight to prevent bigger issues.
How to fix this? 
Am Mittwoch, 2. Oktober 2019, 08:49:50 OESZ hat Philippe D'Anjou 
 Folgendes geschrieben:  
 
 Hi,this is a fresh Nautilus cluster, but there is a second old one that was 
upgraded from Luminous to Nautilus, both experience the same symptoms.
First of all the data distribution on the OSDs is very bad. Now that could be 
due to low PGs although I get no recommendation to raise the PG number so far.

 14   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  68 KiB  15 GiB 2.9 TiB 
67.83 1.07  32 up 
 19   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 22   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  88 KiB  15 GiB 2.8 TiB 
69.39 1.09  32 up 
 25   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  88 KiB  16 GiB 3.1 TiB 
66.01 1.04  31 up 
 30   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 33   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  15 GiB 3.1 TiB 
65.84 1.03  31 up 
 34   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB  14 GiB 3.3 TiB 
63.84 1.00  29 up 
 35   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  32 KiB  15 GiB 3.1 TiB 
65.82 1.03  31 up 
 12   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB  15 GiB 3.0 TiB 
67.32 1.06  31 up 
 16   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  72 KiB  17 GiB 2.4 TiB 
73.52 1.16  35 up 
 17   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  80 KiB  15 GiB 3.0 TiB 
67.24 1.06  32 up 
 20   hdd  9.09470  1.0 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  17 GiB 2.3 TiB 
74.45 1.17  35 up 
 23   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  80 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 26   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  16 GiB 2.7 TiB 
70.45 1.11  33 up 
 28   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB  64 KiB  15 GiB 3.0 TiB 
66.52 1.05  31 up 
 31   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  52 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 13   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  68 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 15   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  44 KiB  17 GiB 2.4 TiB 
73.58 1.16  35 up 
 18   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  56 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 21   hdd  8.0  1.0 9.1 TiB 5.9 TiB 5.9 TiB  84 KiB  15 GiB 3.2 TiB 
65.14 1.02  31 up 
 24   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  76 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 27   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  16 GiB 2.6 TiB 
71.42 1.12  34 up 
 29   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  15 GiB 2.9 TiB 
68.16 1.07  32 up 
 32   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 132 KiB  15 GiB 3.0 TiB 
66.91 1.05  31 up 
 37   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  16 KiB  17 GiB 2.8 TiB 
69.38 1.09  33 up 
 39   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  16 GiB 2.7 TiB 
70.44 1.11  33 up 
 41   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB 108 KiB  15 GiB 2.9 TiB 
67.95 1.07  32 up 
 43   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  24 KiB  15 GiB 3.2 TiB 
65.20 1.02  31 up 
 45   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  72 KiB  15 GiB 3.2 TiB 
65.35 1.03  31 up 
 46   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  60 KiB  15 GiB 2.9 TiB 
68.08 1.07  32 up 
 48   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  64 KiB  15 GiB 2.9 TiB 
67.96 1.07  32 up 
 50   hdd  9.09470  1.0 9.1 TiB 5.7 TiB 5.7 TiB  48 KiB  15 GiB 3.4 TiB 
63.06 0.99  30 up 
 36   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 116 KiB  15 GiB 3.0 TiB 
67.45 1.06  32 up 
 38   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 40   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  84 KiB  15 GiB 3.0 TiB 
67.31 1.06  32 up 
 42   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB 104 KiB  15 GiB 3.0 TiB 
66.59 1.05  31 up 
 44   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  15 GiB 3.2 TiB 
65.12 1.02  31 up 
 47   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  88 KiB  15 GiB 3.2 TiB 
65.09 1.02  31 up 
 49   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  15 GiB 3.0 TiB 
67.19 1.06  32 up 
 51   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.1 TiB  40 KiB  15 GiB 2.9 TiB 
67.75 1.06  32 up 
 52   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB  80 KiB  15 GiB 3.3 TiB 
63.91 1.0

[ceph-users] Issues with data distribution on Nautilus / weird filling behavior

2019-10-01 Thread Philippe D'Anjou 
Hi,this is a fresh Nautilus cluster, but there is a second old one that was 
upgraded from Luminous to Nautilus, both experience the same symptoms.
First of all the data distribution on the OSDs is very bad. Now that could be 
due to low PGs although I get no recommendation to raise the PG number so far.

 14   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  68 KiB  15 GiB 2.9 TiB 
67.83 1.07  32 up 
 19   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 22   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  88 KiB  15 GiB 2.8 TiB 
69.39 1.09  32 up 
 25   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  88 KiB  16 GiB 3.1 TiB 
66.01 1.04  31 up 
 30   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 33   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  15 GiB 3.1 TiB 
65.84 1.03  31 up 
 34   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB  14 GiB 3.3 TiB 
63.84 1.00  29 up 
 35   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  32 KiB  15 GiB 3.1 TiB 
65.82 1.03  31 up 
 12   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB  15 GiB 3.0 TiB 
67.32 1.06  31 up 
 16   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  72 KiB  17 GiB 2.4 TiB 
73.52 1.16  35 up 
 17   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  80 KiB  15 GiB 3.0 TiB 
67.24 1.06  32 up 
 20   hdd  9.09470  1.0 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  17 GiB 2.3 TiB 
74.45 1.17  35 up 
 23   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  80 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 26   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  16 GiB 2.7 TiB 
70.45 1.11  33 up 
 28   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB  64 KiB  15 GiB 3.0 TiB 
66.52 1.05  31 up 
 31   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  52 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 13   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  68 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 15   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  44 KiB  17 GiB 2.4 TiB 
73.58 1.16  35 up 
 18   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  56 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 21   hdd  8.0  1.0 9.1 TiB 5.9 TiB 5.9 TiB  84 KiB  15 GiB 3.2 TiB 
65.14 1.02  31 up 
 24   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  76 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 27   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  16 GiB 2.6 TiB 
71.42 1.12  34 up 
 29   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  15 GiB 2.9 TiB 
68.16 1.07  32 up 
 32   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 132 KiB  15 GiB 3.0 TiB 
66.91 1.05  31 up 
 37   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  16 KiB  17 GiB 2.8 TiB 
69.38 1.09  33 up 
 39   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  16 GiB 2.7 TiB 
70.44 1.11  33 up 
 41   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB 108 KiB  15 GiB 2.9 TiB 
67.95 1.07  32 up 
 43   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  24 KiB  15 GiB 3.2 TiB 
65.20 1.02  31 up 
 45   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  72 KiB  15 GiB 3.2 TiB 
65.35 1.03  31 up 
 46   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  60 KiB  15 GiB 2.9 TiB 
68.08 1.07  32 up 
 48   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  64 KiB  15 GiB 2.9 TiB 
67.96 1.07  32 up 
 50   hdd  9.09470  1.0 9.1 TiB 5.7 TiB 5.7 TiB  48 KiB  15 GiB 3.4 TiB 
63.06 0.99  30 up 
 36   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 116 KiB  15 GiB 3.0 TiB 
67.45 1.06  32 up 
 38   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 40   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  84 KiB  15 GiB 3.0 TiB 
67.31 1.06  32 up 
 42   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB 104 KiB  15 GiB 3.0 TiB 
66.59 1.05  31 up 
 44   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  15 GiB 3.2 TiB 
65.12 1.02  31 up 
 47   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  88 KiB  15 GiB 3.2 TiB 
65.09 1.02  31 up 
 49   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  15 GiB 3.0 TiB 
67.19 1.06  32 up 
 51   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.1 TiB  40 KiB  15 GiB 2.9 TiB 
67.75 1.06  32 up 
 52   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB  80 KiB  15 GiB 3.3 TiB 
63.91 1.00  30 up 
 53   hdd  7.0  1.0 9.1 TiB 7.3 TiB 7.2 TiB  88 KiB  18 GiB 1.8 TiB 
79.84 1.25  38 up 
 54   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB  96 KiB  14 GiB 3.3 TiB 
63.67 1.00  30 up 
 55   hdd  9.09470  1.0 9.1 TiB 5.3 TiB 5.3 TiB  48 KiB  13 GiB 3.8 TiB 
58.20 0.91  27 up 
 56   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB  16 GiB 2.8 TiB 
69.35 1.09  33 up 
 57   hdd  6.0  1.0 9.1 TiB 4.2 TiB 4.2 TiB  36 KiB  11 GiB 4.9 TiB 
46.27 0.73  22 up 
 58   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  15 GiB 2.9 TiB 
68.20 1.07  32