Hi,this is a fresh Nautilus cluster, but there is a second old one that was upgraded from Luminous to Nautilus, both experience the same symptoms. First of all the data distribution on the OSDs is very bad. Now that could be due to low PGs although I get no recommendation to raise the PG number so far.
14 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 68 KiB 15 GiB 2.9 TiB 67.83 1.07 32 up 19 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 56 KiB 16 GiB 2.4 TiB 73.56 1.16 35 up 22 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 88 KiB 15 GiB 2.8 TiB 69.39 1.09 32 up 25 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 88 KiB 16 GiB 3.1 TiB 66.01 1.04 31 up 30 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 84 KiB 16 GiB 2.8 TiB 69.36 1.09 33 up 33 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 15 GiB 3.1 TiB 65.84 1.03 31 up 34 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB 14 GiB 3.3 TiB 63.84 1.00 29 up 35 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 32 KiB 15 GiB 3.1 TiB 65.82 1.03 31 up 12 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 44 KiB 15 GiB 3.0 TiB 67.32 1.06 31 up 16 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 72 KiB 17 GiB 2.4 TiB 73.52 1.16 35 up 17 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 80 KiB 15 GiB 3.0 TiB 67.24 1.06 32 up 20 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 64 KiB 17 GiB 2.3 TiB 74.45 1.17 35 up 23 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 80 KiB 16 GiB 2.6 TiB 71.46 1.12 34 up 26 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 16 GiB 2.7 TiB 70.45 1.11 33 up 28 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 64 KiB 15 GiB 3.0 TiB 66.52 1.05 31 up 31 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 52 KiB 16 GiB 2.5 TiB 72.21 1.13 34 up 13 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 68 KiB 16 GiB 2.4 TiB 73.56 1.16 35 up 15 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 44 KiB 17 GiB 2.4 TiB 73.58 1.16 35 up 18 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 16 GiB 2.6 TiB 71.46 1.12 34 up 21 hdd 8.00000 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 84 KiB 15 GiB 3.2 TiB 65.14 1.02 31 up 24 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 76 KiB 16 GiB 2.5 TiB 72.21 1.13 34 up 27 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 64 KiB 16 GiB 2.6 TiB 71.42 1.12 34 up 29 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 80 KiB 15 GiB 2.9 TiB 68.16 1.07 32 up 32 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 132 KiB 15 GiB 3.0 TiB 66.91 1.05 31 up 37 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 16 KiB 17 GiB 2.8 TiB 69.38 1.09 33 up 39 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 92 KiB 16 GiB 2.7 TiB 70.44 1.11 33 up 41 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 108 KiB 15 GiB 2.9 TiB 67.95 1.07 32 up 43 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 24 KiB 15 GiB 3.2 TiB 65.20 1.02 31 up 45 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 72 KiB 15 GiB 3.2 TiB 65.35 1.03 31 up 46 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 60 KiB 15 GiB 2.9 TiB 68.08 1.07 32 up 48 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 64 KiB 15 GiB 2.9 TiB 67.96 1.07 32 up 50 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 48 KiB 15 GiB 3.4 TiB 63.06 0.99 30 up 36 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 116 KiB 15 GiB 3.0 TiB 67.45 1.06 32 up 38 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 80 KiB 16 GiB 2.8 TiB 69.36 1.09 33 up 40 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 84 KiB 15 GiB 3.0 TiB 67.31 1.06 32 up 42 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 104 KiB 15 GiB 3.0 TiB 66.59 1.05 31 up 44 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 15 GiB 3.2 TiB 65.12 1.02 31 up 47 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 88 KiB 15 GiB 3.2 TiB 65.09 1.02 31 up 49 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB 15 GiB 3.0 TiB 67.19 1.06 32 up 51 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 40 KiB 15 GiB 2.9 TiB 67.75 1.06 32 up 52 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 80 KiB 15 GiB 3.3 TiB 63.91 1.00 30 up 53 hdd 7.00000 1.00000 9.1 TiB 7.3 TiB 7.2 TiB 88 KiB 18 GiB 1.8 TiB 79.84 1.25 38 up 54 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 96 KiB 14 GiB 3.3 TiB 63.67 1.00 30 up 55 hdd 9.09470 1.00000 9.1 TiB 5.3 TiB 5.3 TiB 48 KiB 13 GiB 3.8 TiB 58.20 0.91 27 up 56 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 64 KiB 16 GiB 2.8 TiB 69.35 1.09 33 up 57 hdd 6.00000 1.00000 9.1 TiB 4.2 TiB 4.2 TiB 36 KiB 11 GiB 4.9 TiB 46.27 0.73 22 up 58 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB 15 GiB 2.9 TiB 68.20 1.07 32 up 59 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.3 TiB 92 KiB 16 GiB 2.7 TiB 69.99 1.10 33 up 60 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 108 KiB 15 GiB 3.0 TiB 67.18 1.06 32 up 61 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB 16 GiB 2.8 TiB 69.34 1.09 33 up 62 hdd 9.09470 1.00000 9.1 TiB 3.9 TiB 3.9 TiB 32 KiB 10 GiB 5.2 TiB 42.98 0.68 20 up 63 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 108 KiB 16 GiB 2.8 TiB 69.62 1.09 33 up 64 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 64 KiB 16 GiB 3.0 TiB 67.26 1.06 32 up 65 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB 2.6 TiB 71.46 1.12 34 up 66 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 80 KiB 16 GiB 2.6 TiB 71.37 1.12 34 up 67 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 48 KiB 15 GiB 2.9 TiB 68.60 1.08 32 up 84 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 68 KiB 16 GiB 2.8 TiB 69.40 1.09 33 up 85 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 116 KiB 15 GiB 2.8 TiB 68.80 1.08 32 up 86 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 48 KiB 15 GiB 3.2 TiB 64.30 1.01 30 up 87 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 125 KiB 15 GiB 3.0 TiB 67.23 1.06 32 up 88 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.8 TiB 68 KiB 15 GiB 3.2 TiB 64.41 1.01 30 up 89 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB 14 GiB 3.4 TiB 63.11 0.99 30 up 90 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 96 KiB 15 GiB 2.9 TiB 68.02 1.07 32 up 91 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 68 KiB 17 GiB 2.1 TiB 76.43 1.20 36 up 76 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 84 KiB 16 GiB 2.5 TiB 72.40 1.14 34 up 77 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.5 TiB 56 KiB 17 GiB 2.5 TiB 72.06 1.13 34 up 78 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.3 TiB 132 KiB 16 GiB 2.7 TiB 69.98 1.10 33 up 79 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 32 KiB 18 GiB 2.4 TiB 73.50 1.16 35 up 80 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.4 TiB 76 KiB 14 GiB 3.7 TiB 59.80 0.94 28 up 81 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 44 KiB 15 GiB 2.9 TiB 68.20 1.07 32 up 82 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 68 KiB 16 GiB 2.8 TiB 69.34 1.09 33 up 83 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 124 KiB 15 GiB 3.1 TiB 65.92 1.04 31 up 68 hdd 9.09470 1.00000 9.1 TiB 3.6 TiB 3.6 TiB 104 KiB 8.8 GiB 5.5 TiB 39.45 0.62 15 up 69 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 56 KiB 15 GiB 3.0 TiB 67.43 1.06 31 up 70 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 100 KiB 17 GiB 2.1 TiB 77.39 1.22 36 up 71 hdd 7.50000 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 52 KiB 19 GiB 1.8 TiB 80.73 1.27 38 up 72 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 52 KiB 16 GiB 2.8 TiB 69.28 1.09 33 up 73 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.5 TiB 100 KiB 18 GiB 1.5 TiB 83.02 1.30 39 up 74 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 72 KiB 15 GiB 3.0 TiB 66.91 1.05 31 up 75 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.8 TiB 92 KiB 14 GiB 3.2 TiB 64.41 1.01 29 up 112 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 36 KiB 14 GiB 3.2 TiB 64.28 1.01 30 up 113 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 64 KiB 17 GiB 2.1 TiB 76.47 1.20 36 up 114 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 128 KiB 16 GiB 2.3 TiB 74.86 1.18 35 up 115 hdd 7.00000 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 96 KiB 16 GiB 2.7 TiB 70.35 1.11 33 up 116 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 84 KiB 16 GiB 2.4 TiB 73.84 1.16 35 up 117 hdd 5.00000 1.00000 9.1 TiB 7.7 TiB 7.7 TiB 64 KiB 18 GiB 1.4 TiB 84.55 1.33 40 up 118 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 116 KiB 14 GiB 3.1 TiB 65.59 1.03 30 up 119 hdd 9.09470 1.00000 9.1 TiB 7.1 TiB 7.0 TiB 108 KiB 17 GiB 2.0 TiB 77.52 1.22 36 up 92 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 16 GiB 2.6 TiB 71.51 1.12 34 up 93 hdd 10.00000 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 60 KiB 15 GiB 2.8 TiB 69.58 1.09 32 up 94 hdd 10.00000 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 92 KiB 16 GiB 2.5 TiB 72.79 1.14 34 up 95 hdd 10.00000 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 100 KiB 17 GiB 2.0 TiB 77.71 1.22 37 up 96 hdd 10.00000 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 32 KiB 16 GiB 2.4 TiB 73.37 1.15 34 up 97 hdd 10.00000 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 76 KiB 17 GiB 2.2 TiB 76.35 1.20 36 up STDDEV: 24.83 Second, and biggest issue is that whenever we add new storage to the cluster ceph begins to move data from the old OSDs to other old OSDs which causes them to fill up dangerously high. Then I have to either reweight them or kill them completely to avoid a full OSD. The problem is that reweight is ignored too very often, even halfing the weight doesn't change anything, the OSD just fills up more and more during rebalance/recovery. So in a nutshell: I fill up the tank in my car and when I leave the station my fuel gauge says its empty. That's what happens if we add new storage, starting a whack-a-mole game of reweight OSDs to avoid further issues. Pretty much using all default values for everything, no special OSD configs. I reported this but I was told to ask here. Thanks for any hints. ps. never happend under LuminousPG autoscaler is on recommendation mode only
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
