I had a 5 Node EC (8+3) pool of 6TB disks (some with 12 SAS per a host, some 14 
SAS per a host), with the 5 node’s I had the crush map rule set to 3 OSD across 
4 HOST, this worked perfectly and I ended up with a perfect balance of PG’s and 
disk space % across all the hosts/OSD.

I recently added a 6th HOST with 12 SAS, so I changed the rule to 6 HOST, 2 
OSD, whilst I don’t expect as exactly the same perfect balance, I now have some 
OSD at 50% and some at 80% full, is this because my rule is set to the max 
amount of hosts I have so it’s not able to complete the best balance?

"ceph balance status" shows "Unable to find further optimization, or pool(s) 
pg_num is decreasing, or distribution is already perfect” ,I have also tried to 
run the map manually via osdmaptool with various deviation values but get the 
same 0 changes.

Do I need to change my rule back to 3 OSD 5 HOSTto be able to allow crush to 
get a better balance?

I have attached hopefully some useful outputs as a txt file.

Thanks
ceph osd df
ID    CLASS    WEIGHT      REWEIGHT    SIZE        
  RAW USE    DATA          OMAP          META    
      AVAIL        %USE      VAR      PGS    
STATUS
12        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB    9.3 MiB    9.8 GiB    1.9 TiB    
66.01    1.01      76            up
13        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB      21 MiB      13 GiB    1.8 TiB  
  66.95    1.03      79            up
14        hdd    5.45798      1.00000    5.5 TiB    
4.0 TiB    4.0 TiB      35 MiB      12 GiB    1.4 TiB  
  74.14    1.14      86            up
15        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      10 MiB      13 GiB    1.8 TiB  
  66.12    1.01      76            up
16        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB      20 MiB      13 GiB    1.8 TiB  
  66.92    1.02      78            up
17        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB    9.7 MiB      14 GiB    1.8 TiB    
66.94    1.03      77            up
18        hdd    5.45798      1.00000    5.5 TiB    
3.8 TiB    3.8 TiB      28 MiB      14 GiB    1.7 TiB  
  69.69    1.07      81            up
19        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.7 TiB    9.9 MiB    9.9 GiB    1.8 TiB    
67.73    1.04      79            up
20        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB    9.2 MiB      13 GiB    1.8 TiB    
66.96    1.03      78            up
21        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB    9.2 MiB      14 GiB    1.8 TiB    
66.93    1.03      77            up
22        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB    8.4 MiB      13 GiB    1.9 TiB    
66.01    1.01      76            up
23        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB      13 GiB    1.9 TiB  
  66.07    1.01      77            up
 0        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB      13 GiB    1.9 TiB  
  66.07    1.01      77            up
 1        hdd    5.45798      1.00000    5.5 TiB    
3.9 TiB    3.9 TiB      20 MiB      11 GiB    1.6 TiB  
  71.41    1.09      83            up
 2        hdd    5.45798      1.00000    5.5 TiB    
4.0 TiB    4.0 TiB      19 MiB      14 GiB    1.5 TiB  
  73.27    1.12      85            up
 3        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      19 MiB    9.7 GiB    1.9 TiB    
65.99    1.01      77            up
 4        hdd    5.45798      1.00000    5.5 TiB    
4.2 TiB    4.2 TiB      19 MiB      15 GiB    1.3 TiB  
  76.86    1.18      89            up
 5        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB    9.8 GiB    1.9 TiB    
65.97    1.01      77            up
 6        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.7 TiB      20 MiB      14 GiB    1.8 TiB  
  67.89    1.04      79            up
 7        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB    9.9 MiB      11 GiB    1.8 TiB    
66.89    1.02      78            up
 8        hdd    5.45798      1.00000    5.5 TiB    
4.0 TiB    3.9 TiB      21 MiB      14 GiB    1.5 TiB  
  72.37    1.11      84            up
 9        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.6 TiB    9.7 MiB      14 GiB    1.8 TiB    
66.93    1.02      77            up
10        hdd    5.45798      1.00000    5.5 TiB    
4.0 TiB    4.0 TiB      20 MiB      14 GiB    1.5 TiB  
  73.28    1.12      85            up
11        hdd    5.45798      1.00000    5.5 TiB    
3.9 TiB    3.8 TiB    9.7 MiB      14 GiB    1.6 TiB    
70.56    1.08      81            up
24        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.5 TiB      18 MiB      12 GiB    2.0 TiB  
  64.23    0.98      75            up
25        hdd    5.45798      1.00000    5.5 TiB    
3.4 TiB    3.3 TiB      20 MiB      13 GiB    2.1 TiB  
  61.52    0.94      72            up
26        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.5 TiB      18 MiB      13 GiB    1.9 TiB  
  65.09    1.00      76            up
27        hdd    5.45798      1.00000    5.5 TiB    
3.3 TiB    3.3 TiB      20 MiB      11 GiB    2.2 TiB  
  60.60    0.93      71            up
28        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      31 MiB      14 GiB    1.9 TiB  
  66.04    1.01      77            up
29        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.2 TiB    8.7 MiB    8.9 GiB    2.3 TiB    
58.78    0.90      68            up
30        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      11 MiB      14 GiB    1.9 TiB  
  66.05    1.01      76            up
31        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.5 TiB      19 MiB    9.4 GiB    1.9 TiB    
65.10    1.00      76            up
32        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.2 TiB      19 MiB      12 GiB    2.2 TiB  
  58.82    0.90      69            up
33        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.2 TiB      11 MiB      12 GiB    2.2 TiB  
  58.84    0.90      68            up
34        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB    8.7 MiB      11 GiB    2.3 TiB    
57.88    0.89      68            up
35        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.5 TiB      20 MiB      14 GiB    1.9 TiB  
  65.14    1.00      76            up
36        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB      18 MiB      12 GiB    2.3 TiB  
  57.91    0.89      68            up
37        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.4 TiB      18 MiB      13 GiB    2.0 TiB  
  63.33    0.97      74            up
38        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB      18 MiB      10 GiB    2.3 TiB  
  57.88    0.89      70            up
39        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      11 MiB      14 GiB    1.9 TiB  
  66.10    1.01      76            up
40        hdd    5.45798      1.00000    5.5 TiB    
3.3 TiB    3.3 TiB      26 MiB      11 GiB    2.1 TiB  
  60.61    0.93      71            up
41        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB      10 MiB      12 GiB    2.3 TiB  
  57.91    0.89      67            up
42        hdd    5.45798      1.00000    5.5 TiB    
3.4 TiB    3.4 TiB      18 MiB      13 GiB    2.1 TiB  
  62.42    0.96      73            up
43        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB      24 MiB      13 GiB    2.3 TiB  
  57.93    0.89      68            up
44        hdd    5.45798      1.00000    5.5 TiB    
3.3 TiB    3.3 TiB      37 MiB      10 GiB    2.2 TiB  
  60.57    0.93      71            up
45        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      21 MiB      13 GiB    1.9 TiB  
  66.04    1.01      77            up
46        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      19 MiB      13 GiB    1.9 TiB  
  66.07    1.01      77            up
47        hdd    5.45798      1.00000    5.5 TiB    
3.4 TiB    3.4 TiB      20 MiB      13 GiB    2.1 TiB  
  62.43    0.96      73            up
48        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.4 TiB      19 MiB      10 GiB    2.0 TiB  
  63.27    0.97      74            up
49        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.5 TiB      20 MiB      12 GiB    2.0 TiB  
  64.19    0.98      75            up
50        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.5 TiB      20 MiB      13 GiB    2.0 TiB  
  64.22    0.98      75            up
51        hdd    5.45798      1.00000    5.5 TiB    
3.2 TiB    3.1 TiB    9.1 MiB      12 GiB    2.3 TiB    
57.89    0.89      67            up
52        hdd    5.45798      1.00000    5.5 TiB    
3.0 TiB    3.0 TiB      19 MiB    8.6 GiB    2.5 TiB    
54.27    0.83      64            up
53        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      18 MiB      11 GiB    1.9 TiB  
  65.97    1.01      77            up
54        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      18 MiB      10 GiB    1.9 TiB  
  65.97    1.01      77            up
55        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.4 TiB      21 MiB    9.5 GiB    2.0 TiB    
63.28    0.97      74            up
56        hdd    5.45798      1.00000    5.5 TiB    
2.9 TiB    2.9 TiB    8.8 MiB    8.4 GiB    2.5 TiB    
53.39    0.82      62            up
57        hdd    5.45798      1.00000    5.5 TiB    
3.4 TiB    3.4 TiB      18 MiB      11 GiB    2.1 TiB  
  62.40    0.96      73            up
58        hdd    5.45798      1.00000    5.5 TiB    
3.1 TiB    3.1 TiB      18 MiB      12 GiB    2.3 TiB  
  56.98    0.87      67            up
59        hdd    5.45798      1.00000    5.5 TiB    
3.5 TiB    3.4 TiB      36 MiB      10 GiB    2.0 TiB  
  63.29    0.97      74            up
60        hdd    5.45798      1.00000    5.5 TiB    
3.3 TiB    3.3 TiB      20 MiB      10 GiB    2.2 TiB  
  60.60    0.93      71            up
61        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      10 MiB      10 GiB    1.9 TiB  
  65.97    1.01      76            up
62        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      19 MiB    9.4 GiB    1.9 TiB    
65.93    1.01      77            up
63        hdd    5.45798      1.00000    5.5 TiB    
3.0 TiB    3.0 TiB    9.0 MiB      12 GiB    2.4 TiB    
55.19    0.85      64            up
64        hdd    5.45798      1.00000    5.5 TiB    
3.4 TiB    3.4 TiB      10 MiB    9.3 GiB    2.1 TiB    
62.38    0.96      72            up
65        hdd    5.45798      1.00000    5.5 TiB    
3.3 TiB    3.2 TiB      14 MiB    8.7 GiB    2.2 TiB    
59.67    0.91      69            up
66        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB    9.2 GiB    1.9 TiB    
65.99    1.01      77            up
67        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.7 TiB      23 MiB    9.7 GiB    1.7 TiB    
68.70    1.05      80            up
68        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.7 TiB      19 MiB    9.5 GiB    1.7 TiB    
68.63    1.05      80            up
69        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB    9.1 GiB    1.9 TiB    
65.98    1.01      77            up
70        hdd    5.45798      1.00000    5.5 TiB    
3.8 TiB    3.8 TiB      18 MiB    9.5 GiB    1.6 TiB    
70.45    1.08      82            up
71        hdd    5.45798      1.00000    5.5 TiB    
4.2 TiB    4.2 TiB      11 MiB      11 GiB    1.2 TiB  
  77.68    1.19      89            up
72        hdd    5.45798      1.00000    5.5 TiB    
3.9 TiB    3.9 TiB    9.8 MiB    9.6 GiB    1.6 TiB    
71.37    1.09      82            up
73        hdd    5.45798      1.00000    5.5 TiB    
4.4 TiB    4.4 TiB      14 MiB      11 GiB    1.0 TiB  
  81.37    1.25      93            up
74        hdd    5.45798      1.00000    5.5 TiB    
3.8 TiB    3.7 TiB    9.2 MiB    9.3 GiB    1.7 TiB    
68.71    1.05      79            up
75        hdd    5.45798      1.00000    5.5 TiB    
3.7 TiB    3.7 TiB      20 MiB    9.1 GiB    1.7 TiB    
68.66    1.05      79            up
76        hdd    5.45798      1.00000    5.5 TiB    
4.3 TiB    4.3 TiB      10 MiB      11 GiB    1.1 TiB  
  79.55    1.22      91            up
77        hdd    5.45798      1.00000    5.5 TiB    
3.6 TiB    3.6 TiB      20 MiB    8.9 GiB    1.9 TiB    
65.99    1.01      77            up
                                        
      TOTAL    426 TiB    278 TiB    277 TiB    1.3 
GiB    898 GiB    148 TiB    65.30                
                      
MIN/MAX VAR: 0.82/1.25    STDDEV: 5.45

ceph osd tree
ID      CLASS    WEIGHT          TYPE NAME        
        STATUS    REWEIGHT    PRI-AFF
 -1                  425.72394    root default      
                                        
              
 -5                    65.49600            host 
uk-s01                                    
                  
 12        hdd        5.45798                
    osd.12              up      1.00000    1.00000
 13        hdd        5.45798                
    osd.13              up      1.00000    1.00000
 14        hdd        5.45798                
    osd.14              up      1.00000    1.00000
 15        hdd        5.45798                
    osd.15              up      1.00000    1.00000
 16        hdd        5.45798                
    osd.16              up      1.00000    1.00000
 17        hdd        5.45798                
    osd.17              up      1.00000    1.00000
 18        hdd        5.45798                
    osd.18              up      1.00000    1.00000
 19        hdd        5.45798                
    osd.19              up      1.00000    1.00000
 20        hdd        5.45798                
    osd.20              up      1.00000    1.00000
 21        hdd        5.45798                
    osd.21              up      1.00000    1.00000
 22        hdd        5.45798                
    osd.22              up      1.00000    1.00000
 23        hdd        5.45798                
    osd.23              up      1.00000    1.00000
 -3                    65.49600            host 
uk-s02                                    
                  
    0        hdd        5.45798              
      osd.0                up      1.00000    
1.00000
    1        hdd        5.45798              
      osd.1                up      1.00000    
1.00000
    2        hdd        5.45798              
      osd.2                up      1.00000    
1.00000
    3        hdd        5.45798              
      osd.3                up      1.00000    
1.00000
    4        hdd        5.45798              
      osd.4                up      1.00000    
1.00000
    5        hdd        5.45798              
      osd.5                up      1.00000    
1.00000
    6        hdd        5.45798              
      osd.6                up      1.00000    
1.00000
    7        hdd        5.45798              
      osd.7                up      1.00000    
1.00000
    8        hdd        5.45798              
      osd.8                up      1.00000    
1.00000
    9        hdd        5.45798              
      osd.9                up      1.00000    
1.00000
 10        hdd        5.45798                
    osd.10              up      1.00000    1.00000
 11        hdd        5.45798                
    osd.11              up      1.00000    1.00000
 -7                    76.41200            host 
uk-s03                                    
                  
 24        hdd        5.45798                
    osd.24              up      1.00000    1.00000
 25        hdd        5.45798                
    osd.25              up      1.00000    1.00000
 26        hdd        5.45798                
    osd.26              up      1.00000    1.00000
 27        hdd        5.45798                
    osd.27              up      1.00000    1.00000
 28        hdd        5.45798                
    osd.28              up      1.00000    1.00000
 29        hdd        5.45798                
    osd.29              up      1.00000    1.00000
 30        hdd        5.45798                
    osd.30              up      1.00000    1.00000
 31        hdd        5.45798                
    osd.31              up      1.00000    1.00000
 32        hdd        5.45798                
    osd.32              up      1.00000    1.00000
 33        hdd        5.45798                
    osd.33              up      1.00000    1.00000
 34        hdd        5.45798                
    osd.34              up      1.00000    1.00000
 35        hdd        5.45798                
    osd.35              up      1.00000    1.00000
 36        hdd        5.45798                
    osd.36              up      1.00000    1.00000
 37        hdd        5.45798                
    osd.37              up      1.00000    1.00000
 -9                    76.41200            host 
uk-s04                                    
                  
 38        hdd        5.45798                
    osd.38              up      1.00000    1.00000
 39        hdd        5.45798                
    osd.39              up      1.00000    1.00000
 40        hdd        5.45798                
    osd.40              up      1.00000    1.00000
 41        hdd        5.45798                
    osd.41              up      1.00000    1.00000
 42        hdd        5.45798                
    osd.42              up      1.00000    1.00000
 43        hdd        5.45798                
    osd.43              up      1.00000    1.00000
 44        hdd        5.45798                
    osd.44              up      1.00000    1.00000
 45        hdd        5.45798                
    osd.45              up      1.00000    1.00000
 46        hdd        5.45798                
    osd.46              up      1.00000    1.00000
 47        hdd        5.45798                
    osd.47              up      1.00000    1.00000
 48        hdd        5.45798                
    osd.48              up      1.00000    1.00000
 49        hdd        5.45798                
    osd.49              up      1.00000    1.00000
 50        hdd        5.45798                
    osd.50              up      1.00000    1.00000
 51        hdd        5.45798                
    osd.51              up      1.00000    1.00000
-11                    76.41200            host 
uk-s05                                    
                  
 52        hdd        5.45798                
    osd.52              up      1.00000    1.00000
 53        hdd        5.45798                
    osd.53              up      1.00000    1.00000
 54        hdd        5.45798                
    osd.54              up      1.00000    1.00000
 55        hdd        5.45798                
    osd.55              up      1.00000    1.00000
 56        hdd        5.45798                
    osd.56              up      1.00000    1.00000
 57        hdd        5.45798                
    osd.57              up      1.00000    1.00000
 58        hdd        5.45798                
    osd.58              up      1.00000    1.00000
 59        hdd        5.45798                
    osd.59              up      1.00000    1.00000
 60        hdd        5.45798                
    osd.60              up      1.00000    1.00000
 61        hdd        5.45798                
    osd.61              up      1.00000    1.00000
 62        hdd        5.45798                
    osd.62              up      1.00000    1.00000
 63        hdd        5.45798                
    osd.63              up      1.00000    1.00000
 64        hdd        5.45798                
    osd.64              up      1.00000    1.00000
 65        hdd        5.45798                
    osd.65              up      1.00000    1.00000
-13                    65.49591            host 
uk-s06                                    
                  
 66        hdd        5.45798                
    osd.66              up      1.00000    1.00000
 67        hdd        5.45798                
    osd.67              up      1.00000    1.00000
 68        hdd        5.45798                
    osd.68              up      1.00000    1.00000
 69        hdd        5.45798                
    osd.69              up      1.00000    1.00000
 70        hdd        5.45798                
    osd.70              up      1.00000    1.00000
 71        hdd        5.45798                
    osd.71              up      1.00000    1.00000
 72        hdd        5.45798                
    osd.72              up      1.00000    1.00000
 73        hdd        5.45798                
    osd.73              up      1.00000    1.00000
 74        hdd        5.45798                
    osd.74              up      1.00000    1.00000
 75        hdd        5.45798                
    osd.75              up      1.00000    1.00000
 76        hdd        5.45798                
    osd.76              up      1.00000    1.00000
 77        hdd        5.45798                
    osd.77              up      1.00000    1.00000


rule ecdata {
        id 1
        type erasure
        step set_chooseleaf_tries 5
        step set_choose_tries 100
        step take default
        step choose indep 6 type host
        step choose indep 2 type osd
        step emit
}

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to