I added more OSDs few days ago to reduce usage under 70% (nearfull and full ratio is higher than this value) and it still stuck at backfill_toofull while rebalance data.
I tried to change backfill full ratio and it show error (unchangeable) as below: [root@storcp ~]# ceph tell osd.\* injectargs '--osd_backfill_full_ratio 0.92' osd.0: osd_backfill_full_ratio = '0.92' (unchangeable) osd.1: osd_backfill_full_ratio = '0.92' (unchangeable) osd.2: osd_backfill_full_ratio = '0.92' (unchangeable) osd.3: osd_backfill_full_ratio = '0.92' (unchangeable) osd.4: osd_backfill_full_ratio = '0.92' (unchangeable) osd.5: osd_backfill_full_ratio = '0.92' (unchangeable) osd.6: osd_backfill_full_ratio = '0.92' (unchangeable) osd.7: osd_backfill_full_ratio = '0.92' (unchangeable) osd.8: osd_backfill_full_ratio = '0.92' (unchangeable) osd.9: osd_backfill_full_ratio = '0.92' (unchangeable) osd.10: osd_backfill_full_ratio = '0.92' (unchangeable) osd.11: osd_backfill_full_ratio = '0.92' (unchangeable) osd.12: osd_backfill_full_ratio = '0.92' (unchangeable) osd.13: osd_backfill_full_ratio = '0.92' (unchangeable) osd.14: osd_backfill_full_ratio = '0.92' (unchangeable) osd.15: osd_backfill_full_ratio = '0.92' (unchangeable) osd.16: osd_backfill_full_ratio = '0.92' (unchangeable) osd.17: osd_backfill_full_ratio = '0.92' (unchangeable) osd.18: osd_backfill_full_ratio = '0.92' (unchangeable) osd.19: osd_backfill_full_ratio = '0.92' (unchangeable) osd.20: osd_backfill_full_ratio = '0.92' (unchangeable) osd.21: osd_backfill_full_ratio = '0.92' (unchangeable) osd.22: osd_backfill_full_ratio = '0.92' (unchangeable) osd.23: osd_backfill_full_ratio = '0.92' (unchangeable) osd.24: osd_backfill_full_ratio = '0.92' (unchangeable) osd.25: osd_backfill_full_ratio = '0.92' (unchangeable) osd.26: osd_backfill_full_ratio = '0.92' (unchangeable) osd.27: osd_backfill_full_ratio = '0.92' (unchangeable) osd.28: osd_backfill_full_ratio = '0.92' (unchangeable) [root@storcp ~]# On Wed, Dec 20, 2017 at 1:57 AM, David C <dcsysengin...@gmail.com> wrote: > What's your backfill full ratio? You may be able to get healthy by > increasing your backfill full ratio (in small increments). But your next > immediate task should be to add more OSDs or remove data. > > > On 19 Dec 2017 4:26 p.m., "Nghia Than" <cont...@trungnghia.info> wrote: > > Hi, > > My CEPH is stuck at this for few days, we added new OSD and nothing > changed: > > - *17 pgs backfill_toofull* > - *17 pgs stuck unclean* > - *recovery 21/5156264 objects degraded (0.000%)* > - *recovery 52908/5156264 objects misplaced (1.026%)* > - *8 near full osd(s)* > > And here is my ceph health detail: > > HEALTH_WARN 17 pgs backfill_toofull; 17 pgs stuck unclean; recovery > 21/5156264 objects degraded (0.000%); recovery 52908/5156264 objects > misplaced (1.026%); 8 near full osd(s) > > pg 1.231 is stuck unclean for 4367.088889, current state > active+remapped+backfill_toofull, last acting [24,9] > > pg 1.1e8 is stuck unclean for 7316.364770, current state > active+remapped+backfill_toofull, last acting [16,3] > > pg 1.188 is stuck unclean for 7315.400227, current state > active+remapped+backfill_toofull, last acting [11,7] > > pg 1.158 is stuck unclean for 7321.511627, current state > active+remapped+backfill_toofull, last acting [11,17] > > pg 1.81 is stuck unclean for 4366.683703, current state > active+remapped+backfill_toofull, last acting [10,24] > > pg 1.332 is stuck unclean for 7315.248115, current state > active+remapped+backfill_toofull, last acting [23,1] > > pg 1.2c2 is stuck unclean for 4365.635413, current state > active+remapped+backfill_toofull, last acting [24,13] > > pg 1.3c6 is stuck unclean for 7320.816089, current state > active+remapped+backfill_toofull, last acting [11,20] > > pg 1.26f is stuck unclean for 7315.882215, current state > active+remapped+backfill_toofull, last acting [28,8] > > pg 1.236 is stuck unclean for 7322.152706, current state > active+remapped+backfill_toofull, last acting [8,26] > > pg 1.249 is stuck unclean for 4366.885751, current state > active+remapped+backfill_toofull, last acting [9,24] > > pg 1.7b is stuck unclean for 7315.353072, current state > active+remapped+backfill_toofull, last acting [28,3] > > pg 1.1ec is stuck unclean for 7315.981062, current state > active+remapped+backfill_toofull, last acting [16,0] > > pg 1.248 is stuck unclean for 7324.062482, current state > active+remapped+backfill_toofull, last acting [16,3] > > pg 1.e4 is stuck unclean for 4370.009328, current state > active+remapped+backfill_toofull, last acting [21,24] > > pg 1.144 is stuck unclean for 7317.998393, current state > active+remapped+backfill_toofull, last acting [26,3] > > pg 0.5f is stuck unclean for 5877.987814, current state > active+remapped+backfill_toofull, last acting [24,5] > > pg 1.3c6 is active+remapped+backfill_toofull, acting [11,20] > > pg 1.332 is active+remapped+backfill_toofull, acting [23,1] > > pg 1.2c2 is active+remapped+backfill_toofull, acting [24,13] > > pg 1.26f is active+remapped+backfill_toofull, acting [28,8] > > pg 1.249 is active+remapped+backfill_toofull, acting [9,24] > > pg 1.248 is active+remapped+backfill_toofull, acting [16,3] > > pg 1.236 is active+remapped+backfill_toofull, acting [8,26] > > pg 1.e4 is active+remapped+backfill_toofull, acting [21,24] > > pg 0.5f is active+remapped+backfill_toofull, acting [24,5] > > pg 1.7b is active+remapped+backfill_toofull, acting [28,3] > > pg 1.81 is active+remapped+backfill_toofull, acting [10,24] > > pg 1.144 is active+remapped+backfill_toofull, acting [26,3] > > pg 1.158 is active+remapped+backfill_toofull, acting [11,17] > > pg 1.188 is active+remapped+backfill_toofull, acting [11,7] > > pg 1.1e8 is active+remapped+backfill_toofull, acting [16,3] > > pg 1.1ec is active+remapped+backfill_toofull, acting [16,0] > > pg 1.231 is active+remapped+backfill_toofull, acting [24,9] > > recovery 21/5156264 objects degraded (0.000%) > > recovery 52908/5156264 objects misplaced (1.026%) > > osd.3 is near full at 92% > > osd.4 is near full at 91% > > osd.12 is near full at 92% > > osd.17 is near full at 86% > > osd.18 is near full at 87% > > osd.23 is near full at 90% > > osd.27 is near full at 85% > osd.28 is near full at 85% > > I tried reweight OSD to smaller weight but nothing changed. This is my > dump full_ratio: > > [root@storcp ~]# ceph pg dump |grep full_ratio > > dumped all in format plain > > full_ratio 0.95 > > nearfull_ratio 0.85 > [root@storcp ~]# > > And ceph osd df: > > [root@storcp ~]# ceph osd df > > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > > 0 0.86800 1.00000 888G 754G 134G 84.91 1.09 102 > > 1 0.86800 1.00000 888G 734G 154G 82.63 1.06 90 > > 2 0.86800 1.00000 888G 548G 339G 61.77 0.79 75 > > 9 0.86800 1.00000 888G 658G 230G 74.09 0.95 81 > > 10 0.86800 1.00000 888G 659G 229G 74.17 0.95 79 > > 11 0.86800 1.00000 888G 706G 182G 79.49 1.02 91 > > 18 0.86800 1.00000 888G 774G 114G 87.14 1.12 94 > > 3 0.86800 1.00000 888G 823G 67037M 92.63 1.19 99 > > 4 0.86800 1.00000 888G 816G 73780M 91.89 1.18 102 > > 5 0.86800 1.00000 888G 608G 279G 68.51 0.88 76 > > 12 0.86800 1.00000 888G 818G 72144M 92.07 1.18 111 > > 13 0.86800 1.00000 888G 657G 231G 73.94 0.95 84 > > 14 0.86800 1.00000 888G 668G 220G 75.16 0.96 84 > > 19 0.86800 1.00000 888G 547G 341G 61.59 0.79 75 > > 6 0.86800 1.00000 888G 651G 237G 73.31 0.94 90 > > 7 0.86800 1.00000 888G 542G 346G 61.01 0.78 68 > > 8 0.86800 1.00000 888G 727G 160G 81.90 1.05 90 > > 15 0.86800 1.00000 888G 627G 260G 70.64 0.91 91 > > 16 0.86800 1.00000 888G 668G 220G 75.19 0.96 81 > > 17 0.86800 1.00000 888G 764G 124G 86.04 1.10 92 > > 20 0.86800 1.00000 888G 598G 289G 67.37 0.86 78 > > 21 0.86800 1.00000 888G 726G 162G 81.73 1.05 87 > > 22 0.86800 1.00000 888G 707G 181G 79.60 1.02 92 > > 23 0.86800 1.00000 888G 804G 85861M 90.57 1.16 104 > > 24 0.86800 0.79999 888G 726G 162G 81.73 1.05 90 > > 25 0.86800 1.00000 888G 579G 308G 65.24 0.84 80 > > 26 0.86800 1.00000 888G 696G 192G 78.36 1.00 95 > > 27 0.86800 1.00000 888G 757G 131G 85.20 1.09 98 > > 28 0.86800 1.00000 888G 758G 130G 85.29 1.09 104 > > TOTAL 25775G 20115G 5660G 78.04 > > MIN/MAX VAR: 0.78/1.19 STDDEV: 9.24 > [root@storcp ~]# > > May i know how to get over this? > > -- > ============== > Nghia Than > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- ============== Nghia Than
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
