Re: [ceph-users] How to properly deal with NEAR FULL OSD
It would be helpful to see your crush map (there are some tunables that help with this issue as well available if you're not running ancient versions). However, distribution uniformity isn't that great really. It helps to increase the number of PGs, but beware that there's no turning back. Other than that, play with reweights (and possibly crush weights) regularly - that's what we do... Jan > On 18 Feb 2016, at 01:11, Vlad Blando wrote: > > Hi This been bugging me for some time now, the distribution of data on the > OSD is not balanced so some OSD are near full, i did ceph osd > reweight-by-utilization but it not helping much. > > > [root@controller-node ~]# ceph osd tree > # idweight type name up/down reweight > -1 98.28 root default > -2 32.76 host ceph-node-1 > 0 3.64osd.0 up 1 > 1 3.64osd.1 up 1 > 2 3.64osd.2 up 1 > 3 3.64osd.3 up 1 > 4 3.64osd.4 up 1 > 5 3.64osd.5 up 1 > 6 3.64osd.6 up 1 > 7 3.64osd.7 up 1 > 8 3.64osd.8 up 1 > -3 32.76 host ceph-node-2 > 9 3.64osd.9 up 1 > 10 3.64osd.10 up 1 > 11 3.64osd.11 up 1 > 12 3.64osd.12 up 1 > 13 3.64osd.13 up 1 > 14 3.64osd.14 up 1 > 15 3.64osd.15 up 1 > 16 3.64osd.16 up 1 > 17 3.64osd.17 up 1 > -4 32.76 host ceph-node-3 > 18 3.64osd.18 up 1 > 19 3.64osd.19 up 1 > 20 3.64osd.20 up 1 > 21 3.64osd.21 up 1 > 22 3.64osd.22 up 1 > 23 3.64osd.23 up 1 > 24 3.64osd.24 up 1 > 25 3.64osd.25 up 1 > 26 3.64osd.26 up 1 > [root@controller-node ~]# > > > [root@controller-node ~]# /opt/df-osd.sh > ceph-node-1 > === > /dev/sdb1 3.7T 2.0T 1.7T 54% /var/lib/ceph/osd/ceph-0 > /dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-1 > /dev/sdd1 3.7T 3.3T 431G 89% /var/lib/ceph/osd/ceph-2 > /dev/sde1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-3 > /dev/sdf1 3.7T 3.3T 379G 90% /var/lib/ceph/osd/ceph-4 > /dev/sdg1 3.7T 2.9T 762G 80% /var/lib/ceph/osd/ceph-5 > /dev/sdh1 3.7T 3.0T 733G 81% /var/lib/ceph/osd/ceph-6 > /dev/sdi1 3.7T 3.4T 284G 93% /var/lib/ceph/osd/ceph-7 > /dev/sdj1 3.7T 3.4T 342G 91% /var/lib/ceph/osd/ceph-8 > === > ceph-node-2 > === > /dev/sdb1 3.7T 3.1T 622G 84% /var/lib/ceph/osd/ceph-9 > /dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-10 > /dev/sdd1 3.7T 3.1T 557G 86% /var/lib/ceph/osd/ceph-11 > /dev/sde1 3.7T 3.3T 392G 90% /var/lib/ceph/osd/ceph-12 > /dev/sdf1 3.7T 2.6T 1.1T 72% /var/lib/ceph/osd/ceph-13 > /dev/sdg1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-14 > /dev/sdh1 3.7T 2.7T 984G 74% /var/lib/ceph/osd/ceph-15 > /dev/sdi1 3.7T 3.2T 463G 88% /var/lib/ceph/osd/ceph-16 > /dev/sdj1 3.7T 3.1T 594G 85% /var/lib/ceph/osd/ceph-17 > === > ceph-node-3 > === > /dev/sdb1 3.7T 2.8T 910G 76% /var/lib/ceph/osd/ceph-18 > /dev/sdc1 3.7T 2.7T 1012G 73% /var/lib/ceph/osd/ceph-19 > /dev/sdd1 3.7T 3.2T 537G 86% /var/lib/ceph/osd/ceph-20 > /dev/sde1 3.7T 3.2T 465G 88% /var/lib/ceph/osd/ceph-21 > /dev/sdf1 3.7T 3.0T 663G 83% /var/lib/ceph/osd/ceph-22 > /dev/sdg1 3.7T 3.4T 248G 94% /var/lib/ceph/osd/ceph-23 > /dev/sdh1 3.7T 2.8T 928G 76% /var/lib/ceph/osd/ceph-24 > /dev/sdi1 3.7T 2.9T 802G 79% /var/lib/ceph/osd/ceph-25 > /dev/sdj1 3.7T 2.7T 1.1T 73% /var/lib/ceph/osd/ceph-26 > === > [root@controller-node ~]# > > > [root@controller-node ~]# ceph health detail > HEALTH_ERR 2 pgs inconsistent; 10 near full osd(s); 2 scrub errors > pg 5.7f is acti
Re: [ceph-users] How to properly deal with NEAR FULL OSD
Vlad, First off your cluster is rather full (80.31%). Hopefully you have hardware ordered for an expansion in the near future. Based on your 'ceph osd tree' output, it doesn't look like the reweight-by-utilization did anything for you. That last number for each OSD is set to 1, which means it didn't reweight any of the OSDs. This is a different weight than the CRUSH weight, and something you can manually modify as well. For example you could manually tweak the weights of the fullest OSDs with: ceph osd reweight osd.23 0.95 ceph osd reweight osd.7 0.95 ceph osd reweight osd.8 0.95 Then just keep tweaking those numbers until the cluster gets an even distribution of PGs across the OSDs. The reweight-by-utilization option can help make this quicker. Your volumes pool also doesn't have a power of two for pg_num, so your PGs will have uneven sizes. Since you can't go back down to 256 PGs, you should look at gradually increasing them up to 512 PGs. There are also inconsistent PGs that you should look at repairing. It won't help you with the data distribution, but it's good cluster maintenance. Bryan From: ceph-users on behalf of Vlad Blando Date: Wednesday, February 17, 2016 at 5:11 PM To: ceph-users Subject: [ceph-users] How to properly deal with NEAR FULL OSD >Hi This been bugging me for some time now, the distribution of data on >the OSD is not balanced so some OSD are near full, i did ceph > osd reweight-by-utilization but it not helping much. > > >[root@controller-node ~]# ceph osd tree ># idweight type name up/down reweight >-1 98.28 root default >-2 32.76 host ceph-node-1 >0 3.64osd.0 up 1 >1 3.64osd.1 up 1 >2 3.64osd.2 up 1 >3 3.64osd.3 up 1 >4 3.64osd.4 up 1 >5 3.64osd.5 up 1 >6 3.64osd.6 up 1 >7 3.64osd.7 up 1 >8 3.64osd.8 up 1 >-3 32.76 host ceph-node-2 >9 3.64osd.9 up 1 >10 3.64osd.10 up 1 >11 3.64osd.11 up 1 >12 3.64osd.12 up 1 >13 3.64osd.13 up 1 >14 3.64osd.14 up 1 >15 3.64osd.15 up 1 >16 3.64osd.16 up 1 >17 3.64osd.17 up 1 >-4 32.76 host ceph-node-3 >18 3.64osd.18 up 1 >19 3.64osd.19 up 1 >20 3.64osd.20 up 1 >21 3.64osd.21 up 1 >22 3.64osd.22 up 1 >23 3.64osd.23 up 1 >24 3.64osd.24 up 1 >25 3.64osd.25 up 1 >26 3.64osd.26 up 1 >[root@controller-node ~]# > > >[root@controller-node ~]# /opt/df-osd.sh >ceph-node-1 >=== >/dev/sdb1 3.7T 2.0T 1.7T 54% /var/lib/ceph/osd/ceph-0 >/dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-1 >/dev/sdd1 3.7T 3.3T 431G 89% /var/lib/ceph/osd/ceph-2 >/dev/sde1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-3 >/dev/sdf1 3.7T 3.3T 379G 90% /var/lib/ceph/osd/ceph-4 >/dev/sdg1 3.7T 2.9T 762G 80% /var/lib/ceph/osd/ceph-5 >/dev/sdh1 3.7T 3.0T 733G 81% /var/lib/ceph/osd/ceph-6 >/dev/sdi1 3.7T 3.4T 284G 93% /var/lib/ceph/osd/ceph-7 >/dev/sdj1 3.7T 3.4T 342G 91% /var/lib/ceph/osd/ceph-8 >=== >ceph-node-2 >=== >/dev/sdb1 3.7T 3.1T 622G 84% /var/lib/ceph/osd/ceph-9 >/dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-10 >/dev/sdd1 3.7T 3.1T 557G 86% /var/lib/ceph/osd/ceph-11 >/dev/sde1 3.7T 3.3T 392G 90% /var/lib/ceph/osd/ceph-12 >/dev/sdf1 3.7T 2.6T 1.1T 72% /var/lib/ceph/osd/ceph-13 >/dev/sdg1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-14 >/dev/sdh1 3.7T 2.7T 984G 74% /var/lib/ceph/osd/ceph-15 >/dev/sdi1 3.7T 3.2T 463G 88% /var/lib/ceph/osd/ceph-16 >/dev/sdj1 3.7T 3.1T 594G 85% /var/lib/ceph/osd/ceph-17 >=== >ceph-node-3 >=== >/dev/sdb1 3.7T 2.8T 910G 76% /var/lib/ceph/osd/ceph-18 >/dev/sdc1 3.7T 2.7T 1012G 73% /
Re: [ceph-users] How to properly deal with NEAR FULL OSD
What are your outputs of ceph df ceph osd df Regards, Don > On Feb 17, 2016, at 5:31 PM, Stillwell, Bryan > wrote: > > Vlad, > > First off your cluster is rather full (80.31%). Hopefully you have > hardware ordered for an expansion in the near future. > > Based on your 'ceph osd tree' output, it doesn't look like the > reweight-by-utilization did anything for you. That last number for each > OSD is set to 1, which means it didn't reweight any of the OSDs. This is > a different weight than the CRUSH weight, and something you can manually > modify as well. > > For example you could manually tweak the weights of the fullest OSDs with: > > ceph osd reweight osd.23 0.95 > ceph osd reweight osd.7 0.95 > ceph osd reweight osd.8 0.95 > > Then just keep tweaking those numbers until the cluster gets an even > distribution of PGs across the OSDs. The reweight-by-utilization option > can help make this quicker. > > Your volumes pool also doesn't have a power of two for pg_num, so your PGs > will have uneven sizes. Since you can't go back down to 256 PGs, you > should look at gradually increasing them up to 512 PGs. > > There are also inconsistent PGs that you should look at repairing. It > won't help you with the data distribution, but it's good cluster > maintenance. > > Bryan > > From: ceph-users on behalf of Vlad > Blando > Date: Wednesday, February 17, 2016 at 5:11 PM > To: ceph-users > Subject: [ceph-users] How to properly deal with NEAR FULL OSD > > >> Hi This been bugging me for some time now, the distribution of data on >> the OSD is not balanced so some OSD are near full, i did ceph >> osd reweight-by-utilization but it not helping much. >> >> >> [root@controller-node ~]# ceph osd tree >> # idweight type name up/down reweight >> -1 98.28 root default >> -2 32.76 host ceph-node-1 >> 0 3.64osd.0 up 1 >> 1 3.64osd.1 up 1 >> 2 3.64osd.2 up 1 >> 3 3.64osd.3 up 1 >> 4 3.64osd.4 up 1 >> 5 3.64osd.5 up 1 >> 6 3.64osd.6 up 1 >> 7 3.64osd.7 up 1 >> 8 3.64osd.8 up 1 >> -3 32.76 host ceph-node-2 >> 9 3.64osd.9 up 1 >> 10 3.64osd.10 up 1 >> 11 3.64osd.11 up 1 >> 12 3.64osd.12 up 1 >> 13 3.64osd.13 up 1 >> 14 3.64osd.14 up 1 >> 15 3.64osd.15 up 1 >> 16 3.64osd.16 up 1 >> 17 3.64osd.17 up 1 >> -4 32.76 host ceph-node-3 >> 18 3.64osd.18 up 1 >> 19 3.64osd.19 up 1 >> 20 3.64osd.20 up 1 >> 21 3.64osd.21 up 1 >> 22 3.64osd.22 up 1 >> 23 3.64osd.23 up 1 >> 24 3.64osd.24 up 1 >> 25 3.64osd.25 up 1 >> 26 3.64osd.26 up 1 >> [root@controller-node ~]# >> >> >> [root@controller-node ~]# /opt/df-osd.sh >> ceph-node-1 >> === >> /dev/sdb1 3.7T 2.0T 1.7T 54% /var/lib/ceph/osd/ceph-0 >> /dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-1 >> /dev/sdd1 3.7T 3.3T 431G 89% /var/lib/ceph/osd/ceph-2 >> /dev/sde1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-3 >> /dev/sdf1 3.7T 3.3T 379G 90% /var/lib/ceph/osd/ceph-4 >> /dev/sdg1 3.7T 2.9T 762G 80% /var/lib/ceph/osd/ceph-5 >> /dev/sdh1 3.7T 3.0T 733G 81% /var/lib/ceph/osd/ceph-6 >> /dev/sdi1 3.7T 3.4T 284G 93% /var/lib/ceph/osd/ceph-7 >> /dev/sdj1 3.7T 3.4T 342G 91% /var/lib/ceph/osd/ceph-8 >> === >> ceph-node-2 >> === >> /dev/sdb1 3.7T 3.1T 622G 84% /var/lib/ceph/osd/ceph-9 >> /dev/sdc1 3.7T 2.7T 1.1T 72% /var/lib/ceph/osd/ceph-10 >> /dev/sdd1 3.7T 3.1T 557G 86% /var/lib/ceph/osd/ceph-11 >> /dev/sde1 3.7T 3.3T 392G 90% /var/lib/ceph/osd/ceph-12 >> /dev/sdf1 3.7T 2.6T 1.1T 72% /var/lib/ceph/osd/ceph-13 >> /dev/sdg1 3.7T 2.8T 879G 77% /var/lib/ceph/osd/ceph-14 >> /dev/sdh1 3.7T 2.7T 984G 74% /var/lib/ceph/osd/ceph-15 >> /dev/sdi1 3.7T 3.2T 463G 88% /var/lib/ceph/osd/ceph-16 >> /dev/sdj1
Re: [ceph-users] How to properly deal with NEAR FULL OSD
I changed my volume PGs from 300 to 512 to even out the distribution, right now it is backfilling and remapping and I noticed that it's working. --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 88% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 86% osd.16 is near full at 86% osd.17 is near full at 85% osd.20 is near full at 85% osd.23 is near full at 86% --- We will be adding a new node to the cluster after this. Another question, I'de like to adjust the near full OSD warning from 85% to 90% temporarily. I cant remember the command. @don ceph df --- [root@controller-node ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 18391G 82161G 81.71 POOLS: NAMEID USED %USED OBJECTS images 4 8927G 8.88 1143014 volumes 5 18374G 18.27 4721934 [root@controller-node ~]# --- ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
I tried setting this ceph tell mon.* injectargs "--mon_osd_nearfull_ratio .92" but it seems not working or the mon is busy and the command is on queue? --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 87% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 85% osd.16 is near full at 85% osd.17 is near full at 85% osd.20 is near full at 85% osd.22 is near full at 85% --- On Fri, Feb 19, 2016 at 9:30 AM, Vlad Blando wrote: > I changed my volume PGs from 300 to 512 to even out the distribution, > right now it is backfilling and remapping and I noticed that it's working. > > --- > osd.2 is near full at 85% > osd.4 is near full at 85% > osd.5 is near full at 85% > osd.6 is near full at 85% > osd.7 is near full at 86% > osd.8 is near full at 88% > osd.9 is near full at 85% > osd.11 is near full at 85% > osd.12 is near full at 86% > osd.16 is near full at 86% > osd.17 is near full at 85% > osd.20 is near full at 85% > osd.23 is near full at 86% > --- > > We will be adding a new node to the cluster after this. > > Another question, I'de like to adjust the near full OSD warning from 85% > to 90% temporarily. I cant remember the command. > > > @don > ceph df > --- > [root@controller-node ~]# ceph df > GLOBAL: > SIZEAVAIL RAW USED %RAW USED > 100553G 18391G 82161G 81.71 > POOLS: > NAMEID USED %USED OBJECTS > images 4 8927G 8.88 1143014 > volumes 5 18374G 18.27 4721934 > [root@controller-node ~]# > --- > > > ᐧ > ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
ceph osd reweight must be between 0 and 1. 0.95 is a nice starting point, then go down with small steps. Regards. Dimitar Boichev SysAdmin Team Lead AXSMarine Sofia Phone: +359 889 22 55 42 Skype: dimitar.boichev.axsmarine E-mail: dimitar.boic...@axsmarine.com<mailto:dimitar.boic...@axsmarine.com> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vlad Blando Sent: Friday, February 19, 2016 3:31 AM To: Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I changed my volume PGs from 300 to 512 to even out the distribution, right now it is backfilling and remapping and I noticed that it's working. --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 88% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 86% osd.16 is near full at 86% osd.17 is near full at 85% osd.20 is near full at 85% osd.23 is near full at 86% --- We will be adding a new node to the cluster after this. Another question, I'de like to adjust the near full OSD warning from 85% to 90% temporarily. I cant remember the command. @don ceph df --- [root@controller-node ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 18391G 82161G 81.71 POOLS: NAMEID USED %USED OBJECTS images 4 8927G 8.88 1143014 volumes 5 18374G 18.27 4721934 [root@controller-node ~]# --- [https://mailfoogae.appspot.com/t?sender=admJsYW5kb0Btb3JwaGxhYnMuY29t&type=zerocontent&guid=9c934871-a414-4d80-858c-6b83435d112d]ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
I have seen this when there was a recovery going on some PGs and we were deleting big amounts of data. They disappeared when the recovery process finished. This was on Firefly 0.80.7 Regards. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vlad Blando Sent: Friday, February 19, 2016 3:31 AM To: Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I changed my volume PGs from 300 to 512 to even out the distribution, right now it is backfilling and remapping and I noticed that it's working. --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 88% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 86% osd.16 is near full at 86% osd.17 is near full at 85% osd.20 is near full at 85% osd.23 is near full at 86% --- We will be adding a new node to the cluster after this. Another question, I'de like to adjust the near full OSD warning from 85% to 90% temporarily. I cant remember the command. @don ceph df --- [root@controller-node ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 18391G 82161G 81.71 POOLS: NAMEID USED %USED OBJECTS images 4 8927G 8.88 1143014 volumes 5 18374G 18.27 4721934 [root@controller-node ~]# --- [https://mailfoogae.appspot.com/t?sender=admJsYW5kb0Btb3JwaGxhYnMuY29t&type=zerocontent&guid=9c934871-a414-4d80-858c-6b83435d112d]ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
Sorry, reply to a wrong message. Regards. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dimitar Boichev Sent: Friday, February 19, 2016 10:19 AM To: Vlad Blando; Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I have seen this when there was a recovery going on some PGs and we were deleting big amounts of data. They disappeared when the recovery process finished. This was on Firefly 0.80.7 Regards. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vlad Blando Sent: Friday, February 19, 2016 3:31 AM To: Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I changed my volume PGs from 300 to 512 to even out the distribution, right now it is backfilling and remapping and I noticed that it's working. --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 88% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 86% osd.16 is near full at 86% osd.17 is near full at 85% osd.20 is near full at 85% osd.23 is near full at 86% --- We will be adding a new node to the cluster after this. Another question, I'de like to adjust the near full OSD warning from 85% to 90% temporarily. I cant remember the command. @don ceph df --- [root@controller-node ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 18391G 82161G 81.71 POOLS: NAMEID USED %USED OBJECTS images 4 8927G 8.88 1143014 volumes 5 18374G 18.27 4721934 [root@controller-node ~]# --- [https://mailfoogae.appspot.com/t?sender=admJsYW5kb0Btb3JwaGxhYnMuY29t&type=zerocontent&guid=9c934871-a414-4d80-858c-6b83435d112d]ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
Thanks. To summarize Your data, images+volumes = 27.15% space used Raw used = 81.71% used This is a big difference that I can’t account for? Can anyone? So is your cluster actually full? I had the same problem with my small cluster. Raw used was about 85% and actual data, with replication, was about 30%. My OSDs were also BRTFS. BRTFS was causing its own problems. I fixed my problem by removing each OSD one at a time and re-adding as the default XFS filesystem. Doing so brought the percentages used to be about the same and it’s good now. My observation is that ceph wasn’t reclaiming space used. My version was Hammer /don From: Dimitar Boichev [mailto:dimitar.boic...@axsmarine.com] Sent: Friday, February 19, 2016 1:19 AM To: Dimitar Boichev ; Vlad Blando ; Don Laursen Cc: ceph-users Subject: RE: [ceph-users] How to properly deal with NEAR FULL OSD Sorry, reply to a wrong message. Regards. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dimitar Boichev Sent: Friday, February 19, 2016 10:19 AM To: Vlad Blando; Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I have seen this when there was a recovery going on some PGs and we were deleting big amounts of data. They disappeared when the recovery process finished. This was on Firefly 0.80.7 Regards. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vlad Blando Sent: Friday, February 19, 2016 3:31 AM To: Don Laursen Cc: ceph-users Subject: Re: [ceph-users] How to properly deal with NEAR FULL OSD I changed my volume PGs from 300 to 512 to even out the distribution, right now it is backfilling and remapping and I noticed that it's working. --- osd.2 is near full at 85% osd.4 is near full at 85% osd.5 is near full at 85% osd.6 is near full at 85% osd.7 is near full at 86% osd.8 is near full at 88% osd.9 is near full at 85% osd.11 is near full at 85% osd.12 is near full at 86% osd.16 is near full at 86% osd.17 is near full at 85% osd.20 is near full at 85% osd.23 is near full at 86% --- We will be adding a new node to the cluster after this. Another question, I'de like to adjust the near full OSD warning from 85% to 90% temporarily. I cant remember the command. @don ceph df --- [root@controller-node ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 18391G 82161G 81.71 POOLS: NAMEID USED %USED OBJECTS images 4 8927G 8.88 1143014 volumes 5 18374G 18.27 4721934 [root@controller-node ~]# --- [https://mailfoogae.appspot.com/t?sender=admJsYW5kb0Btb3JwaGxhYnMuY29t&type=zerocontent&guid=9c934871-a414-4d80-858c-6b83435d112d]ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
Le 19/02/2016 17:17, Don Laursen a écrit : > > Thanks. To summarize > > Your data, images+volumes = 27.15% space used > > Raw used = 81.71% used > > > > This is a big difference that I can’t account for? Can anyone? So is > your cluster actually full? > I believe this is the pool size being accounted for and it is harmless: 3 x 27.15 = 81.45 which is awfully close to 81.71. We have the same behavior on our Ceph cluster. > > > I had the same problem with my small cluster. Raw used was about 85% > and actual data, with replication, was about 30%. My OSDs were also > BRTFS. BRTFS was causing its own problems. I fixed my problem by > removing each OSD one at a time and re-adding as the default XFS > filesystem. Doing so brought the percentages used to be about the same > and it’s good now. > That's odd : AFAIK we had the same behaviour with XFS before migrating to BTRFS. Best regards, Lionel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to properly deal with NEAR FULL OSD
Problem is now solved, the cluster is now backfilling/recovering normally, no more NEAR FULL OSD. It turns out that I have RBD objects that should have been deleted long time ago but it's still there. Openstack Glance did not removed it, I think it's an issue with snapshots, an RBD file can't be deleted unless it's snapshots are purged. So I compared all my glance images to the RBD counterpart and identified which are not there and deleted them. So from 81% utilization I am down to 61%. --- [root@controller-node opt]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100553G 39118G 61435G 61.10 POOLS: NAMEID USED %USED OBJECTS images 4 1764G 1.76 225978 volumes 5 18533G 18.43 4762609 [root@controller-node opt]# --- On Sat, Feb 20, 2016 at 5:38 AM, Lionel Bouton < lionel-subscript...@bouton.name> wrote: > Le 19/02/2016 17:1 > > 7, Don Laursen a écrit : > > Thanks. To summarize > > Your data, images+volumes = 27.15% space used > > Raw used = 81.71% used > > > > This is a big difference that I can’t account for? Can anyone? So is your > cluster actually full? > > > I believe this is the pool size being accounted for and it is harmless: 3 > x 27.15 = 81.45 which is awfully close to 81.71. > We have the same behavior on our Ceph cluster. > > > > I had the same problem with my small cluster. Raw used was about 85% and > actual data, with replication, was about 30%. My OSDs were also BRTFS. > BRTFS was causing its own problems. I fixed my problem by removing each OSD > one at a time and re-adding as the default XFS filesystem. Doing so brought > the percentages used to be about the same and it’s good now. > > > That's odd : AFAIK we had the same behaviour with XFS before migrating to > BTRFS. > > Best regards, > > Lionel > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ᐧ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
