Re: [ceph-users] rebalance near full osd
Hello, On Tue, 12 Apr 2016 09:46:55 +0100 (BST) Andrei Mikhailovsky wrote: > I've done the ceph osd reweight-by-utilization and it seems to have > solved the issue. However, not sure if this will be the long term > solution. > No. As I said in my reply, use "crush reweight" to permanently adjust weights. reweight-by-utilization is a band-aid and not permanent, see: http://cephnotes.ksperis.com/blog/2014/12/23/difference-between-ceph-osd-reweight-and-ceph-osd-crush-reweight More OSDs may or may not result in a better uniformity, the 30% difference you're seeing is definitely at the far end of what one would expect with Ceph. Christian > Thanks for your help > > Andrei > > - Original Message - > > From: "Shinobu Kinjo" > > To: "Andrei Mikhailovsky" > > Cc: "Christian Balzer" , "ceph-users" > > Sent: Friday, 8 April, 2016 01:35:18 > > Subject: Re: [ceph-users] rebalance near full osd > > > There was a discussion before regarding to the situation where you are > > facing now. [1] > > Would you have a look, if it's helpful or not for you. > > > > [1] > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007622.html > > > > Cheers, > > Shinobu > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebalance near full osd
I've done the ceph osd reweight-by-utilization and it seems to have solved the issue. However, not sure if this will be the long term solution. Thanks for your help Andrei - Original Message - > From: "Shinobu Kinjo" > To: "Andrei Mikhailovsky" > Cc: "Christian Balzer" , "ceph-users" > > Sent: Friday, 8 April, 2016 01:35:18 > Subject: Re: [ceph-users] rebalance near full osd > There was a discussion before regarding to the situation where you are > facing now. [1] > Would you have a look, if it's helpful or not for you. > > [1] > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007622.html > > Cheers, > Shinobu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebalance near full osd
There was a discussion before regarding to the situation where you are facing now. [1] Would you have a look, if it's helpful or not for you. [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007622.html Cheers, Shinobu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebalance near full osd
Hi Christian, thanks for your input. I don't think the PG count is my issue. I've actually have too many PGs for the size of my cluster consisting of just 18 osds spread amongst 2 osd servers and 3 mons: root@arh-ibstorage1-ib:~# ceph -s health HEALTH_WARN 1 near full osd(s) too many PGs per OSD (604 > max 300) osdmap e79493: 18 osds: 18 up, 18 in pgmap v79839575: 5436 pgs, 18 pools, 15509 GB data, 6019 kobjects 5436 active+clean I will take a look at the "dealing with the full osd / help reweight" thread, thanks for pointing it out. Cheers Andrei - Original Message - > From: "Christian Balzer" > To: "ceph-users" > Cc: "Andrei Mikhailovsky" > Sent: Wednesday, 6 April, 2016 04:36:30 > Subject: Re: [ceph-users] rebalance near full osd > Hello, > > On Wed, 6 Apr 2016 04:18:40 +0100 (BST) Andrei Mikhailovsky wrote: > >> Hi >> >> I've just had a warning ( from ceph -s) that one of the osds is near >> full. Having investigated the warning, i've located that osd.6 is 86% >> full. The data distribution is nowhere near to being equal on my osds as >> you can see from the df command output below: >> > Firstly, read the very recent thread: > "dealing with the full osd / help reweight" > from this ML. > > You really want monitoring SW to keep track of disk utilization if you're > not doing it manually. > >> /dev/sdj1 2.8T 2.4T 413G 86% /var/lib/ceph/osd/ceph-6 >> /dev/sdb1 2.8T 2.1T 625G 78% /var/lib/ceph/osd/ceph-0 >> /dev/sdc1 2.8T 2.0T 824G 71% /var/lib/ceph/osd/ceph-1 >> /dev/sdd1 2.8T 1.5T 1.3T 55% /var/lib/ceph/osd/ceph-2 >> /dev/sde1 2.8T 1.7T 1.1T 63% /var/lib/ceph/osd/ceph-3 >> /dev/sdh1 2.8T 1.7T 1.1T 62% /var/lib/ceph/osd/ceph-4 >> /dev/sdf1 2.8T 1.9T 932G 67% /var/lib/ceph/osd/ceph-8 >> /dev/sdi1 2.8T 1.9T 880G 69% /var/lib/ceph/osd/ceph-5 >> /dev/sdg1 2.8T 2.0T 798G 72% /var/lib/ceph/osd/ceph-7 >> >> I seem to have the spread of over 30% disk utilisation between the osds, >> despite all my osds having the identical weight (ceph osd tree output): >> >> >> -2 24.56999 host arh-ibstorage1-ib >> 1 2.73000 osd.1 up 1.0 1.0 >> 3 2.73000 osd.3 up 1.0 1.0 >> 5 2.73000 osd.5 up 1.0 1.0 >> 6 2.73000 osd.6 up 1.0 1.0 >> 7 2.73000 osd.7 up 1.0 1.0 >> 8 2.73000 osd.8 up 1.0 1.0 >> 4 2.73000 osd.4 up 1.0 1.0 >> 0 2.73000 osd.0 up 1.0 1.0 >> 2 2.73000 osd.2 up 1.0 1.0 >> > This is just one host, are they all like that? > Full osd tree and even more importantly a "ceph -s" output. > Ceph isn't particular good at creating an even distribution, but if you > have too little PGs it gets worse, which would be my first suspicion here. > >> >> What would be the best way to correct the issue without having >> significant impact on the cluster IO? >> > Again, read the thread above. > Increasing the PG count (if that is part of your problem) will have massive > impact, but it needs to be done at some point. > Re-weighting (CRUSH, permanently) OSDs in small increments/decrements (you > want to keep the host weight more or less the same) of course also causes > data movement, but done right (see the thread) the impact can be minimized. > > Christian >> Many thanks >> >> Andrei > > > -- > Christian BalzerNetwork/Systems Engineer > ch...@gol.com Global OnLine Japan/Rakuten Communications > http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebalance near full osd
Hello, On Wed, 6 Apr 2016 04:18:40 +0100 (BST) Andrei Mikhailovsky wrote: > Hi > > I've just had a warning ( from ceph -s) that one of the osds is near > full. Having investigated the warning, i've located that osd.6 is 86% > full. The data distribution is nowhere near to being equal on my osds as > you can see from the df command output below: > Firstly, read the very recent thread: "dealing with the full osd / help reweight" from this ML. You really want monitoring SW to keep track of disk utilization if you're not doing it manually. > /dev/sdj1 2.8T 2.4T 413G 86% /var/lib/ceph/osd/ceph-6 > /dev/sdb1 2.8T 2.1T 625G 78% /var/lib/ceph/osd/ceph-0 > /dev/sdc1 2.8T 2.0T 824G 71% /var/lib/ceph/osd/ceph-1 > /dev/sdd1 2.8T 1.5T 1.3T 55% /var/lib/ceph/osd/ceph-2 > /dev/sde1 2.8T 1.7T 1.1T 63% /var/lib/ceph/osd/ceph-3 > /dev/sdh1 2.8T 1.7T 1.1T 62% /var/lib/ceph/osd/ceph-4 > /dev/sdf1 2.8T 1.9T 932G 67% /var/lib/ceph/osd/ceph-8 > /dev/sdi1 2.8T 1.9T 880G 69% /var/lib/ceph/osd/ceph-5 > /dev/sdg1 2.8T 2.0T 798G 72% /var/lib/ceph/osd/ceph-7 > > I seem to have the spread of over 30% disk utilisation between the osds, > despite all my osds having the identical weight (ceph osd tree output): > > > -2 24.56999 host arh-ibstorage1-ib > 1 2.73000 osd.1 up 1.0 1.0 > 3 2.73000 osd.3 up 1.0 1.0 > 5 2.73000 osd.5 up 1.0 1.0 > 6 2.73000 osd.6 up 1.0 1.0 > 7 2.73000 osd.7 up 1.0 1.0 > 8 2.73000 osd.8 up 1.0 1.0 > 4 2.73000 osd.4 up 1.0 1.0 > 0 2.73000 osd.0 up 1.0 1.0 > 2 2.73000 osd.2 up 1.0 1.0 > This is just one host, are they all like that? Full osd tree and even more importantly a "ceph -s" output. Ceph isn't particular good at creating an even distribution, but if you have too little PGs it gets worse, which would be my first suspicion here. > > What would be the best way to correct the issue without having > significant impact on the cluster IO? > Again, read the thread above. Increasing the PG count (if that is part of your problem) will have massive impact, but it needs to be done at some point. Re-weighting (CRUSH, permanently) OSDs in small increments/decrements (you want to keep the host weight more or less the same) of course also causes data movement, but done right (see the thread) the impact can be minimized. Christian > Many thanks > > Andrei -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rebalance near full osd
Hi I've just had a warning ( from ceph -s) that one of the osds is near full. Having investigated the warning, i've located that osd.6 is 86% full. The data distribution is nowhere near to being equal on my osds as you can see from the df command output below: /dev/sdj1 2.8T 2.4T 413G 86% /var/lib/ceph/osd/ceph-6 /dev/sdb1 2.8T 2.1T 625G 78% /var/lib/ceph/osd/ceph-0 /dev/sdc1 2.8T 2.0T 824G 71% /var/lib/ceph/osd/ceph-1 /dev/sdd1 2.8T 1.5T 1.3T 55% /var/lib/ceph/osd/ceph-2 /dev/sde1 2.8T 1.7T 1.1T 63% /var/lib/ceph/osd/ceph-3 /dev/sdh1 2.8T 1.7T 1.1T 62% /var/lib/ceph/osd/ceph-4 /dev/sdf1 2.8T 1.9T 932G 67% /var/lib/ceph/osd/ceph-8 /dev/sdi1 2.8T 1.9T 880G 69% /var/lib/ceph/osd/ceph-5 /dev/sdg1 2.8T 2.0T 798G 72% /var/lib/ceph/osd/ceph-7 I seem to have the spread of over 30% disk utilisation between the osds, despite all my osds having the identical weight (ceph osd tree output): -2 24.56999 host arh-ibstorage1-ib 1 2.73000 osd.1 up 1.0 1.0 3 2.73000 osd.3 up 1.0 1.0 5 2.73000 osd.5 up 1.0 1.0 6 2.73000 osd.6 up 1.0 1.0 7 2.73000 osd.7 up 1.0 1.0 8 2.73000 osd.8 up 1.0 1.0 4 2.73000 osd.4 up 1.0 1.0 0 2.73000 osd.0 up 1.0 1.0 2 2.73000 osd.2 up 1.0 1.0 What would be the best way to correct the issue without having significant impact on the cluster IO? Many thanks Andrei ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
