Hi Irek, yes, stoping OSD (or seting it to OUT) resulted in only 3% of data degraded and moved/recovered. When I after that removed it from Crush map "ceph osd crush rm id", that's when the stuff with 37% happened.
And thanks Irek for help - could you kindly just let me know of the prefered steps when removing whole node? Do you mean I first stop all OSDs again, or just remove each OSD from crush map, or perhaps, just decompile cursh map, delete the node completely, compile back in, and let it heal/recover ? Do you think this would result in less data missplaces and moved arround ? Sorry for bugging you, I really appreaciate your help. Thanks On 3 March 2015 at 12:58, Irek Fasikhov <malm...@gmail.com> wrote: > A large percentage of the rebuild of the cluster map (But low percentage > degradation). If you had not made "ceph osd crush rm id", the percentage > would be low. > In your case, the correct option is to remove the entire node, rather than > each disk individually > > 2015-03-03 14:27 GMT+03:00 Andrija Panic <andrija.pa...@gmail.com>: > >> Another question - I mentioned here 37% of objects being moved arround - >> this is MISPLACED object (degraded objects were 0.001%, after I removed 1 >> OSD from cursh map (out of 44 OSD or so). >> >> Can anybody confirm this is normal behaviour - and are there any >> workarrounds ? >> >> I understand this is because of the object placement algorithm of CEPH, >> but still 37% of object missplaces just by removing 1 OSD from crush maps >> out of 44 make me wonder why this large percentage ? >> >> Seems not good to me, and I have to remove another 7 OSDs (we are >> demoting some old hardware nodes). This means I can potentialy go with 7 x >> the same number of missplaced objects...? >> >> Any thoughts ? >> >> Thanks >> >> On 3 March 2015 at 12:14, Andrija Panic <andrija.pa...@gmail.com> wrote: >> >>> Thanks Irek. >>> >>> Does this mean, that after peering for each PG, there will be delay of >>> 10sec, meaning that every once in a while, I will have 10sec od the cluster >>> NOT being stressed/overloaded, and then the recovery takes place for that >>> PG, and then another 10sec cluster is fine, and then stressed again ? >>> >>> I'm trying to understand process before actually doing stuff (config >>> reference is there on ceph.com but I don't fully understand the process) >>> >>> Thanks, >>> Andrija >>> >>> On 3 March 2015 at 11:32, Irek Fasikhov <malm...@gmail.com> wrote: >>> >>>> Hi. >>>> >>>> Use value "osd_recovery_delay_start" >>>> example: >>>> [root@ceph08 ceph]# ceph --admin-daemon /var/run/ceph/ceph-osd.94.asok >>>> config show | grep osd_recovery_delay_start >>>> "osd_recovery_delay_start": "10" >>>> >>>> 2015-03-03 13:13 GMT+03:00 Andrija Panic <andrija.pa...@gmail.com>: >>>> >>>>> HI Guys, >>>>> >>>>> I yesterday removed 1 OSD from cluster (out of 42 OSDs), and it caused >>>>> over 37% od the data to rebalance - let's say this is fine (this is when I >>>>> removed it frm Crush Map). >>>>> >>>>> I'm wondering - I have previously set some throtling mechanism, but >>>>> during first 1h of rebalancing, my rate of recovery was going up to 1500 >>>>> MB/s - and VMs were unusable completely, and then last 4h of the duration >>>>> of recover this recovery rate went down to, say, 100-200 MB.s and during >>>>> this VM performance was still pretty impacted, but at least I could work >>>>> more or a less >>>>> >>>>> So my question, is this behaviour expected, is throtling here working >>>>> as expected, since first 1h was almoust no throtling applied if I check >>>>> the >>>>> recovery rate 1500MB/s and the impact on Vms. >>>>> And last 4h seemed pretty fine (although still lot of impact in >>>>> general) >>>>> >>>>> I changed these throtling on the fly with: >>>>> >>>>> ceph tell osd.* injectargs '--osd_recovery_max_active 1' >>>>> ceph tell osd.* injectargs '--osd_recovery_op_priority 1' >>>>> ceph tell osd.* injectargs '--osd_max_backfills 1' >>>>> >>>>> My Jorunals are on SSDs (12 OSD per server, of which 6 journals on one >>>>> SSD, 6 journals on another SSD) - I have 3 of these hosts. >>>>> >>>>> Any thought are welcome. >>>>> -- >>>>> >>>>> Andrija Panić >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>> >>>> >>>> -- >>>> С уважением, Фасихов Ирек Нургаязович >>>> Моб.: +79229045757 >>>> >>> >>> >>> >>> -- >>> >>> Andrija Panić >>> >> >> >> >> -- >> >> Andrija Panić >> > > > > -- > С уважением, Фасихов Ирек Нургаязович > Моб.: +79229045757 > -- Andrija Panić
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
