Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-09 Thread Paul Evans
Congrats Chris and nice "save" on that RBD! -- Paul > On Apr 9, 2015, at 11:11 AM, Chris Kitzmiller > wrote: > > Success! Hopefully my notes from the process will help: > > In the event of multiple disk failures the cluster could lose PGs. Should > this occur it is best to attempt to restar

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-09 Thread Chris Kitzmiller
Success! Hopefully my notes from the process will help: In the event of multiple disk failures the cluster could lose PGs. Should this occur it is best to attempt to restart the OSD process and have the drive marked as up+out. Marking the drive as out will cause data to flow off the drive to el

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-07 Thread Chris Kitzmiller
I'm not having much luck here. Is there a possibility that the imported PGs aren't being picked up because the MONs think that they're older than the empty PGs I find on the up OSDs? I feel that I'm so close to *not* losing my RBD volume because I only have two bad PGs and I've successfully exp

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-06 Thread Chris Kitzmiller
On Apr 6, 2015, at 1:49 PM, Craig Lewis wrote: > In that case, I'd set the crush weight to the disk's size in TiB, and mark > the osd out: > ceph osd crush reweight osd. > ceph osd out > > Then your tree should look like: > -9 2.72 host ithome > 30 2.72

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-06 Thread Craig Lewis
In that case, I'd set the crush weight to the disk's size in TiB, and mark the osd out: ceph osd crush reweight osd. ceph osd out Then your tree should look like: -9 *2.72* host ithome 30 *2.72* osd.30 up *0* An OSD can be UP and OUT, which ca

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-06 Thread Chris Kitzmiller
On Apr 3, 2015, at 12:37 AM, LOPEZ Jean-Charles wrote: > > according to your ceph osd tree capture, although the OSD reweight is set to > 1, the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD > a CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight >

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-04 Thread Chris Kitzmiller
On Apr 3, 2015, at 12:37 AM, LOPEZ Jean-Charles wrote: > according to your ceph osd tree capture, although the OSD reweight is set to > 1, the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD > a CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight > os

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-02 Thread LOPEZ Jean-Charles
Hi Chris, according to your ceph osd tree capture, although the OSD reweight is set to 1, the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD a CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight osd.30 x.y (where 1.0=1TB) Only when this is done will

[ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

2015-04-02 Thread Chris Kitzmiller
I have a cluster running 0.80.9 on Ubuntu 14.04. A couple nights ago I lost two disks from a pool with size=2. :( I replaced the two failed OSDs and I now have two PGs which are marked as incomplete in an otherwise healthy cluster. Following this page ( https://ceph.com/community/incomplete-pgs