Hi Harry,
I was able to replicate this.
What does appear to work (for me) is to do an osd scrub followed by a pg
repair. I've tried this 2x now and in each case the deleted file gets
copied over to the OSD from where it was removed. However, I've tried a
few pg scrub / pg repairs after manually deleting a file and have yet to
see the file get copied back to the OSD on which it was deleted. Like you
said, the pg repair sets the health of the PG back to active+clean, but
then re-running the pg scrub detects the file as missing again and sets it
back to active+clean+inconsistent.
Regards,
Matt
On Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington <git-ha...@live.co.uk>wrote:
> Hi,
>
> I've been taking a look at the repair functionality in ceph. As I
> understand it the osds should try to copy an object from another member of
> the pg if it is missing. I have been attempting to test this by manually
> removing a file from one of the osds however each time the repair
> completes the the file has not been restored. If I run another scrub on the
> pg it gets flagged as inconsistent. See below for the output from my
> testing. I assume I'm missing something obvious, any insight into this
> process would be greatly appreciated.
>
> Thanks,
> Harry
>
> # ceph --version
> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
> # ceph status
> cluster a4e417fe-0386-46a5-4475-ca7e10294273
> health HEALTH_OK
> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
> 0 ceph1
> osdmap e13: 3 osds: 3 up, 3 in
> pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
> 164 GB / 179 GB avail
> mdsmap e1: 0/0/1 up
>
> file removed from osd.2
>
> # ceph pg scrub 0.b
> instructing pg 0.b on osd.1 to scrub
>
> # ceph status
> cluster a4e417fe-0386-46a5-4475-ca7e10294273
> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
> 0 ceph1
> osdmap e13: 3 osds: 3 up, 3 in
> pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
> bytes data, 15465 MB used, 164 GB / 179 GB avail
> mdsmap e1: 0/0/1 up
>
> # ceph pg repair 0.b
> instructing pg 0.b on osd.1 to repair
>
> # ceph status
> cluster a4e417fe-0386-46a5-4475-ca7e10294273
> health HEALTH_OK
> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
> 0 ceph1
> osdmap e13: 3 osds: 3 up, 3 in
> pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
> 164 GB / 179 GB avail
> mdsmap e1: 0/0/1 up
>
> # ceph pg scrub 0.b
> instructing pg 0.b on osd.1 to scrub
>
> # ceph status
> cluster a4e417fe-0386-46a5-4475-ca7e10294273
> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
> 0 ceph1
> osdmap e13: 3 osds: 3 up, 3 in
> pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
> bytes data, 15465 MB used, 164 GB / 179 GB avail
> mdsmap e1: 0/0/1 up
>
>
>
> The logs from osd.1:
> 2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
> 3a643fcb/testfile1/head//0
> 2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1
> missing, 0 inconsistent objects
> 2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
> 2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
> 3a643fcb/testfile1/head//0
> 2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1
> missing, 0 inconsistent objects
> 2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1
> errors, 1 fixed
> 2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
> 3a643fcb/testfile1/head//0
> 2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1
> missing, 0 inconsistent objects
> 2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
