Replication does not occur until the OSD is “out.” This creates a new mapping in the cluster of where the PGs should be and thus data begins to move and/or create sufficient copies. This scheme lets you control how and when you want the replication to occur. If you have plenty of space and you aren’t going to replace the drive immediately, just mark the OSD “down" AND “out.". If you are going to replace the drive immediately, set the “noout” flag. Take the OSD “down” and replace drive. Assuming it is mounted in the same place as the bad drive, bring the OSD back up. This will replicate exactly the same PGs the bad drive held back to the replacement drive. As was stated before don’t forget to “ceph osd unset noout"
Keep in mind that in the case of a machine that has a hardware failure and takes OSD(s) down there is an automatic timeout which will mark them “out" for unattended operation. Unless you are monitoring the cluster 24/7 you should have enough disk space available to handle failures. Related info in: http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/ David Zafman Senior Developer http://www.inktank.com On Nov 15, 2013, at 1:58 AM, Mihály Árva-Tóth <mihaly.arva-t...@virtual-call-center.eu> wrote: > Hello, > > I think you misunderstood me. I known how can I replace bad HDD, thanks. My > problem is the following: > > Object replica number is 3. Objects that in 11.15d PG which store data on > osd.0 bad sectors place inter alia. Ceph should know objects are in 11.15d on > osd.0 is bad (because deep-scrub and repair are both failed), so there is no > 3 clean replica number only two. I think Ceph should replicate its to new > osd. Or not? > > Thank you, > Mihaly > > 2013/11/13 David Zafman <david.zaf...@inktank.com> > > Since the disk is failing and you have 2 other copies I would take osd.0 > down. This means that ceph will not attempt to read the bad disk either for > clients or to make another copy of the data: > > ***** Not sure about the syntax of this for the version of ceph you are > running > ceph osd down 0 > > Mark it “out” which will immediately trigger recovery to create more copies > of the data with the remaining OSDs. > ceph osd out 0 > > You can now finish the process of removing the osd by looking at these > instructions: > > http://ceph.com/docs/master/rados/operations/add-or-rm-osds/#removing-osds-manual > > David Zafman > Senior Developer > http://www.inktank.com > > On Nov 12, 2013, at 3:16 AM, Mihály Árva-Tóth > <mihaly.arva-t...@virtual-call-center.eu> wrote: > > > Hello, > > > > I have 3 node, with 3 OSD in each node. I'm using .rgw.buckets pool with 3 > > replica. One of my HDD (osd.0) has just bad sectors, when I try to read an > > object from OSD direct, I get Input/output errror. dmesg: > > > > [1214525.670065] mpt2sas0: log_info(0x31080000): originator(PL), > > code(0x08), sub_code(0x0000) > > [1214525.670072] mpt2sas0: log_info(0x31080000): originator(PL), > > code(0x08), sub_code(0x0000) > > [1214525.670100] sd 0:0:2:0: [sdc] Unhandled sense code > > [1214525.670104] sd 0:0:2:0: [sdc] > > [1214525.670107] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > > [1214525.670110] sd 0:0:2:0: [sdc] > > [1214525.670112] Sense Key : Medium Error [current] > > [1214525.670117] Info fld=0x60c8f21 > > [1214525.670120] sd 0:0:2:0: [sdc] > > [1214525.670123] Add. Sense: Unrecovered read error > > [1214525.670126] sd 0:0:2:0: [sdc] CDB: > > [1214525.670128] Read(16): 88 00 00 00 00 00 06 0c 8f 20 00 00 00 08 00 00 > > > > Okay I known need to replace HDD. > > > > Fragment of ceph -s output: > > pgmap v922039: 856 pgs: 855 active+clean, 1 active+clean+inconsistent; > > > > ceph pg dump | grep inconsistent > > > > 11.15d 25443 0 0 0 6185091790 3001 3001 > > active+clean+inconsistent 2013-11-06 02:30:45.23416..... > > > > ceph pg map 11.15d > > > > osdmap e1600 pg 11.15d (11.15d) -> up [0,8,3] acting [0,8,3] > > > > pg repair or deep-scrub can not fix this issue. But if I understand > > correctly, osd has to known it can not retrieve object from osd.0 and need > > to be replicate an another osd because there is no 3 working replicas now. > > > > Thank you, > > Mihaly > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
