Luminous 12.1.0(RC) I replaced two OSD drives(old ones were still good, just too small), using:
ceph osd out osd.12 ceph osd crush remove osd.12 ceph auth del osd.12 systemctl stop ceph-osd@osd.12 ceph osd rm osd.12 I later found that I also should have unmounted it from /var/lib/ceph/osd-12 (remove old disk, insert new disk) I added the new disk/osd with ceph-deploy osd prepare stor-vm3:sdg --bluestore This automatically activated the osd (not sure why, I thought it needed a ceph-deploy osd activate as well) Then, working on an unrelated issue, I upgraded one (out of 4 total) nodes to 12.1.1 using apt and rebooted. The mon daemon would not form a quorum with the others on 12.1.0, so, instead of troubleshooting that, I just went ahead and upgraded the other 3 nodes and rebooted. Lots of recovery IO went on afterwards, but now things have stopped at: pools: 10 pools, 6804 pgs objects: 1784k objects, 7132 GB usage: 11915 GB used, 19754 GB / 31669 GB avail pgs: 0.353% pgs not active 70894/2988573 objects degraded (2.372%) 422090/2988573 objects misplaced (14.123%) 6626 active+clean 129 active+remapped+backfill_wait 23 incomplete 14 active+undersized+degraded+remapped+backfill_wait 4 active+undersized+degraded+remapped+backfilling 4 active+remapped+backfilling 2 active+clean+scrubbing+deep 1 peering 1 active+recovery_wait+degraded+remapped when I run ceph pg query on the incompletes, they all list at least one of the two removed OSDs(12,17) in "down_osds_we_would_probe" most pools are size:2 min_size 1(trusting bluestore to tell me which one is valid). One pool is size:1 min size:1 and I'm okay with losing it, except I had it mounted in a directory on cephfs, I rm'd the directory but I can't delete the pool because it's "in use by CephFS" I still have the old drives, can I stick them into another host and re-add them somehow? This data isn't super important, but I'd like to learn a bit on how to recover when bad things happen as we are planning a production deployment in a couple of weeks.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com