Hello,
It looks like we had 2 osd's fail at some point earlier today, here is
the current status of the cluster:
root@rbd1:~# ceph -s
cluster 504b5794-34bd-44e7-a8c3-0494cf800c23
health HEALTH_WARN
2 pgs backfill
5 pgs backfill_toofull
69 pgs backfilling
74 pgs degraded
1 pgs down
1 pgs peering
74 pgs stuck degraded
1 pgs stuck inactive
75 pgs stuck unclean
74 pgs stuck undersized
74 pgs undersized
recovery 1903019/105270534 objects degraded (1.808%)
recovery 1120305/105270534 objects misplaced (1.064%)
crush map has legacy tunables
monmap e1: 3 mons at
{hqceph1=10.35.1.201:6789/0,hqceph2=10.35.1.203:6789/0,hqceph3=10.35.1.205:6789/0}
election epoch 282, quorum 0,1,2 hqceph1,hqceph2,hqceph3
osdmap e25019: 108 osds: 105 up, 105 in; 74 remapped pgs
pgmap v30721368: 3976 pgs, 17 pools, 144 TB data, 51401 kobjects
285 TB used, 97367 GB / 380 TB avail
1903019/105270534 objects degraded (1.808%)
1120305/105270534 objects misplaced (1.064%)
3893 active+clean
69 active+undersized+degraded+remapped+backfilling
6 active+clean+scrubbing
3 active+undersized+degraded+remapped+backfill_toofull
2 active+clean+scrubbing+deep
2
active+undersized+degraded+remapped+wait_backfill+backfill_toofull
1 down+peering
recovery io 248 MB/s, 84 objects/s
We had been running for a while with 107 osd's (not 108), it looks like
osd's 64 and 76 are both now down and out at this point.
I have looked though the ceph logs for each osd and did not see anything
obvious, the raid controller also does not show the disk offline.
I am wondering if I should try to restart the two osd's that are showing
as down...or should I wait until the current recovery is complete?
The pool has a replica level of '2'...and with 2 failed disks I want to
do whatever I can to make sure there is not an issue with missing objects.
Thanks in advance,
Shain
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
