This message seems to be very concerning:
> mds0: Metadata damage detected
but for the rest, the cluster seems still to be recovering. you could try
to seep thing up with ceph tell, like:
ceph tell osd.* injectargs --osd_max_backfills=10
ceph tell osd.* injectargs --osd_recovery_sleep=0.0
ceph tell osd.* injectargs --osd_recovery_threads=2
Regards,
Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*
On Fri, May 11, 2018 at 3:06 PM Daniel Davidson <dani...@igb.illinois.edu>
wrote:
> Below id the information you were asking for. I think they are size=2,
> min size=1.
>
> Dan
>
> # ceph status
> cluster
> 7bffce86-9d7b-4bdf-a9c9-67670e68ca77
>
> health
> HEALTH_ERR
>
> 140 pgs are stuck inactive for more than 300 seconds
> 64 pgs backfill_wait
> 76 pgs backfilling
> 140 pgs degraded
> 140 pgs stuck degraded
> 140 pgs stuck inactive
> 140 pgs stuck unclean
> 140 pgs stuck undersized
> 140 pgs undersized
> 210 requests are blocked > 32 sec
> recovery 38725029/695508092 objects degraded (5.568%)
> recovery 10844554/695508092 objects misplaced (1.559%)
> mds0: Metadata damage detected
> mds0: Behind on trimming (71/30)
> noscrub,nodeep-scrub flag(s) set
> monmap e3: 4 mons at {ceph-0=
> 172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0
> }
> election epoch 824, quorum 0,1,2,3 ceph-0,ceph-1,ceph-2,ceph-3
> fsmap e144928: 1/1/1 up {0=ceph-0=up:active}, 1 up:standby
> osdmap e35814: 32 osds: 30 up, 30 in; 140 remapped pgs
> flags noscrub,nodeep-scrub,sortbitwise,require_jewel_osds
> pgmap v43142427: 1536 pgs, 2 pools, 762 TB data, 331 Mobjects
> 1444 TB used, 1011 TB / 2455 TB avail
> 38725029/695508092 objects degraded (5.568%)
> 10844554/695508092 objects misplaced (1.559%)
> 1396 active+clean
> 76 undersized+degraded+remapped+backfilling+peered
> 64 undersized+degraded+remapped+wait_backfill+peered
> recovery io 1244 MB/s, 1612 keys/s, 705 objects/s
>
> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 2619.54541 root default
> -2 163.72159 host ceph-0
> 0 81.86079 osd.0 up 1.00000 1.00000
> 1 81.86079 osd.1 up 1.00000 1.00000
> -3 163.72159 host ceph-1
> 2 81.86079 osd.2 up 1.00000 1.00000
> 3 81.86079 osd.3 up 1.00000 1.00000
> -4 163.72159 host ceph-2
> 8 81.86079 osd.8 up 1.00000 1.00000
> 9 81.86079 osd.9 up 1.00000 1.00000
> -5 163.72159 host ceph-3
> 10 81.86079 osd.10 up 1.00000 1.00000
> 11 81.86079 osd.11 up 1.00000 1.00000
> -6 163.72159 host ceph-4
> 4 81.86079 osd.4 up 1.00000 1.00000
> 5 81.86079 osd.5 up 1.00000 1.00000
> -7 163.72159 host ceph-5
> 6 81.86079 osd.6 up 1.00000 1.00000
> 7 81.86079 osd.7 up 1.00000 1.00000
> -8 163.72159 host ceph-6
> 12 81.86079 osd.12 up 0.79999 1.00000
> 13 81.86079 osd.13 up 1.00000 1.00000
> -9 163.72159 host ceph-7
> 14 81.86079 osd.14 up 1.00000 1.00000
> 15 81.86079 osd.15 up 1.00000 1.00000
> -10 163.72159 host ceph-8
> 16 81.86079 osd.16 up 1.00000 1.00000
> 17 81.86079 osd.17 up 1.00000 1.00000
> -11 163.72159 host ceph-9
> 18 81.86079 osd.18 up 1.00000 1.00000
> 19 81.86079 osd.19 up 1.00000 1.00000
> -12 163.72159 host ceph-10
> 20 81.86079 osd.20 up 1.00000 1.00000
> 21 81.86079 osd.21 up 1.00000 1.00000
> -13 163.72159 host ceph-11
> 22 81.86079 osd.22 up 1.00000 1.00000
> 23 81.86079 osd.23 up 1.00000 1.00000
> -14 163.72159 host ceph-12
> 24 81.86079 osd.24 up 1.00000 1.00000
> 25 81.86079 osd.25 up 1.00000 1.00000
> -15 163.72159 host ceph-13
> 26 81.86079 osd.26 down 0 1.00000
> 27 81.86079 osd.27 down 0 1.00000
> -16 163.72159 host ceph-14
> 28 81.86079 osd.28 up 1.00000 1.00000
> 29 81.86079 osd.29 up 1.00000 1.00000
> -17 163.72159 host ceph-15
> 30 81.86079 osd.30 up 1.00000 1.00000
> 31 81.86079 osd.31 up 1.00000 1.00000
>
>
>
> On 05/11/2018 11:56 AM, David Turner wrote:
>
> What are some outputs of commands to show us the state of your cluster.
> Most notable is `ceph status` but `ceph osd tree` would be helpful. What
> are the size of the pools in your cluster? Are they all size=3 min_size=2?
>
> On Fri, May 11, 2018 at 12:05 PM Daniel Davidson <dani...@igb.illinois.edu>
> wrote:
>
>> Hello,
>>
>> Today we had a node crash, and looking at it, it seems there is a
>> problem with the RAID controller, so it is not coming back up, maybe
>> ever. It corrupted the local filesytem for the ceph storage there.
>>
>> The remainder of our storage (10.2.10) cluster is running, and it looks
>> to be repairing and our min_size is set to 2. Normally, I would expect
>> that the system would keep running normally from and end user
>> perspective when this happens, but the system is down. All mounts that
>> were up when this started look to be stale, and new mounts give the
>> following error:
>>
>> # mount -t ceph ceph-0:/ /test/ -o
>> name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev,rbytes
>> mount error 5 = Input/output error
>>
>> Any suggestions?
>>
>> Dan
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
