2013/10/31 Иван Кудрявцев <kudryavtsev...@bw-sw.com>
> Hello, List.
>
> I met very big trouble during ceph upgrade from bobtail to cuttlefish.
>
> My OSDs started to crash to stale so LA went to 100+ on node, after I stop
> OSD I unable to launch it again because of errors. So, I started to
> reformat OSDs and eventually meet that I have "incomplete" PGs and It seems
> to me that I lost actual data (It seems some drives I use is buggy and lost
> partition and filesystem at all after reboot node as them was empty, and I
> placed them in all my nodes in one time). I use ceph as source for RBD
> images, so all my images are unaccessible and ceph health is:
>
> root@ceph-osd-1-2:~# ceph -s
> cluster e74c9e0b-40ce-45e4-904a-676cbaaf042d
> health HEALTH_WARN 536 pgs backfill; 28 pgs backfilling; 546 pgs
> degraded; 19 pgs down; 76 pgs incomplete; 33 pgs peering; 23 pgs stale; 109
> pgs stuck inactive; 23 pgs stuck stale; 673 pgs stuck unclean; 11 requests
> are blocked > 32 sec; recovery 2769254/9737405 degraded (28.439%);
> recovering 76 o/s, 169MB/s; mds cluster is degraded; clock skew detected on
> mon.0, mon.1
> monmap e1: 3 mons at {0=
> 10.252.0.3:6789/0,1=10.252.0.4:6789/0,2=10.252.0.2:6789/0}, election
> epoch 5486, quorum 0,1,2 0,1,2
> osdmap e52800: 48 osds: 41 up, 41 in
> pgmap v22294835: 1344 pgs: 671 active+clean, 18
> active+remapped+wait_backfill, 150 active+degraded+wait_backfill, 12
> down+peering, 14 stale+peering, 13 active+degraded+backfilling, 368
> active+degraded+remapped+wait_backfill, 7 stale+down+peering, 74
> incomplete, 15 active+degraded+remapped+backfilling, 2 stale+incomplete;
> 6883 GB data, 14627 GB used, 31364 GB / 46031 GB avail; 2769254/9737405
> degraded (28.439%); recovering 76 o/s, 169MB/s
> mdsmap e564: 1/1/1 up {0=2=up:replay}, 2 up:standby
>
> Actually, my images stored in RBD pool, but when I try to access it I get:
>
> root@ceph-osd-1-2:~# rbd ls
> rbd: pool rbd doesn't contain rbd images
> root@ceph-osd-1-2:~#
>
rbd *-p POOLNAME* ls
>
>
> however, I can access images directly:
>
> root@ceph-osd-1-2:~# rbd info lun-000000017
> rbd image 'lun-000000017':
> size 10000 MB in 320000 objects
> order 15 (32768 bytes objects)
> block_name_prefix: rb.0.3c07.238e1f29
> format: 1
>
> and what I really would like is to export data even with errors as usual
> files and try to check and fix some using fsck. As I understand I should
> someway fix "incomplete" pgs and "inactive" pgs but I don't really
> understand how to do it?
>
> root@ceph-osd-1-2:~# ceph pg dump | awk '{print $1, $14, $2
> ,$6/1024/1024 , $9}' | grep '\.' | grep inc
> dumped all in format plain
> 2.1be [43,15,37] 1603 4018.79 incomplete
> 2.1aa [20,37,44] 4875 11930.5 incomplete
> 0.1a4 [45,37,27] 0 0 incomplete
> 2.1a1 [43,45,37] 988 2497.98 incomplete
> 2.1a0 [30,29,37] 0 0 incomplete
> 0.1a2 [30,29,37] 0 0 incomplete
> 1.1a3 [45,37,27] 0 0 incomplete
> 2.1a2 [45,37,27] 0 0 incomplete
> 1.1a1 [30,29,37] 0 0 incomplete
> 2.19d [18,42,24] 0 0 incomplete
> 0.19f [18,42,24] 0 0 incomplete
> 1.19e [18,42,24] 0 0 incomplete
> 0.198 [1,43,37] 0 0 incomplete
> 1.197 [1,43,37] 0 0 incomplete
> 2.196 [1,43,37] 0 0 incomplete
> 2.183 [28,30,37] 0 0 incomplete
> 2.15f [27,32,37] 0 0 incomplete
> 0.15c [42,31,37] 0 0 incomplete
> 1.15b [42,31,37] 0 0 incomplete
> 2.15a [42,31,37] 0 0 incomplete
> 2.137 [7,30,37] 0 0 incomplete
> 0.130 [41,31,25] 139 446.865 stale+incomplete
> 0.12d [31,37,27] 0 0 incomplete
> 1.12c [31,37,27] 0 0 incomplete
> 2.12b [31,37,27] 0 0 incomplete
> 0.124 [30,37,42] 0 0 incomplete
> 1.123 [30,37,42] 0 0 incomplete
> 2.122 [30,37,42] 0 0 incomplete
> 0.109 [31,37,27] 0 0 incomplete
> 1.108 [31,37,27] 0 0 incomplete
> 2.107 [31,37,27] 0 0 incomplete
> 2.f4 [12,37,20] 3132 7941.63 incomplete
> 2.ee [12,11,37] 3959 9825.61 incomplete
> 0.ec [32,37,42] 0 0 incomplete
> 1.eb [32,37,42] 0 0 incomplete
> 2.ea [32,37,42] 0 0 incomplete
> 2.de [45,37,21] 1228 2960.71 incomplete
> 2.d8 [9,28,37] 832 2029.15 incomplete
> 2.c1 [13,30,37] 0 0 incomplete
> 2.b9 [32,2,37] 2971 7305.59 incomplete
> 2.a4 [30,22,33] 1343 3365.41 incomplete
> 2.9e [43,37,45] 651 1633.18 incomplete
> 2.9a [30,2,37] 0 0 incomplete
> 2.95 [6,43,37] 0 0 incomplete
> 1.96 [6,43,37] 0 0 incomplete
> 0.97 [6,43,37] 4 8.00209 incomplete
> 2.91 [47,28,1] 4518 11066.5 stale+incomplete
> 0.64 [32,37,8] 0 0 incomplete
> 1.63 [32,37,8] 0 0 incomplete
> 2.62 [32,37,8] 0 0 incomplete
> 0.59 [32,37,21] 311 1010.57 incomplete
> 2.55 [27,16,37] 0 0 incomplete
> 0.57 [27,16,37] 0 0 incomplete
> 1.56 [27,16,37] 0 0 incomplete
> 2.57 [32,37,21] 42 72.6953 incomplete
> 2.4c [11,37,30] 3315 8320.87 incomplete
> 0.49 [18,7,37] 0 0 incomplete
> 1.48 [18,7,37] 0 0 incomplete
> 2.47 [18,7,37] 0 0 incomplete
> 2.3d [28,44,37] 0 0 incomplete
> 0.3f [28,44,37] 0 0 incomplete
> 1.3e [28,44,37] 0 0 incomplete
> 2.2a [43,16,37] 5379 13236.4 incomplete
> 0.18 [32,42,37] 0 0 incomplete
> 1.17 [32,42,37] 0 0 incomplete
> 2.16 [32,42,37] 0 0 incomplete
> 2.d [45,29,37] 0 0 incomplete
> 1.e [45,29,37] 0 0 incomplete
> 0.f [45,29,37] 0 0 incomplete
> 2.c [18,20,37] 1208 3002.21 incomplete
> 2.9 [45,37,27] 0 0 incomplete
> 1.a [45,37,27] 0 0 incomplete
> 0.b [45,37,27] 28 80.5886 incomplete
> 2.7 [22,37,30] 2889 7272.09 incomplete
> 0.1 [31,37,27] 39 135.707 incomplete
> 1.0 [31,37,27] 0 0 incomplete
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
--
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
