On Wed, Oct 3, 2018 at 10:18 AM Graham Allan <g...@umn.edu> wrote:
> Following on from my previous adventure with recovering pgs in the face
> of failed OSDs, I now have my EC 4+2 pool oeprating with min_size=5
> which is as things should be.
>
> However I have one pg which is stuck in state remapped+incomplete
> because it has only 4 out of 6 osds running, and I have been unable to
> bring the missing two back into service.
>
> > PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
> > pg 70.82d is remapped+incomplete, acting [2147483647
> <(214)%20748-3647>,2147483647 <(214)%20748-3647>,190,448,61,315]
> (reducing pool .rgw.buckets.ec42 min_size from 5 may help; search
> ceph.com/docs for 'incomplete')
>
> I don't think I want to do anything with min_size as that would make all
> other pgs vulnerable to running dangerously undersized (unless there is
> any way to force that state for only a single pg). It seems to me that
> with 4/6 osds available, it should maybe be possible to force ceph to
> select one or two new osds to rebalance this pg to?
>
I think unfortunately the easiest thing for you to fix this will be to set
the min_size back to 4 until the PG is recovered (or at least has 5 shards
done). This will be fixed in a later version of Ceph and probably
backported, but sadly it's not done yet.
-Greg
>
> ceph pg query gives me (snippet):
>
> > "down_osds_we_would_probe": [
> > 98,
> > 233,
> > 238,
> > 239
> > ],
> > "peering_blocked_by": [],
> > "peering_blocked_by_detail": [
> > {
> > "detail": "peering_blocked_by_history_les_bound"
> > }
> > ]
>
> Of these, osd 98 appears to have a corrupt xfs filesystem
>
> osd 239 was the original osd to hold a shard of this pg but would not
> keep running, exiting with:
>
> > /build/ceph-12.2.7/src/osd/ECBackend.cc: 619: FAILED
> assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset(
> after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))
>
> osds 233 and 238 were otherwise evacuated (weight 0) osds to which I
> imported the pg shard from osd 239 (using ceph-objectstore-tool). After
> which they crash with the same assert. More specifically they seem to
> crash in the same way each time the pg becomes active and starts to
> backfill, on the same object:
>
> > -9> 2018-10-03 11:30:28.174586 7f94ce9c4700 5 osd.233 pg_epoch:
> 704441 pg[70.82ds1( v 704329'703106 (586066'698574,704329'703106]
> local-lis/les=704439/704440 n=102585 ec=21494/21494 lis/c 704439/588565
> les/c/f 704440/588566/0 68066
> > 6/704439/704439) [820,761,105,789,562,485]/[2147483647
> <(214)%20748-3647>,233,190,448,61,315]p233(1) r=1 lpr=704439
> pi=[21494,704439)/4 rops=1 bft=105(2),485(5),562(4),761(1),789(3),820(0)
> crt=704329'703106 lcod 0'0 mlcod 0'0 active+undersized+remapped+ba
> > ckfilling] backfill_pos is
> 70:b415ca14:::default.630943.7__shadow_Barley_GC_Project%2fBarley_GC_Project%2fRawdata%2fReads%2fCZOA.6150.7.38741.TGCTGG.fastq.gz.2~Vn8g0rMwpVY8eaW83TDzJ2mczLXAl3z.3_24:head
> > -8> 2018-10-03 11:30:28.174887 7f94ce9c4700 1 --
> 10.31.0.1:6854/2210291 --> 10.31.0.1:6854/2210291 --
> MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1,
> attrs_read=0)) v2 -- 0x7f9500472280 con 0
> > -7> 2018-10-03 11:30:28.174902 7f94db9de700 1 --
> 10.31.0.1:6854/2210291 <== osd.233 10.31.0.1:6854/2210291 0 ====
> MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1,
> attrs_read=0)) v2 ==== 0+0+0 (0 0 0) 0x7f9500472280
> > con 0x7f94fb72b000
> > -6> 2018-10-03 11:30:28.176267 7f94ead66700 5 --
> 10.31.0.1:6854/2210291 >> 10.31.0.4:6880/2181727 conn(0x7f94ff2a6000 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=946 cs=1 l=0). rx osd.61
> seq 9 0x7f9500472500 MOSDECSubOpRe
> > adReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2
> > -5> 2018-10-03 11:30:28.176281 7f94ead66700 1 --
> 10.31.0.1:6854/2210291 <== osd.61 10.31.0.4:6880/2181727 9 ====
> MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1,
> attrs_read=0)) v2 ==== 786745+0+0 (875698380 0 0) 0x
> > 7f9500472500 con 0x7f94ff2a6000
> > -4> 2018-10-03 11:30:28.177723 7f94ead66700 5 --
> 10.31.0.1:6854/2210291 >> 10.31.0.9:6920/13427 conn(0x7f94ff2bc800 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=46152 cs=1 l=0). rx
> osd.448 seq 8 0x7f94fe9d5980 MOSDECSubOpR
> > eadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2
> > -3> 2018-10-03 11:30:28.177738 7f94ead66700 1 --
> 10.31.0.1:6854/2210291 <== osd.448 10.31.0.9:6920/13427 8 ====
> MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1,
> attrs_read=0)) v2 ==== 786745+0+0 (2772477454 0 0) 0x
> > 7f94fe9d5980 con 0x7f94ff2bc800
> > -2> 2018-10-03 11:30:28.185788 7f94ea565700 5 --
> 10.31.0.1:6854/2210291 >> 10.31.0.7:6868/2012671 conn(0x7f94ff5c3800
> :6854 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4193 cs=1 l=0). rx
> osd.190 seq 10 0x7f9500472780 MOSDECSu
> > bOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0))
> v2
> > -1> 2018-10-03 11:30:28.185815 7f94ea565700 1 --
> 10.31.0.1:6854/2210291 <== osd.190 10.31.0.7:6868/2012671 10 ====
> MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1,
> attrs_read=0)) v2 ==== 786745+0+0 (2670842780 0 0)
> > 0x7f9500472780 con 0x7f94ff5c3800
> > 0> 2018-10-03 11:30:28.194795 7f94ce9c4700 -1
> /build/ceph-12.2.7/src/osd/ECBackend.cc: In function 'void
> ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)'
> thread 7f94ce9c4700 time 2018-10-03 11:30:28.19026
> > 0
> > /build/ceph-12.2.7/src/osd/ECBackend.cc: 619: FAILED
> assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset(
> after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))
>
>
> Is there anything I can do to help one of these osds (probably 233 or
> 238) start, such as "ceph-objectstore-tool --op repair"...? There seems
> little to lose by trying but there isn't a lot of documentation on the
> operations available in ceph-objectstore-tool.
>
> I also know of the option "osd_find_best_info_ignore_history_les" but
> little of what it actually does, other than being "dangerous". There are
> many past intervals listed by pg query, but no "maybe_went_rw" flags so
> perhaps it is safe to revert...?
>
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
