I've managed to get joao's assistance in tracking down the issue. I'll be updating the bug 7210.
Thanks joao and all! - WP On Thu, Mar 6, 2014 at 6:25 PM, YIP Wai Peng <yi...@comp.nus.edu.sg> wrote: > Ok, I think I got bitten by http://tracker.ceph.com/issues/7210, or > rather, the cppool command in > http://www.sebastien-han.fr/blog/2013/03/12/ceph-change-pg-number-on-the-fly/ > > I did use "rados cppool <oldpool> <newpool>" in a pool with snapshots > (openstack glance). A user feedback that ceph crashed when he deleted an > image in openstack. > > I'm now wondering if I can ignore the operation, or the openstack glance > pool, and get the mons to start up again. Any help will be greatly > appreciated! > > - WP > > > On Thu, Mar 6, 2014 at 5:33 PM, YIP Wai Peng <yi...@comp.nus.edu.sg>wrote: > >> Hi, >> >> I am currently facing a horrible situation. All my mons are crashing on >> startup. >> >> Here's a dump of mon.a.log. The last few ops are below. It seems to crash >> trying to remove a snap? Any ideas? >> >> - WP >> >> <snip> >> -10> 2014-03-06 17:04:38.838490 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 --> osd.9 192.168.116.27:6955/11604 -- >> mon_subscribe_ack(300s) v1 -- ?+0 0x32d0540 >> -9> 2014-03-06 17:04:38.838511 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 <== osd.1 192.168.116.24:6812/30221 6 ==== >> mon_subscribe({monmap=14+,osd_pg_creates=0}) v2 ==== 50+0+0 (3009623588 0 >> 0) 0x32d71c0 con 0x3224200 >> -8> 2014-03-06 17:04:38.838527 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 --> osd.1 192.168.116.24:6812/30221 -- >> mon_subscribe_ack(300s) v1 -- ?+0 0x32d3640 >> -7> 2014-03-06 17:04:38.838545 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286886 ==== >> forward(pg_stats(0 pgs tid 790 v 0) v1 caps allow rwx) to leader v1 ==== >> 398+0+0 (2470115819 0 0) 0x32f8c80 con 0x2f91760 >> -6> 2014-03-06 17:04:38.838570 7fb2a541a700 1 >> mon.a@0(leader).paxos(paxos >> active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838571 >> lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906 >> -5> 2014-03-06 17:04:38.838595 7fb2a541a700 1 >> mon.a@0(leader).paxos(paxos >> active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838597 >> lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906 >> -4> 2014-03-06 17:04:38.838626 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286887 ==== >> forward(osd_pgtemp(e26089 {6.fe=[]} v26089) v1 caps allow rwx) to leader v1 >> ==== 297+0+0 (2442013554 0 0) 0x32f8780 con 0x2f91760 >> -3> 2014-03-06 17:04:38.838662 7fb2a541a700 1 >> mon.a@0(leader).paxos(paxos >> active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838665 >> lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906 >> -2> 2014-03-06 17:04:38.838696 7fb2a541a700 1 -- >> 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286888 ==== >> forward(pool_op(delete unmanaged snap pool 10 auid 0 tid 27 name v0) v4 >> caps allow r) to leader v1 ==== 313+0+0 (3176715156 0 0) 0x32f8500 con >> 0x2f91760 >> -1> 2014-03-06 17:04:38.838715 7fb2a541a700 1 >> mon.a@0(leader).paxos(paxos >> active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838717 >> lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906 >> 0> 2014-03-06 17:04:38.840833 7fb2a541a700 -1 osd/osd_types.cc: In >> function 'void pg_pool_t::remove_unmanaged_snap(snapid_t)' thread >> 7fb2a541a700 time 2014-03-06 17:04:38.838745 >> osd/osd_types.cc: 799: FAILED assert(is_unmanaged_snaps_mode()) >> >> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) >> 1: /usr/bin/ceph-mon() [0x6c96e9] >> 2: (OSDMonitor::prepare_pool_op(MPoolOp*)+0x970) [0x5c3ad0] >> 3: (OSDMonitor::prepare_update(PaxosServiceMessage*)+0x1ab) [0x5c3d8b] >> 4: (PaxosService::dispatch(PaxosServiceMessage*)+0xa1a) [0x5940ea] >> 5: (Monitor::dispatch(MonSession*, Message*, bool)+0xdb) [0x56320b] >> 6: (Monitor::_ms_dispatch(Message*)+0x1fb) [0x5639fb] >> 7: (Monitor::handle_forward(MForward*)+0x9c2) [0x565092] >> 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x400) [0x563530] >> 9: (Monitor::_ms_dispatch(Message*)+0x1fb) [0x5639fb] >> 10: (Monitor::ms_dispatch(Message*)+0x32) [0x57f212] >> 11: (DispatchQueue::entry()+0x582) [0x7de6c2] >> 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7d994d] >> 13: (()+0x79d1) [0x7fb2ac05e9d1] >> 14: (clone()+0x6d) [0x7fb2aad95b6d] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to interpret this. >> <snip> >> > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
