Thank you, Brad. We’ll try to upgrade 14.2.9 today.
> 2020年4月23日 上午7:21,Brad Hubbard <bhubb...@redhat.com> 写道: > > On Tue, Apr 21, 2020 at 11:39 PM XuYun <yu...@me.com <mailto:yu...@me.com>> > wrote: >> >> Dear ceph users, >> >> We are experiencing sporadic mgr crash in all three ceph clusters (version >> 14.2.6 and version 14.2.8), the crash log is: >> >> 2020-04-17 23:10:08.986 7fed7fe07700 -1 >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: >> In function 'const char* ceph::buffer::v14_2_0::ptr::c_str() const' thread >> 7fed7fe07700 time 2020-04-17 23:10:08.984887 >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: >> 578: FAILED ceph_assert(_raw) >> >> ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus >> (stable) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x14a) [0x7fed8605c325] >> 2: (()+0x2534ed) [0x7fed8605c4ed] >> 3: (()+0x5a21ed) [0x7fed863ab1ed] >> 4: (PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)+0xbd) >> [0x7fed863840ed] >> 5: (AsyncConnection::_try_send(bool)+0xb6) [0x7fed8632fc76] >> 6: (ProtocolV2::write_message(Message*, bool)+0x832) [0x7fed8635bf52] >> 7: (ProtocolV2::write_event()+0x175) [0x7fed863718c5] >> 8: (AsyncConnection::handle_write()+0x40) [0x7fed86332600] >> 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned >> long, std::ratio<1l, 1000000000l> >*)+0x1397) [0x7fed8637f997] >> 10: (()+0x57c977) [0x7fed86385977] >> 11: (()+0x80bdaf) [0x7fed86614daf] >> 12: (()+0x7e65) [0x7fed8394ce65] >> 13: (clone()+0x6d) [0x7fed825fa88d] >> >> 2020-04-17 23:10:08.990 7fed7ee05700 -1 *** Caught signal (Segmentation >> fault) ** >> in thread 7fed7ee05700 thread_name:msgr-worker-2 >> >> ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus >> (stable) >> 1: (()+0xf5f0) [0x7fed839545f0] >> 2: (ceph::buffer::v14_2_0::ptr::release()+0x8) [0x7fed863aafd8] >> 3: >> (ceph::crypto::onwire::AES128GCM_OnWireTxHandler::~AES128GCM_OnWireTxHandler()+0x59) >> [0x7fed86388669] >> 4: (ProtocolV2::reset_recv_state()+0x11f) [0x7fed8635f5af] >> 5: (ProtocolV2::stop()+0x77) [0x7fed8635f857] >> 6: >> (ProtocolV2::handle_existing_connection(boost::intrusive_ptr<AsyncConnection>)+0x5ef) >> [0x7fed86374f8f] >> 7: (ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)+0xd9c) >> [0x7fed8637673c] >> 8: (ProtocolV2::handle_frame_payload()+0x1fb) [0x7fed86376c1b] >> 9: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7fed86376e70] >> 10: >> (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, >> ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x44d) [0x7fed863773cd] >> 11: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x7fed86360534] >> 12: (AsyncConnection::process()+0x186) [0x7fed86330656] >> 13: (EventCenter::process_events(unsigned int, >> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) >> [0x7fed8637f015] >> 14: (()+0x57c977) [0x7fed86385977] >> 15: (()+0x80bdaf) [0x7fed86614daf] >> 16: (()+0x7e65) [0x7fed8394ce65] >> 17: (clone()+0x6d) [0x7fed825fa88d] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to >> interpret this. >> >> Any thoughts about this issue? > > Looks like https://tracker.ceph.com/issues/42026 > <https://tracker.ceph.com/issues/42026> which was recently > backported to the Nautilus branch via > https://github.com/ceph/ceph/pull/33820 > <https://github.com/ceph/ceph/pull/33820> > > You could try a build with those patches or wait for 14.2.9 > > -- > Cheers, > Brad > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-le...@ceph.io > <mailto:ceph-users-le...@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io