[ceph-users] Re: ceph mon failing to start
Are the two running mons also running 14.2.9 ? --- dan On Mon, Mar 28, 2022 at 8:27 AM Tomáš Hodek wrote: > > Hi, I have 3 node ceph cluster (managed via proxmox). Got single node > fatal failure and replaced it. Os boots correctly, however monitor on > failed node did not start successfully; Other 2 monitors are OK, ceph > status is healthy: > > ceph -s > cluster: > id: 845868a1-9902-4b61-aa06-0767cb09f1c2 > health: HEALTH_OK > > services: > mon: 2 daemons, quorum pxmx1,pxmx3 (age 2h) > mgr: pxmx1(active, since 56m), standbys: pxmx3 > osd: 18 osds: 18 up (since 111m), 18 in (since 3h) > > data: > pools: 1 pools, 256 pgs > objects: 2.12M objects, 8.1 TiB > usage: 24 TiB used, 21 TiB / 45 TiB avail > pgs: 256 active+clean > > content of ceph.conf > > [global] > auth_client_required = cephx > auth_cluster_required = cephx > auth_service_required = cephx > cluster_network = 10.60.10.1/24 > fsid = 845868a1-9902-4b61-aa06-0767cb09f1c2 > mon_allow_pool_delete = true > mon_host = 10.60.10.1 10.60.10.3 10.60.10.2 > osd_pool_default_min_size = 2 > osd_pool_default_size = 3 > public_network = 10.60.10.1/24 > > [client] > keyring = /etc/pve/priv/$cluster.$name.keyring > > [mds] > keyring = /var/lib/ceph/mds/ceph-$id/keyring > > Monitor is failing (at least as I understand the problem) with following > logged error: > > mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id > > whole mon log attached. > > I have tried to scrap dead monitor and recreate it via proxmoxes gui, > shell and even have created content /var/lib/ceph/mon/ manually and > tried to run monitor from terminal. It starts, listens to connections on > port 3300 and 6789, but does not communicate properly with other > remaining mons. > > thanks for info > > Tomas Hodek___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RBD Exclusive lock to shared lock
> > > > > My use case would be a HA cluster where a VM is mapping an rbd image, > and then it encounters some network issue. An other node of the HA > cluster could start the VM and map again the image, but if the > networking is fixed on the first VM that would keep using the already > mapped image. Here If I could instruct my second VM to treat the lock as > exclusive after an automatic failover, then I'm protected against data > corruption when the networking of initial VM is fixed. But I assume that > a STONITH kind of fencing could also do the job (if it can be > implemented). Is it already possible to configure libvirt directly for this, or are the hook scripts still necessary? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] PG down, due to 3 OSD failing
Hallo, all of a sudden, 3 of my OSDs failed, showing similar messages in the log: . -5> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 ec=148456/148456 lis/c 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 unknown mbc={}] enter Started -4> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 ec=148456/148456 lis/c 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 unknown mbc={}] enter Start -3> 2022-03-28 14:19:02.451 7fc20fe99700 1 osd.145 pg_epoch: 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 ec=148456/148456 lis/c 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 unknown mbc={}] state: transitioning to Stray -2> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 ec=148456/148456 lis/c 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 unknown mbc={}] exit Start 0.08 0 0.00 -1> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 ec=148456/148456 lis/c 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 unknown mbc={}] enter Started/Stray 0> 2022-03-28 14:19:02.451 7fc20f698700 -1 *** Caught signal (Aborted) ** in thread 7fc20f698700 thread_name:tp_osd_tp ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0x12ce0) [0x7fc2327dcce0] 2: (gsignal()+0x10f) [0x7fc231452a4f] 3: (abort()+0x127) [0x7fc231425db5] 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b4) [0x55b8139cb671] 5: (PG::check_past_interval_bounds() const+0xc16) [0x55b813b586f6] 6: (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x3e8) [0x55b813b963d8] 7: (boost::statechart::simple_statePG::RecoveryState::RecoveryMachine, boost::mpl::listmpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x7d) [0x55b813bdd32d] 8: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PG::RecoveryCtx*)+0x39d) [0x55b813b7b5fd] 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x2e9) [0x55b813ad14e9] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0xaa) [0x55b813ae345a] 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x55) [0x55b813d66c15] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1366) [0x55b813adff46] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b8140dc944] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b8140df514] 15: (()+0x81cf) [0x7fc2327d21cf] 16: (clone()+0x43) [0x7fc23143dd83] Trying to "activate --all", rebotting server, and such, did not help. I am now stuck with one PG (85.25) down, find below the output from "query". The PG belongs to a 3+2 erasure-coded pool. As the devices corresponding to the 3 down OSDs are properly mounted, is there a way to get PG.ID=85.25 from the devices and copy it elsewhere? Actually, I tried to find 85.25 in the 3 down OSDs with command: ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/ --no-mon-config --pgid 85.25 --op export --file /tmp/pg_85-25 PG '85.25' not found which puzzled me... is there a way to search such PG.ID over the whole cluster? Thanks for your help! Fulvio ~]# ceph --cluster cephpa1 health detail | grep down . PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down pg 85.25 is down+remapped, acting [2147483647,2147483647,96,2147483647,2147483647] ~]# ceph --cluster cephpa1 pg 85.25 query { "state": "down+remapped", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 617667, "up": [ 2147483647, 2147483647, 2147483647, 2147483647, 2147483647 ], "acting": [ 2147483647, 2147483647, 96, 2147483647, 2147483647 ], "info": { "pgid": "85.25s2", "last_update": "606021'521273", "last_complete": "606021'521273", "log_tail": "605873'518175", "last
[ceph-users] Re: PG down, due to 3 OSD failing
Hi Fulvio, You can check (offline) which PGs are on an OSD with the list-pgs op, e.g. ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/ --op list-pgs The EC pgs have a naming convention like 85.25s1 etc.. for the various k/m EC shards. -- dan On Mon, Mar 28, 2022 at 2:29 PM Fulvio Galeazzi wrote: > > Hallo, > all of a sudden, 3 of my OSDs failed, showing similar messages in > the log: > > . > -5> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > ec=148456/148456 lis/c > 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > unknown mbc={}] > enter Started > -4> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > ec=148456/148456 lis/c > 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > unknown mbc={}] > enter Start > -3> 2022-03-28 14:19:02.451 7fc20fe99700 1 osd.145 pg_epoch: > 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > ec=148456/148456 lis/c > 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > unknown mbc={}] > state: transitioning to Stray > -2> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > ec=148456/148456 lis/c > 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > unknown mbc={}] > exit Start 0.08 0 0.00 > -1> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > ec=148456/148456 lis/c > 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > unknown mbc={}] > enter Started/Stray > 0> 2022-03-28 14:19:02.451 7fc20f698700 -1 *** Caught signal > (Aborted) ** > in thread 7fc20f698700 thread_name:tp_osd_tp > > ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable) > 1: (()+0x12ce0) [0x7fc2327dcce0] > 2: (gsignal()+0x10f) [0x7fc231452a4f] > 3: (abort()+0x127) [0x7fc231425db5] > 4: (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string, > std::allocator > const&)+0x1b4) [0x55b8139cb671] > 5: (PG::check_past_interval_bounds() const+0xc16) [0x55b813b586f6] > 6: (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x3e8) > [0x55b813b963d8] > 7: (boost::statechart::simple_state PG::RecoveryState::RecoveryMachine, boost::mpl::list mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na>, > (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base > const&, void const*)+0x7d) [0x55b813bdd32d] > 8: (PG::handle_advance_map(std::shared_ptr, > std::shared_ptr, std::vector >&, > int, std::vector >&, int, > PG::RecoveryCtx*)+0x39d) [0x55b813b7b5fd] > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, > PG::RecoveryCtx*)+0x2e9) [0x55b813ad14e9] > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, > std::shared_ptr, ThreadPool::TPHandle&)+0xaa) > [0x55b813ae345a] > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, > ThreadPool::TPHandle&)+0x55) [0x55b813d66c15] > 12: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x1366) [0x55b813adff46] > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) > [0x55b8140dc944] > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b8140df514] > 15: (()+0x81cf) [0x7fc2327d21cf] > 16: (clone()+0x43) [0x7fc23143dd83] > > Trying to "activate --all", rebotting server, and such, did not help. > > I am now stuck with one PG (85.25) down, find below the output from "query". > > The PG belongs to a 3+2 erasure-coded pool. > As the devices corresponding to the 3 down OSDs are properly mounted, is > there a way to get PG.ID=85.25 from the devices and copy it elsewhere? > Actually, I tried to find 85.25 in the 3 down OSDs with command: > ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/ > --no-mon-config --pgid 85.25 --op export --file /tmp/pg_85-25 > PG '85.25' not found >which puzzled me... is there a way to search such PG.ID over the > whole cluster? > >Thanks for your help! > > Fulvio > > > > ~]# ceph --cluster cephpa1 health detail | grep down > . > PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down > pg 85.25 is down+remapped, acting > [2147483647,2147483647,96,2147483647,2147483647] > ~]# ceph --cluster cephpa1 pg 85.25 query > { >
[ceph-users] Re: quincy v17.2.0 QE Validation status
On Mon, Mar 28, 2022 at 2:48 PM Yuri Weinstein wrote: > > We are trying to release v17.2.0 as soon as possible. > And need to do a quick approval of tests and review failures. > > Still outstanding are two PRs: > https://github.com/ceph/ceph/pull/45673 > https://github.com/ceph/ceph/pull/45604 > > Build failing and I need help to fix it ASAP. > ( > https://shaman.ceph.com/builds/ceph/wip-yuri11-testing-2022-03-28-0907-quincy/61b142c76c991abe3fe77390e384b025e1711757/ > ) > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/55089 > Release Notes - https://github.com/ceph/ceph/pull/45048 > > Seeking approvals for: > > smoke - Neha, Josh (the failure appears reproducible) smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_pjd}} is failing consistently in the kclient task. Venky, can you please check whether this is a configuration/test issue or a bug? Here's an example http://pulpito.front.sepia.ceph.com/yuriw-2022-03-28_19:24:40-smoke-quincy-distro-default-smithi/6765567/. Thanks, Neha > rgw - Casey > fs - Venky, Gerg > rbd - Ilya, Deepika > krbd Ilya, Deepika > upgrade/octopus-x - Casey > powercycle - Brag (SELinux denials) > ceph-volume - Guillaume, David G > > Please reply to this email with approval and/or tracks of know issues/PRs to > address them. > > Thx > YuriW > ___ > Dev mailing list -- d...@ceph.io > To unsubscribe send an email to dev-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: quincy v17.2.0 QE Validation status
Hey Yuri, On Tue, Mar 29, 2022 at 3:18 AM Yuri Weinstein wrote: > > We are trying to release v17.2.0 as soon as possible. > And need to do a quick approval of tests and review failures. > > Still outstanding are two PRs: > https://github.com/ceph/ceph/pull/45673 > https://github.com/ceph/ceph/pull/45604 There are some outstanding CephFS PRs - labeled needs-qa, quincy-batch-1. Do you plan to include those? > > Build failing and I need help to fix it ASAP. > ( > https://shaman.ceph.com/builds/ceph/wip-yuri11-testing-2022-03-28-0907-quincy/61b142c76c991abe3fe77390e384b025e1711757/ > ) > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/55089 > Release Notes - https://github.com/ceph/ceph/pull/45048 > > Seeking approvals for: > > smoke - Neha, Josh (the failure appears reproducible) > rgw - Casey > fs - Venky, Gerg > rbd - Ilya, Deepika > krbd Ilya, Deepika > upgrade/octopus-x - Casey > powercycle - Brag (SELinux denials) > ceph-volume - Guillaume, David G > > Please reply to this email with approval and/or tracks of know issues/PRs to > address them. > > Thx > YuriW > ___ > Dev mailing list -- d...@ceph.io > To unsubscribe send an email to dev-le...@ceph.io -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: quincy v17.2.0 QE Validation status
On Tue, Mar 29, 2022 at 10:56 AM Venky Shankar wrote: > > Hey Yuri, > > > On Tue, Mar 29, 2022 at 3:18 AM Yuri Weinstein wrote: > > > > We are trying to release v17.2.0 as soon as possible. > > And need to do a quick approval of tests and review failures. > > > > Still outstanding are two PRs: > > https://github.com/ceph/ceph/pull/45673 > > https://github.com/ceph/ceph/pull/45604 > > There are some outstanding CephFS PRs - labeled needs-qa, quincy-batch-1. > > Do you plan to include those? We would need PR https://github.com/ceph/ceph/pull/45558 at the minimum to avoid a regression > > > > > Build failing and I need help to fix it ASAP. > > ( > > https://shaman.ceph.com/builds/ceph/wip-yuri11-testing-2022-03-28-0907-quincy/61b142c76c991abe3fe77390e384b025e1711757/ > > ) > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/55089 > > Release Notes - https://github.com/ceph/ceph/pull/45048 > > > > Seeking approvals for: > > > > smoke - Neha, Josh (the failure appears reproducible) > > rgw - Casey > > fs - Venky, Gerg > > rbd - Ilya, Deepika > > krbd Ilya, Deepika > > upgrade/octopus-x - Casey > > powercycle - Brag (SELinux denials) > > ceph-volume - Guillaume, David G > > > > Please reply to this email with approval and/or tracks of know issues/PRs > > to address them. > > > > Thx > > YuriW > > ___ > > Dev mailing list -- d...@ceph.io > > To unsubscribe send an email to dev-le...@ceph.io > > > > -- > Cheers, > Venky -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io