Hi Fulvio, I'm not sure why that PG doesn't register. But let's look into your log. The relevant lines are:
-635> 2022-03-30 14:49:57.810 7ff904970700 -1 log_channel(cluster) log [ERR] : 85.12s0 past_intervals [616435,616454) start interval does not contain the required bound [605868,616454) start -628> 2022-03-30 14:49:57.810 7ff904970700 -1 osd.158 pg_epoch: 616454 pg[85.12s0( empty local-lis/les=0/0 n=0 ec=616435/616435 lis/c 605866/605866 les/c/f 605867/605868/0 616453/616454/616454) [158,168,64,102,156]/[67,91,82,121,112]p67(0) r=-1 lpr=616454 pi=[616435,616454)/0 crt=0'0 remapped NOTIFY mbc={}] 85.12s0 past_intervals [616435,616454) start interval does not contain the required bound [605868,616454) start -355> 2022-03-30 14:49:57.816 7ff904970700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc: In function 'void PG::check_past_interval_bounds() const' thread 7ff904970700 time 2022-03-30 14:49:57.811165 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc: 956: ceph_abort_msg("past_interval start interval mismatch") What is the output of `ceph pg 85.12 query` ? What's the history of that PG? was it moved around recently prior to this crash? Are the other down osds also hosting broken parts of PG 85.12 ? Cheers, Dan On Wed, Mar 30, 2022 at 3:00 PM Fulvio Galeazzi <fulvio.galea...@garr.it> wrote: > > Ciao Dan, > this is what I did with chunk s3, copying it from osd.121 to > osd.176 (which is managed by the same host). > > But still > pg 85.25 is stuck stale for 85029.707069, current state > stale+down+remapped, last acting > [2147483647,2147483647,96,2147483647,2147483647] > > So "health detail" apparently plainly ignores osd.176: moreover, its > output only shows OSD 96, but I just checked again and the other chunks > are still on OSDs 56,64,140,159 which are all "up". > > By the way, you talk about a "bug" in your message: do you have any > specific one in mind, or was it just a generic synonym for "problem"? > By the way, I uploaded here: > https://pastebin.ubuntu.com/p/dTfPkMb7mD/ > a few hundreds of lines from one of the failed OSDs upon "activate --all". > > Thanks > > Fulvio > > On 29/03/2022 10:53, Dan van der Ster wrote: > > Hi Fulvio, > > > > I don't think upmap will help -- that is used to remap where data > > should be "up", but your problem is more that the PG chunks are not > > going active due to the bug. > > > > What happens if you export one of the PG chunks then import it to > > another OSD -- does that chunk become active? > > > > -- dan > > > > > > > > On Tue, Mar 29, 2022 at 10:51 AM Fulvio Galeazzi > > <fulvio.galea...@garr.it> wrote: > >> > >> Hallo again Dan, I am afraid I'd need a little more help, please... > >> > >> Current status is as follows. > >> > >> This is where I moved the chunk which was on osd.121: > >> ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-176 > >> --no-mon-config --op list-pgs | grep ^85\.25 > >> 85.25s3 > >> > >> while other chunks are (server - osd.id): > >> === r2srv101.pa1.box.garr - 96 > >> 85.25s2 > >> === r2srv100.pa1.box.garr - 121 <-- down, chunk is on osd.176 > >> 85.25s3 > >> === r2srv100.pa1.box.garr - 159 > >> 85.25s1 > >> 85.25s4 > >> === r1-sto09.pa1.box.garr - 56 > >> 85.25s4 > >> === r1-sto09.pa1.box.garr - 64 > >> 85.25s0 > >> === r3srv15.pa1.box.garr - 140 > >> 85.25s1 > >> > >> Health detail shows that just one chunk can be found (if I understand > >> the output correctly): > >> > >> ~]# ceph health detail | grep 85\.25 > >> pg 85.25 is stuck stale for 5680.315732, current state > >> stale+down+remapped, last acting > >> [2147483647,2147483647,96,2147483647,2147483647] > >> > >> Can I run some magic upmap command to explain my cluster where all the > >> chunks are? What would be the right syntax? > >> Little additional problem: I see s1 and s4 twice... I guess this was due > >> to remapping, as I was adding disks to the cluster: which one is the > >> right copy? > >> > >> Thanks! > >> > >> Fulvio > >> > >> Il 3/29/2022 9:35 AM, Fulvio Galeazzi ha scritto: > >>> Thanks a lot, Dan! > >>> > >>> > The EC pgs have a naming convention like 85.25s1 etc.. for the various > >>> > k/m EC shards. > >>> > >>> That was the bit of information I was missing... I was looking for the > >>> wrong object. > >>> I can now go on and export/import that one PGid chunk. > >>> > >>> Thanks again! > >>> > >>> Fulvio > >>> > >>> On 28/03/2022 16:27, Dan van der Ster wrote: > >>>> Hi Fulvio, > >>>> > >>>> You can check (offline) which PGs are on an OSD with the list-pgs op, > >>>> e.g. > >>>> > >>>> ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/ > >>>> --op list-pgs > >>>> > >>>> > >>>> -- dan > >>>> > >>>> > >>>> On Mon, Mar 28, 2022 at 2:29 PM Fulvio Galeazzi > >>>> <fulvio.galea...@garr.it> wrote: > >>>>> > >>>>> Hallo, > >>>>> all of a sudden, 3 of my OSDs failed, showing similar messages in > >>>>> the log: > >>>>> > >>>>> ..... > >>>>> -5> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > >>>>> 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > >>>>> ec=148456/148456 lis/c > >>>>> 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > >>>>> [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > >>>>> unknown mbc={}] > >>>>> enter Started > >>>>> -4> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > >>>>> 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > >>>>> ec=148456/148456 lis/c > >>>>> 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > >>>>> [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > >>>>> unknown mbc={}] > >>>>> enter Start > >>>>> -3> 2022-03-28 14:19:02.451 7fc20fe99700 1 osd.145 pg_epoch: > >>>>> 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > >>>>> ec=148456/148456 lis/c > >>>>> 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > >>>>> [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > >>>>> unknown mbc={}] > >>>>> state<Start>: transitioning to Stray > >>>>> -2> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > >>>>> 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > >>>>> ec=148456/148456 lis/c > >>>>> 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > >>>>> [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > >>>>> unknown mbc={}] > >>>>> exit Start 0.000008 0 0.000000 > >>>>> -1> 2022-03-28 14:19:02.451 7fc20fe99700 5 osd.145 pg_epoch: > >>>>> 616454 pg[70.2c6s1( empty local-lis/les=612106/612107 n=0 > >>>>> ec=148456/148456 lis/c > >>>>> 612106/612106 les/c/f 612107/612107/0 612106/612106/612101) > >>>>> [168,145,102,96,112,124,128,134,56,34]p168(0) r=1 lpr=616429 crt=0'0 > >>>>> unknown mbc={}] > >>>>> enter Started/Stray > >>>>> 0> 2022-03-28 14:19:02.451 7fc20f698700 -1 *** Caught signal > >>>>> (Aborted) ** > >>>>> in thread 7fc20f698700 thread_name:tp_osd_tp > >>>>> > >>>>> ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > >>>>> nautilus (stable) > >>>>> 1: (()+0x12ce0) [0x7fc2327dcce0] > >>>>> 2: (gsignal()+0x10f) [0x7fc231452a4f] > >>>>> 3: (abort()+0x127) [0x7fc231425db5] > >>>>> 4: (ceph::__ceph_abort(char const*, int, char const*, > >>>>> std::__cxx11::basic_string<char, std::char_traits<char>, > >>>>> std::allocator<char> > const&)+0x1b4) [0x55b8139cb671] > >>>>> 5: (PG::check_past_interval_bounds() const+0xc16) [0x55b813b586f6] > >>>>> 6: (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x3e8) > >>>>> [0x55b813b963d8] > >>>>> 7: (boost::statechart::simple_state<PG::RecoveryState::Reset, > >>>>> PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, > >>>>> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > >>>>> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > >>>>> mpl_::na, mpl_::na, mpl_::na, mpl_::na>, > >>>>> (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base > >>>>> > >>>>> const&, void const*)+0x7d) [0x55b813bdd32d] > >>>>> 8: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, > >>>>> std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, > >>>>> int, std::vector<int, std::allocator<int> >&, int, > >>>>> PG::RecoveryCtx*)+0x39d) [0x55b813b7b5fd] > >>>>> 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, > >>>>> PG::RecoveryCtx*)+0x2e9) [0x55b813ad14e9] > >>>>> 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, > >>>>> std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xaa) > >>>>> [0x55b813ae345a] > >>>>> 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, > >>>>> ThreadPool::TPHandle&)+0x55) [0x55b813d66c15] > >>>>> 12: (OSD::ShardedOpWQ::_process(unsigned int, > >>>>> ceph::heartbeat_handle_d*)+0x1366) [0x55b813adff46] > >>>>> 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned > >>>>> int)+0x5c4) > >>>>> [0x55b8140dc944] > >>>>> 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) > >>>>> [0x55b8140df514] > >>>>> 15: (()+0x81cf) [0x7fc2327d21cf] > >>>>> 16: (clone()+0x43) [0x7fc23143dd83] > >>>>> > >>>>> Trying to "activate --all", rebotting server, and such, did not help. > >>>>> > >>>>> I am now stuck with one PG (85.25) down, find below the output from > >>>>> "query". > >>>>> > >>>>> The PG belongs to a 3+2 erasure-coded pool. > >>>>> As the devices corresponding to the 3 down OSDs are properly mounted, is > >>>>> there a way to get PG.ID=85.25 from the devices and copy it elsewhere? > >>>>> Actually, I tried to find 85.25 in the 3 down OSDs with command: > >>>>> ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/ > >>>>> --no-mon-config --pgid 85.25 --op export --file /tmp/pg_85-25 > >>>>> PG '85.25' not found > >>>>> which puzzled me... is there a way to search such PG.ID over the > >>>>> whole cluster? > >>>>> > >>>>> Thanks for your help! > >>>>> > >>>>> Fulvio > >>>>> > >>>>> ==================== > >>>>> > >>>>> ~]# ceph --cluster cephpa1 health detail | grep down > >>>>> ..... > >>>>> PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down > >>>>> pg 85.25 is down+remapped, acting > >>>>> [2147483647,2147483647,96,2147483647,2147483647] > >>>>> ~]# ceph --cluster cephpa1 pg 85.25 query > >>>>> { > >>>>> "state": "down+remapped", > >>>>> "snap_trimq": "[]", > >>>>> "snap_trimq_len": 0, > >>>>> "epoch": 617667, > >>>>> "up": [ > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647 > >>>>> ], > >>>>> "acting": [ > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 96, > >>>>> 2147483647, > >>>>> 2147483647 > >>>>> ], > >>>>> "info": { > >>>>> "pgid": "85.25s2", > >>>>> "last_update": "606021'521273", > >>>>> "last_complete": "606021'521273", > >>>>> "log_tail": "605873'518175", > >>>>> "last_user_version": 521273, > >>>>> "last_backfill": "MAX", > >>>>> "last_backfill_bitwise": 1, > >>>>> "purged_snaps": [], > >>>>> "history": { > >>>>> "epoch_created": 478130, > >>>>> "epoch_pool_created": 478130, > >>>>> "last_epoch_started": 611605, > >>>>> "last_interval_started": 611604, > >>>>> "last_epoch_clean": 595152, > >>>>> "last_interval_clean": 595151, > >>>>> "last_epoch_split": 0, > >>>>> "last_epoch_marked_full": 0, > >>>>> "same_up_since": 617661, > >>>>> "same_interval_since": 617661, > >>>>> "same_primary_since": 616519, > >>>>> "last_scrub": "597862'517979", > >>>>> "last_scrub_stamp": "2022-03-15 05:08:42.561720", > >>>>> "last_deep_scrub": "597454'511650", > >>>>> "last_deep_scrub_stamp": "2022-03-09 18:05:34.205121", > >>>>> "last_clean_scrub_stamp": "2022-03-15 05:08:42.561720" > >>>>> }, > >>>>> "stats": { > >>>>> "version": "606021'521273", > >>>>> "reported_seq": "1733225", > >>>>> "reported_epoch": "617667", > >>>>> "state": "down+remapped", > >>>>> "last_fresh": "2022-03-28 14:17:48.920542", > >>>>> "last_change": "2022-03-28 14:15:42.025193", > >>>>> "last_active": "2022-03-25 15:24:42.557667", > >>>>> "last_peered": "2022-03-25 15:24:25.912029", > >>>>> "last_clean": "2022-03-15 01:50:02.979366", > >>>>> "last_became_active": "2022-03-25 12:44:52.379711", > >>>>> "last_became_peered": "2022-03-25 12:44:52.379711", > >>>>> "last_unstale": "2022-03-28 14:17:48.920542", > >>>>> "last_undegraded": "2022-03-28 14:17:48.920542", > >>>>> "last_fullsized": "2022-03-28 14:17:48.920542", > >>>>> "mapping_epoch": 617661, > >>>>> "log_start": "605873'518175", > >>>>> "ondisk_log_start": "605873'518175", > >>>>> "created": 478130, > >>>>> "last_epoch_clean": 595152, > >>>>> "parent": "0.0", > >>>>> "parent_split_bits": 0, > >>>>> "last_scrub": "597862'517979", > >>>>> "last_scrub_stamp": "2022-03-15 05:08:42.561720", > >>>>> "last_deep_scrub": "597454'511650", > >>>>> "last_deep_scrub_stamp": "2022-03-09 18:05:34.205121", > >>>>> "last_clean_scrub_stamp": "2022-03-15 05:08:42.561720", > >>>>> "log_size": 3098, > >>>>> "ondisk_log_size": 3098, > >>>>> "stats_invalid": false, > >>>>> "dirty_stats_invalid": false, > >>>>> "omap_stats_invalid": false, > >>>>> "hitset_stats_invalid": false, > >>>>> "hitset_bytes_stats_invalid": false, > >>>>> "pin_stats_invalid": false, > >>>>> "manifest_stats_invalid": false, > >>>>> "snaptrimq_len": 0, > >>>>> "stat_sum": { > >>>>> "num_bytes": 165108592640, > >>>>> "num_objects": 39402, > >>>>> "num_object_clones": 0, > >>>>> "num_object_copies": 197010, > >>>>> "num_objects_missing_on_primary": 0, > >>>>> "num_objects_missing": 0, > >>>>> "num_objects_degraded": 0, > >>>>> "num_objects_misplaced": 0, > >>>>> "num_objects_unfound": 0, > >>>>> "num_objects_dirty": 39402, > >>>>> "num_whiteouts": 0, > >>>>> "num_read": 338705, > >>>>> "num_read_kb": 124297874, > >>>>> "num_write": 316971, > >>>>> "num_write_kb": 311968402, > >>>>> "num_scrub_errors": 0, > >>>>> "num_shallow_scrub_errors": 0, > >>>>> "num_deep_scrub_errors": 0, > >>>>> "num_objects_recovered": 196778, > >>>>> "num_bytes_recovered": 824577241088, > >>>>> "num_keys_recovered": 0, > >>>>> "num_objects_omap": 0, > >>>>> "num_objects_hit_set_archive": 0, > >>>>> "num_bytes_hit_set_archive": 0, > >>>>> "num_flush": 0, > >>>>> "num_flush_kb": 0, > >>>>> "num_evict": 0, > >>>>> "num_evict_kb": 0, > >>>>> "num_promote": 0, > >>>>> "num_flush_mode_high": 0, > >>>>> "num_flush_mode_low": 0, > >>>>> "num_evict_mode_some": 0, > >>>>> "num_evict_mode_full": 0, > >>>>> "num_objects_pinned": 0, > >>>>> "num_legacy_snapsets": 0, > >>>>> "num_large_omap_objects": 0, > >>>>> "num_objects_manifest": 0, > >>>>> "num_omap_bytes": 0, > >>>>> "num_omap_keys": 0, > >>>>> "num_objects_repaired": 0 > >>>>> }, > >>>>> "up": [ > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 2147483647 > >>>>> ], > >>>>> "acting": [ > >>>>> 2147483647, > >>>>> 2147483647, > >>>>> 96, > >>>>> 2147483647, > >>>>> 2147483647 > >>>>> ], > >>>>> "avail_no_missing": [], > >>>>> "object_location_counts": [], > >>>>> "blocked_by": [ > >>>>> 121 > >>>>> ], > >>>>> "up_primary": -1, > >>>>> "acting_primary": 96, > >>>>> "purged_snaps": [] > >>>>> }, > >>>>> "empty": 0, > >>>>> "dne": 0, > >>>>> "incomplete": 0, > >>>>> "last_epoch_started": 616455, > >>>>> "hit_set_history": { > >>>>> "current_last_update": "0'0", > >>>>> "history": [] > >>>>> } > >>>>> }, > >>>>> "peer_info": [], > >>>>> "recovery_state": [ > >>>>> { > >>>>> "name": "Started/Primary/Peering/Down", > >>>>> "enter_time": "2022-03-28 14:15:42.025185", > >>>>> "comment": "not enough up instances of this PG to go > >>>>> active" > >>>>> }, > >>>>> { > >>>>> "name": "Started/Primary/Peering", > >>>>> "enter_time": "2022-03-28 14:15:42.025128", > >>>>> "past_intervals": [ > >>>>> { > >>>>> "first": "595151", > >>>>> "last": "617660", > >>>>> "all_participants": [ > >>>>> { > >>>>> "osd": 77, > >>>>> "shard": 0 > >>>>> }, > >>>>> { > >>>>> "osd": 82, > >>>>> "shard": 2 > >>>>> }, > >>>>> { > >>>>> "osd": 90, > >>>>> "shard": 0 > >>>>> }, > >>>>> { > >>>>> "osd": 91, > >>>>> "shard": 4 > >>>>> }, > >>>>> { > >>>>> "osd": 96, > >>>>> "shard": 2 > >>>>> }, > >>>>> { > >>>>> "osd": 121, > >>>>> "shard": 3 > >>>>> }, > >>>>> { > >>>>> "osd": 140, > >>>>> "shard": 1 > >>>>> }, > >>>>> { > >>>>> "osd": 159, > >>>>> "shard": 4 > >>>>> } > >>>>> ], > >>>>> "intervals": [ > >>>>> { > >>>>> "first": "610237", > >>>>> "last": "611219", > >>>>> "acting": "96(2),121(3),159(4)" > >>>>> }, > >>>>> { > >>>>> "first": "611604", > >>>>> "last": "616452", > >>>>> "acting": "121(3),140(1),159(4)" > >>>>> }, > >>>>> { > >>>>> "first": "616461", > >>>>> "last": "616518", > >>>>> "acting": "96(2),140(1),159(4)" > >>>>> } > >>>>> ] > >>>>> } > >>>>> ], > >>>>> "probing_osds": [ > >>>>> "77(0)", > >>>>> "82(2)", > >>>>> "90(0)", > >>>>> "91(4)", > >>>>> "96(2)", > >>>>> "140(1)", > >>>>> "159(4)" > >>>>> ], > >>>>> "blocked": "peering is blocked due to down osds", > >>>>> "down_osds_we_would_probe": [ > >>>>> 121 > >>>>> ], > >>>>> "peering_blocked_by": [ > >>>>> { > >>>>> "osd": 121, > >>>>> "current_lost_at": 0, > >>>>> "comment": "starting or marking this osd lost may > >>>>> let us proceed" > >>>>> } > >>>>> ] > >>>>> }, > >>>>> { > >>>>> "name": "Started", > >>>>> "enter_time": "2022-03-28 14:15:42.025068" > >>>>> } > >>>>> ], > >>>>> "agent_state": {} > >>>>> } > >>>>> [ > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Fulvio Galeazzi > >>>>> GARR-CSD Department > >>>>> skype: fgaleazzi70 > >>>>> tel.: +39-334-6533-250 > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users@ceph.io > >>>>> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > >> -- > >> Fulvio Galeazzi > >> GARR-CSD Department > >> skype: fgaleazzi70 > >> tel.: +39-334-6533-250 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io