[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Hi,
have you tried restarting the primary OSD (currently 343)? It looks
like this PG is part of an EC pool, are there enough hosts available,
assuming your failure-domain is host? I assume that ceph isn't able to
recreate the shard on a different OSD. You could share your osd tree
and the crush rule as well as the erasure profile so we could get a
better picture.
Thanks,
Eugen
Zitat von siddhit.ren...@nxtgen.com:
Hello All,
Ceph version: 14.2.5-382-g8881d33957
(8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)
Issue:
1 PG stucked in "active+undersized+degraded for long time
Degraded data redundancy: 44800/8717052637 objects degraded
(0.001%), 1 pg degraded, 1 pg undersized
#ceph pg dump_stuck
PG_STAT STATE UP
UP_PRIMARY ACTING
ACTING_PRIMARY
15.28f0 active+undersized+degraded
[2147483647,343,355,415,426,640,302,392,78,202,607]343
[2147483647,343,355,415,426,640,302,392,78,202,607]343
PG Query:
#ceph pg 15.28f0 query
{
"state": "active+undersized+degraded",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 303362,
"up": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting_recovery_backfill": [
"78(8)",
"202(9)",
"302(6)",
"343(1)",
"355(2)",
"392(7)",
"415(3)",
"426(4)",
"607(10)",
"640(5)"
],
"info": {
"pgid": "15.28f0s1",
"last_update": "303161'598853",
"last_complete": "303161'598853",
"log_tail": "261289'595825",
"last_user_version": 598853,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19841,
"epoch_pool_created": 16141,
"last_epoch_started": 303017,
"last_interval_started": 303016,
"last_epoch_clean": 250583,
"last_interval_clean": 250582,
"last_epoch_split": 19841,
"last_epoch_marked_full": 0,
"same_up_since": 303016,
"same_interval_since": 303016,
"same_primary_since": 256311,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439"
},
"stats": {
"version": "303161'598853",
"reported_seq": "3594551",
"reported_epoch": "303362",
"state": "active+undersized+degraded",
"last_fresh": "2023-06-20 19:03:59.135295",
"last_change": "2023-06-20 15:11:12.569114",
"last_active": "2023-06-20 19:03:59.135295",
"last_peered": "2023-06-20 19:03:59.135295",
"last_clean": "2021-04-11 15:21:44.271834",
"last_became_active": "2023-06-20 15:11:12.569114",
"last_became_peered": "2023-06-20 15:11:12.569114",
"last_unstale": "2023-06-20 19:03:59.135295",
"last_undegraded": "2023-06-20 15:11:10.430426",
"last_fullsized": "2023-06-20 15:11:10.430154",
"mapping_epoch": 303016,
"log_start": "261289'595825",
"ondisk_log_start": "261289'595825",
"created": 19841,
"last_epoch_clean": 250583,
"parent": "0.0",
"parent_split_bits": 14,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439",
"log_size": 3028,
"ondisk_log_size": 3028,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 54989065178,
"num_objects": 44800,
"num_object_clones": 0,
"num_object_copies": 492800,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 44800,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Hi Siddhit
You need more OSD's. Please read:
https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/#erasure-coded-pgs-are-not-active-clean
Greetings
Damian
On 2023-06-20 15:53, siddhit.ren...@nxtgen.com wrote:
Hello All,
Ceph version: 14.2.5-382-g8881d33957
(8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)
Issue:
1 PG stucked in "active+undersized+degraded for long time
Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1
pg degraded, 1 pg undersized
#ceph pg dump_stuck
PG_STAT STATE UP
UP_PRIMARY ACTING
ACTING_PRIMARY
15.28f0 active+undersized+degraded
[2147483647,343,355,415,426,640,302,392,78,202,607]343
[2147483647,343,355,415,426,640,302,392,78,202,607]343
PG Query:
#ceph pg 15.28f0 query
{
"state": "active+undersized+degraded",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 303362,
"up": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting_recovery_backfill": [
"78(8)",
"202(9)",
"302(6)",
"343(1)",
"355(2)",
"392(7)",
"415(3)",
"426(4)",
"607(10)",
"640(5)"
],
"info": {
"pgid": "15.28f0s1",
"last_update": "303161'598853",
"last_complete": "303161'598853",
"log_tail": "261289'595825",
"last_user_version": 598853,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19841,
"epoch_pool_created": 16141,
"last_epoch_started": 303017,
"last_interval_started": 303016,
"last_epoch_clean": 250583,
"last_interval_clean": 250582,
"last_epoch_split": 19841,
"last_epoch_marked_full": 0,
"same_up_since": 303016,
"same_interval_since": 303016,
"same_primary_since": 256311,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439"
},
"stats": {
"version": "303161'598853",
"reported_seq": "3594551",
"reported_epoch": "303362",
"state": "active+undersized+degraded",
"last_fresh": "2023-06-20 19:03:59.135295",
"last_change": "2023-06-20 15:11:12.569114",
"last_active": "2023-06-20 19:03:59.135295",
"last_peered": "2023-06-20 19:03:59.135295",
"last_clean": "2021-04-11 15:21:44.271834",
"last_became_active": "2023-06-20 15:11:12.569114",
"last_became_peered": "2023-06-20 15:11:12.569114",
"last_unstale": "2023-06-20 19:03:59.135295",
"last_undegraded": "2023-06-20 15:11:10.430426",
"last_fullsized": "2023-06-20 15:11:10.430154",
"mapping_epoch": 303016,
"log_start": "261289'595825",
"ondisk_log_start": "261289'595825",
"created": 19841,
"last_epoch_clean": 250583,
"parent": "0.0",
"parent_split_bits": 14,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439",
"log_size": 3028,
"ondisk_log_size": 3028,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 54989065178,
"num_objects": 44800,
"num_object_clones": 0,
"num_object_copies": 492800,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 44800,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 44800,
"num_whiteouts": 0,
"num_read": 201078,
"num_read_kb": 30408632,
"num_write": 219335,
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Hello Eugen,
Requested details are as below.
PG ID: 15.28f0
Pool ID: 15
Pool: default.rgw.buckets.data
Pool EC Ratio: 8: 3
Number of Hosts: 12
## crush dump for rule ##
#ceph osd crush rule dump data_ec_rule
{
"rule_id": 1,
"rule_name": "data_ec_rule",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 11,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -50,
"item_name": "root_data~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
## From Crushmap dump ##
rule data_ec_rule {
id 1
type erasure
min_size 3
max_size 11
step set_chooseleaf_tries 5
step set_choose_tries 100
step take root_data class hdd
step chooseleaf indep 0 type host
step emit
}
## EC Profile ##
ceph osd erasure-code-profile get data
crush-device-class=hdd
crush-failure-domain=host
crush-root=root_data
jerasure-per-chunk-alignment=false
k=8
m=3
plugin=jerasure
technique=reed_sol_van
w=8
OSD Tree:
https://pastebin.com/raw/q6u7aSeu
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
What should be appropriate way to restart primary OSD in this case (343) ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Assuming you're running systemctl OSDs you can run the following command on the host that OSD 343 resides on. systemctl restart ceph-osd@343 From: siddhit.ren...@nxtgen.com At: 07/20/23 13:44:36 UTC-4:00To: ceph-users@ceph.io Subject: [ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time What should be appropriate way to restart primary OSD in this case (343) ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Sometimes one can even get away with "ceph osd down 343" which doesn't affect the process. I have had occasions when this goosed peering in a less-intrusive way. I believe it just marks the OSD down in the mons' map, and when that makes it to the OSD, the OSD responds with "I'm not dead yet" and gets marked up again. > On Jul 20, 2023, at 13:50, Matthew Leonard (BLOOMBERG/ 120 PARK) > wrote: > > Assuming you're running systemctl OSDs you can run the following command on > the host that OSD 343 resides on. > > systemctl restart ceph-osd@343 > > From: siddhit.ren...@nxtgen.com At: 07/20/23 13:44:36 UTC-4:00To: > ceph-users@ceph.io > Subject: [ceph-users] Re: 1 PG stucked in "active+undersized+degraded for > long time > > What should be appropriate way to restart primary OSD in this case (343) ? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
I can provide some more details, these were the recovery steps taken so far, they started from here (I don't know the whole/exact story though): 70/868386704 objects unfound (0.000%) Reduced data availability: 8 pgs inactive, 8 pgs incomplete Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 45558/8766139136 objects degraded (0.001%), 2 pgs degraded, 1 pg undersized And with reducing min_size for the EC pools some of the inactive PGs were cleaned up. From the remaining 4 incomplete PGs they got further by marking them unfound_lost: # ceph pg 15.f4f mark_unfound_lost delete pg has 70 objects unfound and apparently lost marking And now one PG is stuck degraded: # ceph pg ls degraded PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 15.28f0 4499444994 0 0 55288092914 0 0 3077 active+undersized+degraded 93s 310625'599302 310657:3603406 [2147483647,343,355,415,426,640,302,392,78,202,607]p343 [2147483647,343,355,415,426,640,302,392,78,202,607]p343 2021-04-11 03:18:39.164439 2021-04-10 01:42:16.182528 Setting osd.343 down didn't have any effect, I then suggested to increase set_choose_retries from 100 to 150 for the respective crush_rule (found a thread where that seemed to have helped), don't have a response to that yet. If nothing else helps, would it help marking the PG as unfound_lost (with data loss) help here? Zitat von Anthony D'Atri : Sometimes one can even get away with "ceph osd down 343" which doesn't affect the process. I have had occasions when this goosed peering in a less-intrusive way. I believe it just marks the OSD down in the mons' map, and when that makes it to the OSD, the OSD responds with "I'm not dead yet" and gets marked up again. On Jul 20, 2023, at 13:50, Matthew Leonard (BLOOMBERG/ 120 PARK) wrote: Assuming you're running systemctl OSDs you can run the following command on the host that OSD 343 resides on. systemctl restart ceph-osd@343 From: siddhit.ren...@nxtgen.com At: 07/20/23 13:44:36 UTC-4:00To: ceph-users@ceph.io Subject: [ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time What should be appropriate way to restart primary OSD in this case (343) ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
