[ceph-users] Re: owner locked out of bucket via bucket policy
Hello Casey, Our Production buckets are impacted due to this issue. We have downgraded Ceph version from 17.2.7 to 17.2.6 but still we are getting "bucket policy parsing" error while accessing the buckets. rgw_policy_reject_invalid_principals is not present in 17.2.6 as configurable parameter. Would appreciate response from your end. Nov 8 16:39:03 [1485064]: req 4696096351995892977 0.0s s3:get_obj Error reading IAM Policy: Terminate parsing due to Handler error. Nov 8 16:39:03 [1485064]: req 8949648957608194335 0.0s s3:get_obj Error reading IAM Policy: Terminate parsing due to Handler error. Nov 8 16:39:03 1485064]: req 3856551010860810445 0.00348s s3:get_obj Error reading IAM Policy: Terminate parsing due to Handler error. Nov 8 16:39:03 [1485064]: req 18116384331500039920 0.0s s3:get_obj Error reading IAM Policy: Terminate parsing due to Handler error. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Unable to fix 1 Inconsistent PG
Hello Wes, Thank you for your response. brc1admin:~ # rados list-inconsistent-obj 15.f4f No scrub information available for pg 15.f4f brc1admin:~ # ceph osd ok-to-stop osd.238 OSD(s) 238 are ok to stop without reducing availability or risking data, provided there are no other concurrent failures or interventions. 341 PGs are likely to be degraded (but remain available) as a result. Before I proceed with your suggested action plan, needed clarification on below. In order to list all objects residing on the inconsistent PG, we had stopped the primary osd (osd.238) and extracted the list of all objects residing on this osd using ceph-objectstore tool. We notice that that when we stop the osd (osd.238) using systemctl, RGW gateways continuously restarts which is impacting our S3 service availability. This was observed twice when we stopped osd.238 for general maintenance activity w.r.t ceph-objectstore tool. How can we ensure that stopping and marking out osd.238 ( primary osd of inconsistent pg) does not impact RGW service availability ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
What should be appropriate way to restart primary OSD in this case (343) ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time
Hello Eugen,
Requested details are as below.
PG ID: 15.28f0
Pool ID: 15
Pool: default.rgw.buckets.data
Pool EC Ratio: 8: 3
Number of Hosts: 12
## crush dump for rule ##
#ceph osd crush rule dump data_ec_rule
{
"rule_id": 1,
"rule_name": "data_ec_rule",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 11,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -50,
"item_name": "root_data~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
## From Crushmap dump ##
rule data_ec_rule {
id 1
type erasure
min_size 3
max_size 11
step set_chooseleaf_tries 5
step set_choose_tries 100
step take root_data class hdd
step chooseleaf indep 0 type host
step emit
}
## EC Profile ##
ceph osd erasure-code-profile get data
crush-device-class=hdd
crush-failure-domain=host
crush-root=root_data
jerasure-per-chunk-alignment=false
k=8
m=3
plugin=jerasure
technique=reed_sol_van
w=8
OSD Tree:
https://pastebin.com/raw/q6u7aSeu
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 1 PG stucked in "active+undersized+degraded for long time
Hello All,
Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8)
nautilus (stable)
Issue:
1 PG stucked in "active+undersized+degraded for long time
Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg
degraded, 1 pg undersized
#ceph pg dump_stuck
PG_STAT STATE UP
UP_PRIMARY ACTING
ACTING_PRIMARY
15.28f0 active+undersized+degraded
[2147483647,343,355,415,426,640,302,392,78,202,607]343
[2147483647,343,355,415,426,640,302,392,78,202,607]343
PG Query:
#ceph pg 15.28f0 query
{
"state": "active+undersized+degraded",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 303362,
"up": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting_recovery_backfill": [
"78(8)",
"202(9)",
"302(6)",
"343(1)",
"355(2)",
"392(7)",
"415(3)",
"426(4)",
"607(10)",
"640(5)"
],
"info": {
"pgid": "15.28f0s1",
"last_update": "303161'598853",
"last_complete": "303161'598853",
"log_tail": "261289'595825",
"last_user_version": 598853,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19841,
"epoch_pool_created": 16141,
"last_epoch_started": 303017,
"last_interval_started": 303016,
"last_epoch_clean": 250583,
"last_interval_clean": 250582,
"last_epoch_split": 19841,
"last_epoch_marked_full": 0,
"same_up_since": 303016,
"same_interval_since": 303016,
"same_primary_since": 256311,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439"
},
"stats": {
"version": "303161'598853",
"reported_seq": "3594551",
"reported_epoch": "303362",
"state": "active+undersized+degraded",
"last_fresh": "2023-06-20 19:03:59.135295",
"last_change": "2023-06-20 15:11:12.569114",
"last_active": "2023-06-20 19:03:59.135295",
"last_peered": "2023-06-20 19:03:59.135295",
"last_clean": "2021-04-11 15:21:44.271834",
"last_became_active": "2023-06-20 15:11:12.569114",
"last_became_peered": "2023-06-20 15:11:12.569114",
"last_unstale": "2023-06-20 19:03:59.135295",
"last_undegraded": "2023-06-20 15:11:10.430426",
"last_fullsized": "2023-06-20 15:11:10.430154",
"mapping_epoch": 303016,
"log_start": "261289'595825",
"ondisk_log_start": "261289'595825",
"created": 19841,
"last_epoch_clean": 250583,
"parent": "0.0",
"parent_split_bits": 14,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439",
"log_size": 3028,
"ondisk_log_size": 3028,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 54989065178,
"num_objects": 44800,
"num_object_clones": 0,
"num_object_copies": 492800,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 44800,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 44800,
"num_whiteouts": 0,
"num_read": 201078,
"num_read_kb": 30408632,
"num_write": 219335,
"num_write_kb": 59459782,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 121970,
