[ceph-users] Recover pgs from failed osds

2020-08-27 Thread Vahideh Alinouri
vahideh.alino...@gmail.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


[ceph-users] Recover pgs from failed osds

2020-08-27 Thread Vahideh Alinouri
Ceph cluster is updated from nautilus to octopus. On ceph-osd nodes we have
high I/O wait.

After increasing one of pool’s pg_num from 64 to 128 according to warning
message (more objects per pg), this lead to high cpu load and ram usage on
ceph-osd nodes and finally crashed the whole cluster. Three osds, one on
each host, stuck at down state (osd.34 osd.35 osd.40).

Starting the down osd service causes high ram usage and cpu load and
ceph-osd node to crash until the osd service fails.

The active mgr service on each mon host will crash after consuming almost
all available ram on the physical hosts.

I need to recover pgs and solving corruption. How can i recover unknown and
down pgs? Is there any way to starting up failed osd?


Below steps are done:

1- osd nodes’ kernel was upgraded to 5.4.2 before ceph cluster upgrading.
Reverting to previous kernel 4.2.1 is tested for iowate decreasing, but it
had no effect.

2- Recovering 11 pgs on failed osds by export them using
ceph-objectstore-tools utility and import them on other osds. The result
followed: 9 pgs are “down” and 2 pgs are “unknown”.

2-1) 9 pgs export and import successfully but status is “down” because of
"peering_blocked_by" 3 failed osds. I cannot lost osds because of
preventing unknown pgs from getting lost. pgs size in K and M.

"peering_blocked_by": [

{

"osd": 34,

"current_lost_at": 0,

"comment": "starting or marking this osd lost may let us proceed"

},

{

"osd": 35,

"current_lost_at": 0,

"comment": "starting or marking this osd lost may let us proceed"

},

{

"osd": 40,

"current_lost_at": 0,

"comment": "starting or marking this osd lost may let us proceed"

}

]


2-2) 1 pg (2.39) export and import successfully, but after starting osd
service (pg import to it), ceph-osd node RAM and CPU consumption increase
and cause ceph-osd node to crash until the osd service fails. Other osds
become "down" on ceph-osd node. pg status is “unknown”. I cannot use
"force-create-pg" because of data lost. pg 2.39 size is 19G.

# ceph pg map 2.39

osdmap e40347 pg 2.39 (2.39) -> up [32,37] acting [32,37]

# ceph pg 2.39 query

Error ENOENT: i don't have pgid 2.39


*pg 2.39 info on failed osd:

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/*ceph-34* --op info
--pgid 2.39

{

"pgid": "2.39",

"last_update": "35344'6456084",

"last_complete": "35344'6456084",

"log_tail": "35344'6453182",

"last_user_version": 10595821,

"last_backfill": "MAX",

"purged_snaps": [],

"history": {

"epoch_created": 146,

"epoch_pool_created": 79,

"last_epoch_started": 25208,

"last_interval_started": 25207,

"last_epoch_clean": 25208,

"last_interval_clean": 25207,

"last_epoch_split": 370,

"last_epoch_marked_full": 0,

"same_up_since": 8347,

"same_interval_since": 25207,

"same_primary_since": 8321,

"last_scrub": "35328'6440139",

"last_scrub_stamp": "2020-08-19T12:00:59.377593+0430",

"last_deep_scrub": "35261'6031075",

"last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430",

"last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430",

"prior_readable_until_ub": 0

},

"stats": {

"version": "35344'6456082",

"reported_seq": "11733156",

"reported_epoch": "35344",

"state": "active+clean",

"last_fresh": "2020-08-19T14:16:18.587435+0430",

"last_change": "2020-08-19T12:00:59.377747+0430",

"last_active": "2020-08-19T14:16:18.587435+0430",

"last_peered": "2020-08-19T14:16:18.587435+0430",

"last_clean": "2020-08-19T14:16:18.587435+0430",

"last_became_active": "2020-08-06T00:23:51.016769+0430",

"last_became_peered": "2020-08-06T00:23:51.016769+0430",

"last_unstale": "2020-08-19T14:16:18.587435+0430",

"last_undegraded": "2020-08-19T14:16:18.587435+0430",

"last_fullsized": "2020-08-19T14:16:18.587435+0430",

"mapping_epoch": 8347,

"log_start": "35344'6453182",

"ondisk_log_start": "35344'6453182",

"created": 146,

"last_epoch_clean": 25208,

"parent": "0.0",

"parent_split_bits": 7,

"last_scrub": "35328'6440139",

"last_scrub_stamp": "2020-08-19T12:00:59.377593+0430",

"last_deep_scrub": "35261'6031075",

"last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430",

"last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430",

"log_size": 2900,

"ondisk_log_size": 2900,

"stats_invalid": false,

"dirty_stats_invalid": false,

"omap_stats_invalid": false,

"hitset_stats_invalid": false,

"hitset_bytes_stats_invalid": false,

"pin_stats_invalid": false,

"manifest_stats_invalid": false,

"snaptrimq_len": 0,

"stat_sum": {

"num_bytes": 19749578960,

"num_objects": 2442,

"num_object_clones": 20,

"num_object_copies": 7326,

"num_objects_missing_on_primary": 0,

"num_objects_missing": 0,

"num_objects_degraded": 0,

"num_objects_misplaced": 0,

"num_objects_unfound": 0,

"num_objects_dirty": 2442,

"num_whiteouts": 0,

"num_read": 16120686,

"num_read_kb": 82264126,

"num_write": 19731882,

"num_write_kb": 379030181,

"num_scrub_errors": 0,

"num_shallow_scrub_errors": 0,

"num_deep_scrub_errors": 0,

"num_objects_recovered": 2861,

"num_bytes_recov