[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-18 Thread Joshua West
Okay, I gave up, as I should have long ago. Unlinking was a massive pain in the rear, even when scripted, as I never found a good method to catch the hanging without hanging the script, or the filesystem. After several attempts, because I only have a couple of primary directories which were

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-14 Thread Joshua West
Additional to my last note, I should have mentioned, I am exploring options to delete the damaged data, but in hopes to preserve what I can, prior to moving to simply deleting all data on that pool. When trying to simply empty pgs, it seems like the pgs don't exist. In attempting to follow:

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-14 Thread Joshua West
Just working this through, how does one identify the OIDs within a PG, without list_unfound? I've been poking around, but can't seem to find a command that outputs the necessary OIDs. I tried a handful of cephfs commands, but they of course become stuck, and ceph pg commands haven't revealed the

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-09 Thread Michael Thomas
Hi Joshua, I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case. Correct me if I'm wrong,

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-08 Thread Szabo, Istvan (Agoda)
Hi, So finally how did you solve it? Which method out of the three? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message-

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

2021-04-08 Thread Michael Thomas
Hi Joshua, I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered. Here's how I recovered from the situation. Note