Re: [lustre-discuss] Help with recovery of data

2022-06-28 Thread Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss
An update. We were able to recover our filesystem (minus the two days between when the ZFS swap occurred and when we detected it and shut down the filesystem). Simply promoting the cloned ZFS volume (which was really our primary volume) and cleaning up the snapshot and clone got us back to

Re: [lustre-discuss] Help with recovery of data

2022-06-22 Thread Andreas Dilger via lustre-discuss
First thing, if you haven't already done so, would be to make a separate "dd" backup of the ldiskfs MDT(s) to some external storage before you do anything else. That will give you a fallback in case whatever changes you make don't work out well. I would also suggest to contact the ZFS mailing

Re: [lustre-discuss] Help with recovery of data

2022-06-22 Thread Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss
A quick follow up. I thought an lfsck would only clean up (i.e. remove orphaned MDT and OST objects) but it appears this might have a good shot at repairing the file system – specifically, recreating the MDT objects with the --create-mdtobj option. We have started this command:

[lustre-discuss] Help with recovery of data

2022-06-21 Thread Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss
Hi everyone, We ran into a problem with our lustre filesystem this weekend and could use a sanity check and/or advice on recovery. We are running on CentOS 7.9, ZFS 2.1.4 and Lustre 2.14. We are using ZFS OST’s but and an ldiskfs MDT (for better MDT performance). For various reasons, the