>OPTION THREE: mounted as ldiskfs remove O/*/[1234567890]*[1234567890] >and then remount the file system. > >This would be one option. Note that with DNE the OST object names will be >in hex, so the above regexp would not catch all objects. >
We don't DNE, so our filenames were simple. I was still a little afraid to brute force it with a full removal, so instead I started by manually removing the two files that the kernel was generating syslog messages about, which then lead to the kernel complaining about two more. Using the newly discovered ll_decode_filter_fid command I had made a list of all the FIDs for all of the files visible in ldiskfs. I knew that there were roughly 8 pairs of files that had the same FID, so I removed all of those. Once I did that lustre was able to start cleaning the filesystem on its own. It took about three hours, and when it was done there were 3 destroys_in_progress reported on the MDS, and the kernel was still starting an OI trigger scrub for one FID. When I looked in the ldiskfs I found 20 files remaining. I removed those, and the kernel messages went away. Somewhere along this journey the file system became marked dirty at the ext4 level. An fsck was quick and did almost nothing. I've double checked all of the OSTs and nothing is producing kernel errors about lustre anywhere. So I feel like our lustre is hale and healthy now. We haven't set the write count back up yet, since it is a Friday, but we will next week. Thank you for the advice, it was the push we needed to be more aggressive in trying to fix this. I had spent a lot of time doing very minor things because I was afraid of breaking everything and losing two petabytes. In the end we lost probably hundreds of files, but they can all be reproduced, so there is no great hardship here other than making a list. --Schlake Sysadmin IV, NRAO Work: 575-835-7281 (BACK IN THE OFFICE!) Cell: 575-517-5668 (out of work hours) _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org