Hi Kurt, I was following the recommendation that the OST be active to allow the deletion to happen, hence the reactivation followed by mount/unmount is necessary to trigger the cleanup. The OST was therefore active during the mount/unmount.
Best, Sean ________________________________________ From: Kurt Strosahl [stros...@jlab.org] Sent: 12 July 2015 02:03 To: Sean Brisbane Cc: Shawn Hall; lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining Thanks, I'll have to see if I can run this test myself. Did you notice if the "inactive" status persisted through the unmount/remount? w/r, Kurt ----- Original Message ----- From: "Sean Brisbane" <sean.brisb...@physics.ox.ac.uk> To: "Kurt Strosahl" <stros...@jlab.org>, "Shawn Hall" <shawn.h...@nag.com> Cc: lustre-discuss@lists.lustre.org Sent: Saturday, July 11, 2015 4:29:42 AM Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining Dear Kurt, I have the same issue as you in that deleted files on deactivated OST could not be cleaned up even after re-activation. It was on my todo list to work out at some point how to get around this. I was told that an unmount/mount cycle on the servers will trigger a clean-up. I have just performed the experiment and it was in fact the MDT not the OST which needed to be unmounted and re-mounted in my case. Unmounting and remounting the OST during this process appeared to make no difference either way. All the best, Sean ________________________________________ From: Kurt Strosahl [stros...@jlab.org] Sent: 10 July 2015 19:53 To: Shawn Hall Cc: Sean Brisbane; lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining Yes, there are quite a few issues with lustre 2.5.3 (it would be sad if it wasn't so frustrating... 1.8.x was solid). The full osts have a higher index then the one that broke the weighted round robin... plus all the ones above the most recent are exceptionally full (>=80%). I'm not sure how I'm going to go forward, I've heard that maybe an unmount / mount of the osts would push a purge. I'm also compiling a list of all the files on the ost... the idea being that I could then enable it, and launch multiple lfs_migrates... trying to race everyone else using the file system. I think I'd have the advantage, as my moves would be targeted directly to the ost, while the other writes would just land where ever they could. w/r, Kurt ----- Original Message ----- From: "Shawn Hall" <shawn.h...@nag.com> To: "Kurt Strosahl" <stros...@jlab.org>, "Sean Brisbane" <sean.brisb...@physics.ox.ac.uk> Cc: lustre-discuss@lists.lustre.org Sent: Friday, July 10, 2015 11:49:06 AM Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining It sounds like you have a couple of issues that are working against each other then. You’ll probably need to fight one at a time. My recommendation of clearing up file system space still stands. I don’t have scientific proof, but giving Lustre more space to work with definitely helps. Does your full OST have a lower index than your slow OST? Then you could disable the slow one (and because of the bug everything above it) and let space clear up on the full one. Beyond that you might have to get creative and try something similar to Tommy. Migrate data but manually specify stripe offsets. Shawn On 7/10/15, 11:13 AM, "lustre-discuss on behalf of Kurt Strosahl" <lustre-discuss-boun...@lists.lustre.org on behalf of stros...@jlab.org> wrote: >No, I'm aware of why the ost is getting new writes... it is because I had to >set the qos_threshold_rr to 100 due to >https://jira.hpdd.intel.com/browse/LU-5778 (I have an ost that has to be >ignored due to terrible write performance...) > >w/r, >Kurt > >----- Original Message ----- >From: "Sean Brisbane" <sean.brisb...@physics.ox.ac.uk> >To: "Kurt Strosahl" <stros...@jlab.org> >Cc: "Patrick Farrell" <p...@cray.com>, "lustre-discuss@lists.lustre.org" ><lustre-discuss@lists.lustre.org> >Sent: Friday, July 10, 2015 11:04:27 AM >Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining > >Dear Kurt, > >Apologies. After leaving it some number of days it did *not* clean itself up, >but I feel that some number of days is long enough to verify that it is a >problem. > >Sounds like you have another issue if the OST is not being marked as full and >writes are not being re-allocated to other OSTS . I also have that second >issue on my system as well and I have only workarounds to offer you for the >problem. > >Thanks, >Sean > >-----Original Message----- >From: Kurt Strosahl [mailto:stros...@jlab.org] >Sent: 10 July 2015 16:01 >To: Sean Brisbane >Cc: Patrick Farrell; lustre-discuss@lists.lustre.org >Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining > >The problem there is that I cannot afford to leave it "some number of days"... >it is at 97% full, so new writes are going to it faster then it can clean >itself off. > >w/r, >Kurt > >----- Original Message ----- >From: "Sean Brisbane" <sean.brisb...@physics.ox.ac.uk> >To: "Patrick Farrell" <p...@cray.com>, "Kurt Strosahl" <stros...@jlab.org> >Cc: lustre-discuss@lists.lustre.org >Sent: Friday, July 10, 2015 10:44:39 AM >Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining > >Hi, > >The 'space not freed' issue also happened to me and I left it 'some number of >days' I don't recall how many, it was a while back. > >Cheers, >Sean >_______________________________________________ >lustre-discuss mailing list >lustre-discuss@lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
