Good Morning,
Earlier in the week some of the OSTs in the lustre 2.5.3 system I'm managing
hit 97% full. I set them to read only, and yesterday I started an lfs_migrate
on one of them, hoping to drain it down overnight. This morning, after the
system spent about 12 hours moving files off o
Kurt,
Did you deactivate the OST? Deactivated OSTs do not show reduced usage until
activated again.
- Patrick
From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of
Kurt Strosahl [stros...@jlab.org]
Sent: Friday, July 10, 2015 7:27
I set it offline using lctl --device deactivate earlier in the
week. I then set it back online this morning using lctl --device
activate I let it sit in active mode for almost an hour, but then I noticed
that the used space had gone up significantly on the ost, so I set it offline
again usi
That sounds like a separate issue, where people are filling the OST quickly.
Like I said, freed space will not be visible until the OST is active
(and I think it might need to be active for a little while).
On 07/10/2015 08:40 AM, Kurt Strosahl wrote:
I set it offline using lctl --device dea
Hi,
The 'space not freed' issue also happened to me and I left it 'some number of
days' I don't recall how many, it was a while back.
Cheers,
Sean
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lus
The problem there is that I cannot afford to leave it "some number of days"...
it is at 97% full, so new writes are going to it faster then it can clean
itself off.
w/r,
Kurt
- Original Message -
From: "Sean Brisbane"
To: "Patrick Farrell" , "Kurt Strosahl"
Cc: lustre-discuss@lists.lu
Dear Kurt,
Apologies. After leaving it some number of days it did *not* clean itself up,
but I feel that some number of days is long enough to verify that it is a
problem.
Sounds like you have another issue if the OST is not being marked as full and
writes are not being re-allocated to other
I agree it does take some time to clear delete the objects - unfortunately I
don’t know if there’s a way to speed that up. Object allocation, even when
weighted, I believe is still probabilistic, so you can never guarantee that OST
won’t be written to without deactivating it. And people can app
No, I'm aware of why the ost is getting new writes... it is because I had to
set the qos_threshold_rr to 100 due to
https://jira.hpdd.intel.com/browse/LU-5778 (I have an ost that has to be
ignored due to terrible write performance...)
w/r,
Kurt
- Original Message -
From: "Sean Brisban
Unfortunately... due to https://jira.hpdd.intel.com/browse/LU-5778, I'm stuck
at just a flat round robin.
- Original Message -
From: "Shawn Hall"
To: "Kurt Strosahl" , "Sean Brisbane"
Cc: lustre-discuss@lists.lustre.org
Sent: Friday, July 10, 2015 11:08:57 AM
Subject: Re: [lustre-discu
Kurt,
That is fixed in the latest 2.5 source, though you'd have to build
yourself since Intel has stopped doing public maintenance releases. That
patch landed to 2.5 in between the last public maintenance release and
the decision to stop doing them.
- Patrick
On 07/10/2015 10:17 AM, Kurt Str
It sounds like you have a couple of issues that are working against each other
then. You’ll probably need to fight one at a time.
My recommendation of clearing up file system space still stands. I don’t have
scientific proof, but giving Lustre more space to work with definitely helps.
Does
Hi Kurt,
to keep traffic from almost full OST we usually set ost in degraded mode like
described in manual:
> Handling Degraded OST RAID Arrays
> To mark the OST as degraded, use:
> lctl set_param obdfilter.{OST_name}.degraded=1
Alex.
On Jul 10, 2015, at 10:13 AM, Kurt Strosahl wrote:
> No,
Will that let deletes happen against it?
w/r,
Kurt
- Original Message -
From: "aik"
To: "Kurt Strosahl"
Cc: "aik" , "Sean Brisbane" ,
lustre-discuss@lists.lustre.org
Sent: Friday, July 10, 2015 11:52:00 AM
Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
Hi Kurt,
to keep t
I think so, try it.
We do set ost degraded on 1.8 when ost nears 95% and we migrate data to another
ost.
On 1.8 lfs_migrate uses 'rm' and objects are indeed deallocated.
Alex
On Jul 10, 2015, at 10:55 AM, Kurt Strosahl wrote:
> Will that let deletes happen against it?
>
> w/r,
> Kurt
>
> --
Yes, there are quite a few issues with lustre 2.5.3 (it would be sad if it
wasn't so frustrating... 1.8.x was solid).
The full osts have a higher index then the one that broke the weighted round
robin... plus all the ones above the most recent are exceptionally full
(>=80%). I'm not sure how I
Note that the “lfs migrate” command (not the lfs_migrate script, which I
believe uses the lfs migrate command) has a --block option. When performing
the migrate, this will block all other I/O to that file. That should help you
win the race in case you’re competing for a file.
Shawn
On 7/10
Thanks,
But I don't see that command in the lfs documentaiton.
w/r,
Kurt
- Original Message -
From: "Shawn Hall"
To: "Kurt Strosahl"
Cc: "Sean Brisbane" ,
lustre-discuss@lists.lustre.org
Sent: Friday, July 10, 2015 3:27:31 PM
Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draini
Yeah doesn’t look like it made it into the documentation. Looks like it’s
still in the lfs code though. It got added from this:
https://jira.hpdd.intel.com/browse/LU-2445
The lfs_migrate script attempts to do use this before a rsync based approach.
It does not use the --block option though.
All,
I understand that doing recursive file operations can be taxing on lustre.
So, I wonder if there is a preferred performance-minded way to remove an entire
directory tree that is several TB in size.
The standard rm -rf ./dir seems to spike the cpu usage on my OSSes where it
sits and sometime
There's something in the rm command that makes recursive deletes really
expensive, although I don't know why. I've found in the past that even running
a regular find ... -exec rm {} \; has been quicker. Running lfs find to build
the file list would presumably be quicker still.
Malcolm.
From:
Hi,
Just like Malcolm suggested, before I implemented Robinhood, I built the
list and removed files (that were not accessed in more than 31 days). I
found it much faster than all other methods.
/usr/bin/lfs find /scratch/${folder} -A +31 -p -t f | xargs -0 -n 10
-P 8 /bin/rm
10 and 8 for fl
FWIW, last September in the context of reducing the memory usage on the
mgs, Andreas Dilger had this to say
"ls" is itself fairly inefficient at directory traversal, because all of
the GNU file utilities are bloated and do much more work than necessary
(e.g. "rm" will stat() every file before un
23 matches
Mail list logo
