Hi,
We had an issue a few months ago with the underlying zpool for one of our OSTs.
I managed to get it mounted in read only mode and migrated all of the files off
it with lfs migrate, then recreated the OST and reintroduced it. This all went
pretty smoothly - at the same time I updated our pro
Hi all,
Some further developments, which we don't understand.
As files on this OST get written and deleted, it seems that they are
removed from the MDS, but not actually deleted from the OST. The OST then
gradually fills up.
If we do a (on the mds):
lctl set_param osc.snap8-OST004e-*.active
Hi Cory,
Servers and clients are all 2.12.6, and were installed as such (i.e.
haven't been updated from an older version).
Cheers,
Alastair.
On Thu, 16 Sep 2021, Spitz, Cory James wrote:
[EXTERNAL EMAIL]
What versions do you have on your servers and clients? Do you have some wide
gap in
What versions do you have on your servers and clients? Do you have some wide
gap in versions? Is your sever very old?
There was a change to the object deletion protocol that you may need to contend
with. It was related to LU-5814. If you don't have an older server then this
is not your prob
Hi all,
We mounted as ext4, removed the files, and then remounted as lustre (and
did the lfsck scans).
All seemed fine, and the OST went back into production.
However, it again has the same problem - it is filling up. Currently
lfs df reports it as 89% full with 4.8TB used.
However, an lfs
On Sep 8, 2021, at 04:42, Alastair Basden
mailto:a.g.bas...@durham.ac.uk>> wrote:
Next step would be to unmount OST004e, run a full e2fsck, and then check
lost+found and/or a regular "find /mnt/ost -type f -size +1M" or similar to
find where the files are.
Thanks. e2fsck returns clean (on
Next step would be to unmount OST004e, run a full e2fsck, and then check lost+found
and/or a regular "find /mnt/ost -type f -size +1M" or similar to find where the
files are.
Thanks. e2fsck returns clean (on its own, with -p and with -f).
Now, the find command does return a large number
On Sep 3, 2021, at 01:49, Alastair Basden
mailto:a.g.bas...@durham.ac.uk>> wrote:
Hi,
We have a file system where each OST is a single SSD.
One of those is reporting as 100% full (lfs df -h /snap8):
snap8-OST004d_UUID 5.8T2.0T3.5T 37% /snap8[OST:77]
snap8-OST004e_UUID
Hi Aurélien,
Thanks.
Within O/1/d0 to O/1/d31, these are empty directories.
Within O/0/d0 to d31, these have some files in them. However, of the ones
I've tried, the
lfs fid2path /snap8 [0x1004e:0xe0:0x0]
returns e.g.
lfs fid2path: cannot find '[0x1004e:0xe0:0x0]': Invalid argument
w
Hi
>Not quite sure what you meant by the O/*/d* as there are no directories
> within O/, and there is no d/ or d*/ either at top level or within O/
As you can confirm with the 'stat' output you provided, '23c400' is a
directory and actually all other entries also are.
Not straightforwa
Hi Andreas,
Thanks.
With debugfs /dev/nvme6n1, I get:
debugfs: ls -l O
393217 40755 (2) 0 04096 28-Jul-2021 17:06 .
2 40755 (2) 0 04096 28-Jul-2021 17:02 ..
393218 40755 (2) 0 04096 28-Jul-2021 17:02 20003
524291 40755 (2) 0
You could run debugfs on that OST and use "ls -l" to examine the O/*/d*
directories for large objects, then "stat" any suspicious objects within
debugfs to dump the parent FID, and "lfs fid2path" on a client to determine the
path.
Alternately, see "lctl-lfsck-start.8" man page for options to l
Ah, of course - has to be done on a client.
None of these files are on the dodgy OST.
Any further suggestions? Essentially we have what seems to be a full OST
with nothing on it.
Thanks,
Alastair.
On Sat, 4 Sep 2021, Andreas Dilger wrote:
[EXTERNAL EMAIL]
$ man lfs-fid2path.1
lfs-fid2path
$ man lfs-fid2path.1
lfs-fid2path(1) user utilities
lfs-fid2path(1)
NAME
lfs fid2path - print the pathname(s) for a file identifier
SYNOPSIS
lfs fid2path [OPTION]... ...
DESCRIPTION
lfs fid2path ma
Hi,
lctl get_param mdt.*.exports.*.open_files returns:
mdt.snap8-MDT.exports.172.18.180.21@o2ib.open_files=
[0x2b90e:0x10aa:0x0]
mdt.snap8-MDT.exports.172.18.180.22@o2ib.open_files=
[0x2b90e:0x21b3:0x0]
mdt.snap8-MDT.exports.172.18.181.19@o2ib.open_files=
[0x2b90e:0x21b3:
You can also check "mdt.*.exports.*.open_files" on the MDTs for a list of FIDs
open on each client, and use "lfs fid2path" to resolve them to a pathname.
On Sep 3, 2021, at 02:09, Degremont, Aurelien via lustre-discuss
mailto:lustre-discuss@lists.lustre.org>> wrote:
Hi
It could be a bug, but m
Hi,
Thanks. It seems odd that the OST reports having no files on it at all
(I'd expect several hundred, based on the number of files on the other
OSTs).
Unless an open-unlinked file would have that effect on lfs find, but I
don't think it should.
I'm not sure whether lsof would help - the
Hi
It could be a bug, but most of the time, this is due to an open-unlinked file,
typically a log file which is still in use and some processes keep writing to
it until it fills the OSTs it is using.
Look for such files on your clients (use lsof).
Aurélien
Le 03/09/2021 09:50, « lustre-dis
Hi,
We have a file system where each OST is a single SSD.
One of those is reporting as 100% full (lfs df -h /snap8):
snap8-OST004d_UUID 5.8T2.0T3.5T 37% /snap8[OST:77]
snap8-OST004e_UUID 5.8T5.5T7.5G 100% /snap8[OST:78]
snap8-OST004f_UUID
In my brief attempts to use lfs migrate, I found performance pretty slow
(it's serial). I also got some ASSERTs which should be fixed in 2.10 per
LU-8807; note that I was using 2.8. On a more trivial level, I found the
-v|--verbose option to the command doesn't work.
On 01/07/2019 12:26 PM, Mo
> On Jan 7, 2019, at 2:09 PM, Jason Williams wrote:
>
> One last question, How safe is lfs_migrate? The man page on the installation
> says it's UNSAFE for possibly in-use files. The lustre manual doesn't have
> the same warning and says something about it being a bit more integrated with
liams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
> On Jan 7, 2019, at 12:53 PM, Jason Williams wrote:
>
> As I have gone through the testing, I think you may be right. I think I
> disabled the OST in a slightly d
> On Jan 7, 2019, at 12:53 PM, Jason Williams wrote:
>
> As I have gone through the testing, I think you may be right. I think I
> disabled the OST in a slightly different way and that caused issues.
>
> Do you happen to know where I could find out a bit more about what the "lctl
> set_para
M
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
Jason,
The results you described sound like the correct behavior when you deactivate
an OST on the MDS. When you run “lctl —device deactivate”, you are
nt: Monday, January 7, 2019 11:47:09 AM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without
> hanging?
>
> So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I
&
u<mailto:jas...@jhu.edu>
From: lustre-discuss on behalf of
Jason Williams
Sent: Monday, January 7, 2019 11:47:09 AM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without han
Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was
following the wrong instructions. Do you know where the best instructions for
wh
From: Mohr Jr, Richard Frank (Rick Mohr)
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
> On Jan 5, 2019, at 9:49 PM, Jason Williams wrote:
>
&g
> On Jan 5, 2019, at 9:49 PM, Jason Williams wrote:
>
> I have looked around the internet and found you can disable an OST, but when
> I have tried that, any writes (including deletes) to the OST hang the clients
> indefinitely. Does anyone know a way to make an OST basically "read-only"
>
Jason,
If there are files located on the full OST that you can delete then delete
them without trying to deactivate the OST. There is a process where you can
find files solely located on the full OST using lfs then deactivate the
full OST, copy the files to new files with some suffix (.new perhaps
Hello,
We have a lustre system (version 2.10.4) that has unfortunately fallen victim
to a 100% full OST... Every time we clear some space on it, the system fills it
right back up again.
I have looked around the internet and found you can disable an OST, but when I
have tried that, any writes
On 2010-01-29, at 03:07, Johann Lombardi wrote:
> On Fri, Jan 29, 2010 at 10:32:26AM +0100, gvozden rovina wrote:
>> OST. For instance i copied 2.5 GB file to lustre which had 120 GB
>> storage
>> space (I have 2GB test OSTs) and it didn't automatically recognized
>> full
>> OST but it simply s
thx for answer. good bye for now
On Fri, Jan 29, 2010 at 12:02 PM, Johann Lombardi wrote:
> On Fri, Jan 29, 2010 at 11:48:58AM +0100, gvozden rovina wrote:
> > Thank you for your swift answer. I have just one more question. Is it
> > possible to
> > configure lustre system so that it writes not
On Fri, Jan 29, 2010 at 11:48:58AM +0100, gvozden rovina wrote:
> Thank you for your swift answer. I have just one more question. Is it
> possible to
> configure lustre system so that it writes not just the file but also the
> copy of the same file in the same time as you create it?
No, we don't s
Hi!
Thank you for your swift answer. I have just one more question. Is it
possible to
configure lustre system so that it writes not just the file but also the
copy of the same file
in the same time as you create it?
thx!
On Fri, Jan 29, 2010 at 11:07 AM, Johann Lombardi wrote:
> On Fri, Jan 29
On Fri, Jan 29, 2010 at 10:32:26AM +0100, gvozden rovina wrote:
> OST. For instance i copied 2.5 GB file to lustre which had 120 GB storage
> space (I have 2GB test OSTs) and it didn't automatically recognized full
> OST but it simply stopped working with " No space left on device" error
> message.
Hello!
I'm new to lustre and I have just one question.
While reading literature and playing with my test
lustre system I've noticed that there is strange (for me) issue with full
OST. For instance i copied 2.5 GB file to lustre which had 120 GB storage
space (I have 2GB test OSTs) and it didn't au
37 matches
Mail list logo