Hi,
We had an issue a few months ago with the underlying zpool for one of our OSTs.
I managed to get it mounted in read only mode and migrated all of the files off
it with lfs migrate, then recreated the OST and reintroduced it. This all went
pretty smoothly - at the same time I updated our
Hi all,
Some further developments, which we don't understand.
As files on this OST get written and deleted, it seems that they are
removed from the MDS, but not actually deleted from the OST. The OST then
gradually fills up.
If we do a (on the mds):
lctl set_param
Hi Cory,
Servers and clients are all 2.12.6, and were installed as such (i.e.
haven't been updated from an older version).
Cheers,
Alastair.
On Thu, 16 Sep 2021, Spitz, Cory James wrote:
[EXTERNAL EMAIL]
What versions do you have on your servers and clients? Do you have some wide
gap in
What versions do you have on your servers and clients? Do you have some wide
gap in versions? Is your sever very old?
There was a change to the object deletion protocol that you may need to contend
with. It was related to LU-5814. If you don't have an older server then this
is not your
Hi all,
We mounted as ext4, removed the files, and then remounted as lustre (and
did the lfsck scans).
All seemed fine, and the OST went back into production.
However, it again has the same problem - it is filling up. Currently
lfs df reports it as 89% full with 4.8TB used.
However, an lfs
On Sep 8, 2021, at 04:42, Alastair Basden
mailto:a.g.bas...@durham.ac.uk>> wrote:
Next step would be to unmount OST004e, run a full e2fsck, and then check
lost+found and/or a regular "find /mnt/ost -type f -size +1M" or similar to
find where the files are.
Thanks. e2fsck returns clean
Next step would be to unmount OST004e, run a full e2fsck, and then check lost+found
and/or a regular "find /mnt/ost -type f -size +1M" or similar to find where the
files are.
Thanks. e2fsck returns clean (on its own, with -p and with -f).
Now, the find command does return a large number
On Sep 3, 2021, at 01:49, Alastair Basden
mailto:a.g.bas...@durham.ac.uk>> wrote:
Hi,
We have a file system where each OST is a single SSD.
One of those is reporting as 100% full (lfs df -h /snap8):
snap8-OST004d_UUID 5.8T2.0T3.5T 37% /snap8[OST:77]
snap8-OST004e_UUID
Hi Aurélien,
Thanks.
Within O/1/d0 to O/1/d31, these are empty directories.
Within O/0/d0 to d31, these have some files in them. However, of the ones
I've tried, the
lfs fid2path /snap8 [0x1004e:0xe0:0x0]
returns e.g.
lfs fid2path: cannot find '[0x1004e:0xe0:0x0]': Invalid argument
Hi
>Not quite sure what you meant by the O/*/d* as there are no directories
> within O/, and there is no d/ or d*/ either at top level or within O/
As you can confirm with the 'stat' output you provided, '23c400' is a
directory and actually all other entries also are.
Not
Hi Andreas,
Thanks.
With debugfs /dev/nvme6n1, I get:
debugfs: ls -l O
393217 40755 (2) 0 04096 28-Jul-2021 17:06 .
2 40755 (2) 0 04096 28-Jul-2021 17:02 ..
393218 40755 (2) 0 04096 28-Jul-2021 17:02 20003
524291 40755 (2) 0
You could run debugfs on that OST and use "ls -l" to examine the O/*/d*
directories for large objects, then "stat" any suspicious objects within
debugfs to dump the parent FID, and "lfs fid2path" on a client to determine the
path.
Alternately, see "lctl-lfsck-start.8" man page for options to
Ah, of course - has to be done on a client.
None of these files are on the dodgy OST.
Any further suggestions? Essentially we have what seems to be a full OST
with nothing on it.
Thanks,
Alastair.
On Sat, 4 Sep 2021, Andreas Dilger wrote:
[EXTERNAL EMAIL]
$ man lfs-fid2path.1
$ man lfs-fid2path.1
lfs-fid2path(1) user utilities
lfs-fid2path(1)
NAME
lfs fid2path - print the pathname(s) for a file identifier
SYNOPSIS
lfs fid2path [OPTION]... ...
DESCRIPTION
lfs fid2path
Hi,
lctl get_param mdt.*.exports.*.open_files returns:
mdt.snap8-MDT.exports.172.18.180.21@o2ib.open_files=
[0x2b90e:0x10aa:0x0]
mdt.snap8-MDT.exports.172.18.180.22@o2ib.open_files=
[0x2b90e:0x21b3:0x0]
mdt.snap8-MDT.exports.172.18.181.19@o2ib.open_files=
You can also check "mdt.*.exports.*.open_files" on the MDTs for a list of FIDs
open on each client, and use "lfs fid2path" to resolve them to a pathname.
On Sep 3, 2021, at 02:09, Degremont, Aurelien via lustre-discuss
mailto:lustre-discuss@lists.lustre.org>> wrote:
Hi
It could be a bug, but
Hi,
Thanks. It seems odd that the OST reports having no files on it at all
(I'd expect several hundred, based on the number of files on the other
OSTs).
Unless an open-unlinked file would have that effect on lfs find, but I
don't think it should.
I'm not sure whether lsof would help -
Hi,
We have a file system where each OST is a single SSD.
One of those is reporting as 100% full (lfs df -h /snap8):
snap8-OST004d_UUID 5.8T2.0T3.5T 37% /snap8[OST:77]
snap8-OST004e_UUID 5.8T5.5T7.5G 100% /snap8[OST:78]
snap8-OST004f_UUID
In my brief attempts to use lfs migrate, I found performance pretty slow
(it's serial). I also got some ASSERTs which should be fixed in 2.10 per
LU-8807; note that I was using 2.8. On a more trivial level, I found the
-v|--verbose option to the command doesn't work.
On 01/07/2019 12:26 PM,
> On Jan 7, 2019, at 2:09 PM, Jason Williams wrote:
>
> One last question, How safe is lfs_migrate? The man page on the installation
> says it's UNSAFE for possibly in-use files. The lustre manual doesn't have
> the same warning and says something about it being a bit more integrated with
lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
> On Jan 7, 2019, at 12:53 PM, Jason Williams wrote:
>
> As I have gone through the testing, I think you may be right. I think I
> disabled the OST in a slightly different w
> On Jan 7, 2019, at 12:53 PM, Jason Williams wrote:
>
> As I have gone through the testing, I think you may be right. I think I
> disabled the OST in a slightly different way and that caused issues.
>
> Do you happen to know where I could find out a bit more about what the "lctl
>
M
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
Jason,
The results you described sound like the correct behavior when you deactivate
an OST on the MDS. When you run “lctl —device deactivate”, you are
, January 7, 2019 11:47:09 AM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without
> hanging?
>
> So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I
>
o:jas...@jhu.edu>
From: lustre-discuss on behalf of
Jason Williams
Sent: Monday, January 7, 2019 11:47:09 AM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
So
Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was
following the wrong instructions. Do you know where the best instructions for
wh
From: Mohr Jr, Richard Frank (Rick Mohr)
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?
> On Jan 5, 2019, at 9:49 PM, Jason Williams wrote:
>
&g
> On Jan 5, 2019, at 9:49 PM, Jason Williams wrote:
>
> I have looked around the internet and found you can disable an OST, but when
> I have tried that, any writes (including deletes) to the OST hang the clients
> indefinitely. Does anyone know a way to make an OST basically "read-only"
>
Jason,
If there are files located on the full OST that you can delete then delete
them without trying to deactivate the OST. There is a process where you can
find files solely located on the full OST using lfs then deactivate the
full OST, copy the files to new files with some suffix (.new
Hello,
We have a lustre system (version 2.10.4) that has unfortunately fallen victim
to a 100% full OST... Every time we clear some space on it, the system fills it
right back up again.
I have looked around the internet and found you can disable an OST, but when I
have tried that, any
On 2010-01-29, at 03:07, Johann Lombardi wrote:
On Fri, Jan 29, 2010 at 10:32:26AM +0100, gvozden rovina wrote:
OST. For instance i copied 2.5 GB file to lustre which had 120 GB
storage
space (I have 2GB test OSTs) and it didn't automatically recognized
full
OST but it simply stopped
Hello!
I'm new to lustre and I have just one question.
While reading literature and playing with my test
lustre system I've noticed that there is strange (for me) issue with full
OST. For instance i copied 2.5 GB file to lustre which had 120 GB storage
space (I have 2GB test OSTs) and it didn't
On Fri, Jan 29, 2010 at 10:32:26AM +0100, gvozden rovina wrote:
OST. For instance i copied 2.5 GB file to lustre which had 120 GB storage
space (I have 2GB test OSTs) and it didn't automatically recognized full
OST but it simply stopped working with No space left on device error
message.
Hi!
Thank you for your swift answer. I have just one more question. Is it
possible to
configure lustre system so that it writes not just the file but also the
copy of the same file
in the same time as you create it?
thx!
On Fri, Jan 29, 2010 at 11:07 AM, Johann Lombardi joh...@sun.com wrote:
On Fri, Jan 29, 2010 at 11:48:58AM +0100, gvozden rovina wrote:
Thank you for your swift answer. I have just one more question. Is it
possible to
configure lustre system so that it writes not just the file but also the
copy of the same file in the same time as you create it?
No, we don't
thx for answer. good bye for now
On Fri, Jan 29, 2010 at 12:02 PM, Johann Lombardi joh...@sun.com wrote:
On Fri, Jan 29, 2010 at 11:48:58AM +0100, gvozden rovina wrote:
Thank you for your swift answer. I have just one more question. Is it
possible to
configure lustre system so that it
36 matches
Mail list logo
