Re: [lustre-discuss] MDT 100% full

[Dilger, Andreas]

There isn't Data-on-MDT (DoM) functionality available yet, so there is no data 
being stored on the MDT.  With DNE1 there is no automatic distribution of files 
between MDTs, so unless you have explicitly created subdirectories on the 
second MDT with "lfs mkdir -i 1" there will not be any files created there.  
DNE1 is intended to create some high-level directories (e.g. per user or per 
project) on the second MDT, and files created in those directories will be 
created on the second MDT.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division

On 2016/07/27, 17:53, "lustre-discuss on behalf of Andrus, Brian Contractor" 

 on behalf of bdand...@nps.edu> wrote:

Rick,

Thanks for the info. I added more to the ZFS pool and it was immediately 
available and all was happy.

Now I need to figure out how/why the MDT got so full.
I have 2 MDTs and the other was barely used. Both were 100GB, which I would 
have thought was fine. I suspect I have a user that put many small files on the 
filesystem and they all landed on the MDT for storage.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


From: Rick Wagner [mailto:rpwag...@sdsc.edu]
Sent: Tuesday, July 26, 2016 5:58 PM
To: Andrus, Brian Contractor 
Cc: lustre-discuss@lists.lustre.org; Cai, Haisong 
Subject: Re: [lustre-discuss] MDT 100% full

Hi Brian,

On Jul 26, 2016, at 5:45 PM, Andrus, Brian Contractor 
> wrote:
All,

Ok, I thought 100GB would be sufficient for an MDT.
I have 2 MDTs as well, BUT…

MDT0 is 100% full and now I cannot write anything to my lustre filesystem.
The MDT is on a ZFS backing filesystem.

So, what is the proper way to grow my MDT using ZFS? Do I need to shut the 
filesystem down completely? Can I just add a disk or space to the pool and 
Lustre will see it?

Any advice or direction is appreciated.

We just did this successfully on the two MDTs backing one of our Lustre file 
systems and everything happened at the ZFS layer. We added drives to the pool 
and Lustre immediately saw the additional capacity. Whether you take down the 
file system or do it live is a question of your architecture, skills, and 
confidence. Having test file system is also worthwhile to go over the steps.

--Rick






Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Problem after implementing Lustre & Filesystem Transfer sync

[yasir]

Hi,

Recently I've installed Lustre over Centos 6.7 and Lustre Client on Rocks
6.2 (Centos 6.6) after installation I'm getting following error. Need all
your help & support for below 2 Query.

 

Client Error: LustreError: 5508:0:(mdc_locks.c:918:mdc_enqueue())
ldlm_cli_enqueue: -13

OST Error: Lustre: lustre-OST: Connection restored to
47a64818-04a3-f5ab-ae55-5e1ff7a79eed (at 10.1.1.XX@tcp)

MDS Error: Lustre: MGS: Connection restored to
c8b97f5c-5f17-d340-de69-3d15e514ad84 (at 10.1.1.XX@tcp)

Lustre: Skipped 1 previous similar message

Lustre: MGS: Connection restored to 56b2603a-b757-f4e1-68c5-bdb91b11c634 (at
10.1.1.XX@tcp)

Lustre: Skipped 1 previous similar message

 

 

Here I would like to add one more query & need your support to clear same.

I've deploy lustre with separate server of Metadata MDT/MDS & Object OST/OSS
and mounted over lustre client machine which is having some user like: A,B,C

All infra is running on 1Gbps Ethernet Network.

 

Query: When I'm copying file near about 900Mb on lustre client in /lustre
(directory) it get copied quick but when I want to access it from any user
A.B.C it's giving permission denied error & some time directory permission
problem but after 1hrs-2hrs it allow us to access directory or files.

 

Is this just because of 1Gbps network or something else is making problem.

 

 

 

Thanks & Regards,

 

 

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDT 100% full

[Andrus, Brian Contractor]

Rick,

Thanks for the info. I added more to the ZFS pool and it was immediately 
available and all was happy.

Now I need to figure out how/why the MDT got so full.
I have 2 MDTs and the other was barely used. Both were 100GB, which I would 
have thought was fine. I suspect I have a user that put many small files on the 
filesystem and they all landed on the MDT for storage.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


From: Rick Wagner [mailto:rpwag...@sdsc.edu]
Sent: Tuesday, July 26, 2016 5:58 PM
To: Andrus, Brian Contractor 
Cc: lustre-discuss@lists.lustre.org; Cai, Haisong 
Subject: Re: [lustre-discuss] MDT 100% full

Hi Brian,

On Jul 26, 2016, at 5:45 PM, Andrus, Brian Contractor 
> wrote:
All,

Ok, I thought 100GB would be sufficient for an MDT.
I have 2 MDTs as well, BUT...

MDT0 is 100% full and now I cannot write anything to my lustre filesystem.
The MDT is on a ZFS backing filesystem.

So, what is the proper way to grow my MDT using ZFS? Do I need to shut the 
filesystem down completely? Can I just add a disk or space to the pool and 
Lustre will see it?

Any advice or direction is appreciated.

We just did this successfully on the two MDTs backing one of our Lustre file 
systems and everything happened at the ZFS layer. We added drives to the pool 
and Lustre immediately saw the additional capacity. Whether you take down the 
file system or do it live is a question of your architecture, skills, and 
confidence. Having test file system is also worthwhile to go over the steps.

--Rick





Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Analog of ll_recover_lost_found_objs for MDS

[Martin Hecht]

Hi James,

I'm not aware of a ready-to use tool, but if you have captured the
output of e2fsck you can use that as a basis for a script that puts the
files back to their original location.
e2fsck usually prints out the full path and the inode numbers of the
files/directories which it moves to lost+found and there, they are named
"#$inode" (which makes scripting a bit ugly, but if you properly escape
the '#'-sign and do some magic with awk, perl or alike, you can
transform the log to a shell script that moves your files back to the
orignal path). I have done this once after a file system corruption
after an upgrade from 1.8 to 2.4 (which contained an ugly bug when
enabling the "FID in dirent" feature).

The backup... well, it gives you the chance to go back to the state
where you started *before* the e2fsck. That would be a chance to capture
the output again, in case you did not store it (actually, you could do
this offline, on a copy of the backup). Restoring the MDT out of the
backup however is only useful as long as you did not go in production
after the e2fsck. And as I said, you still have to "repair" the restored
MDT (probably by doing the same steps as you already did on the live
system), but a second chance is better than no chance to go back... The
backup is also good to investigate what happened during the e2fsck (in
case it did something weird) or to go in with e2fsdebug for manual
investigations... (e.g. manually look up inode<->path relations).

Martin


On 07/26/2016 04:08 PM, jbellinger wrote:
> Is there, or could there be, something analogous to the OST recovery
> tool that works on the lost+found on the MDT?  e2fsck went berserk.
>
> We're running 2.5.3.
>
>
> Thanks,
> James Bellinger
>
> Yes, we have an older (therefore somewhat inconsistent) backup of the
> mdt, so we should be able to recover most things, _in theory_.  In
> practice -- we'd love to hear other people's experience about recovery
> using an inconsistent backup.
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





smime.p7s
Description: S/MIME Cryptographic Signature
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org