Re: [lustre-discuss] Slow release of inodes on OST

[Åke Sandgren]

On 2/6/20 6:58 AM, Andreas Dilger wrote:
> On Feb 4, 2020, at 07:23, Åke Sandgren  > wrote:
>>
>> When I create a large number of files on an OST and then remove them,
>> the used inode count on the OST decreases very slowly, it takes several
>> hours for it to go from 3M to the correct ~10k.
>>
>> (I'm running the io500 test suite)
>>
>> Is there something I can do to make it release them faster?
>> Right now it has gone from 3M to 1.5M in 6 hours, (lfs df -i).
> 
> It this the object count or the file count?  Are you possibly using a lot of
> stripes on the files being deleted that is multiplying the work needed?
> 
>> These are SSD based OST's in case it matters.
> 
> The MDS controls the destroy of the OST objects, so there is a rate
> limit, but ~700/s seems low to me, especially for SSD OSTs.
> 
> You could check "lctl get_param osp.*.sync*" on the MDS to see how
> many destroys are pending.  Also, increasing osp.*.max_rpcs_in_flight
> on the MDS might speed this up?  It should default to 32 per OST on
> the MDS vs. default 8 for clients

Should have checked before sending first mail...
The default on the MDS is apparently 8. This is 2.13.0 and I did not
change any params.

lctl get_param osp.*.max_rpcs_in_flight
osp.rsos-OST-osc-MDT.max_rpcs_in_flight=8

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90-580 14
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Slow release of inodes on OST

[Åke Sandgren]

On 2/6/20 6:58 AM, Andreas Dilger wrote:
> On Feb 4, 2020, at 07:23, Åke Sandgren  > wrote:
>>
>> When I create a large number of files on an OST and then remove them,
>> the used inode count on the OST decreases very slowly, it takes several
>> hours for it to go from 3M to the correct ~10k.
>>
>> (I'm running the io500 test suite)
>>
>> Is there something I can do to make it release them faster?
>> Right now it has gone from 3M to 1.5M in 6 hours, (lfs df -i).
> 
> It this the object count or the file count?  Are you possibly using a lot of
> stripes on the files being deleted that is multiplying the work needed?

I'm checking lfs df -i for of the OST value, no stripes single OST.

>> These are SSD based OST's in case it matters.
> 
> The MDS controls the destroy of the OST objects, so there is a rate
> limit, but ~700/s seems low to me, especially for SSD OSTs.
> 
> You could check "lctl get_param osp.*.sync*" on the MDS to see how
> many destroys are pending.  Also, increasing osp.*.max_rpcs_in_flight
> on the MDS might speed this up?  It should default to 32 per OST on
> the MDS vs. default 8 for clients

Thanks, that gives me something to play with. Didn't manage to find
anything about this in the manual, but on the other hand I didn't really
know what to search for...

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90-580 14
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Slow release of inodes on OST

[Andreas Dilger]

On Feb 4, 2020, at 07:23, Åke Sandgren 
mailto:ake.sandg...@hpc2n.umu.se>> wrote:

When I create a large number of files on an OST and then remove them,
the used inode count on the OST decreases very slowly, it takes several
hours for it to go from 3M to the correct ~10k.

(I'm running the io500 test suite)

Is there something I can do to make it release them faster?
Right now it has gone from 3M to 1.5M in 6 hours, (lfs df -i).

It this the object count or the file count?  Are you possibly using a lot of
stripes on the files being deleted that is multiplying the work needed?

These are SSD based OST's in case it matters.

The MDS controls the destroy of the OST objects, so there is a rate
limit, but ~700/s seems low to me, especially for SSD OSTs.

You could check "lctl get_param osp.*.sync*" on the MDS to see how
many destroys are pending.  Also, increasing osp.*.max_rpcs_in_flight
on the MDS might speed this up?  It should default to 32 per OST on
the MDS vs. default 8 for clients

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LUSTRE - Installation on DEBIAN 10.x ????

[Matthias Krawutschke]

Hello everybody,

 

I am new here on this platform and I have a question about LUSTRE – 
Installation on DEBIAN 10.x ….. 😊

It is possible to install this on this LINUX – Version & if yes: which version 
of LUSTRE does it work and how?

 

Best regards….

 

 

 

Matthias Krawutschke, Dipl. Inf.

 

Universität Potsdam
ZIM - Zentrum für Informationstechnologie und Medienmanagement
Team High-Performance-Computing

Campus Am Neuen Palais: Am Neuen Palais 10 | 14469 Potsdam
Tel: +49 331 977-, Fax: +49 331 977-1750

 

Internet:   
https://www.uni-potsdam.de/de/zim/angebote-loesungen/hpc.html

 

 

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org