> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar <youssefelda...@gmail.com> wrote: > > For one Lustre filesystem, inode count in the summary is notably less than > what the individual OST inode counts would add up to:
The first thing to understand is that every Lustre file will consume one inode on the MDT, and this inode uses attributes to store information about which OSTs the file is striped over. Then for each file stripe, there will also be an inode consumed on the corresponding OSTs. For example, a file with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs (one inode on each OST the file is striped over). > # lfs df -i /lfs01 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT0000_UUID 2402287616 46560885 2355726731 2% > /share/lfs01[MDT:0] > lustrefs-OST0001_UUID 24117248 22883788 1233460 95% > /share/lfs01[OST:1] > lustrefs-OST0003_UUID 24117248 22903308 1213940 95% > /share/lfs01[OST:3] > lustrefs-OST0004_UUID 24117248 22895442 1221806 95% > /share/lfs01[OST:4] > lustrefs-OST0006_UUID 24117248 22890201 1227047 95% > /share/lfs01[OST:6] > > filesystem_summary: 51457138 46560885 4896253 90% /share/lfs01 On this file system, there are already 46,560,885 files which also consume the same number of inodes on the MDT (so IUsed=46560885). However, even though the MDT has over 2 billion inodes free, every file created in the future will use at least one inode on an OST. If you add up all the free inodes on all the OSTs, you get 4896253. So at best, there is only space for 4,896,253 more files. That is why IFree=4896253. Then, Inodes = IUsed + IFree = 46,560,885 + 4,896,253 = 51,457,138. > On another filesystem, this is not the case: > > # lfs df -i /lfs02 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT0000_UUID 1288503296 19222318 1269280978 1% > /share/lfs02[MDT:0] > lustrefs-OST0001_UUID 24117248 5942156 18175092 25% > /share/lfs02[OST:1] > lustrefs-OST0002_UUID 24117248 5816469 18300779 24% > /share/lfs02[OST:2] > lustrefs-OST0003_UUID 24117248 5982962 18134286 25% > /share/lfs02[OST:3] > > filesystem_summary: 73832475 19222318 54610157 26% /share/lfs02 Again, there are already 19,222,318 files on the file system, so IUsed=19222318. All the OSTs together only have 18,175,092 + 18,300,779 + 18,134,286 = 54,610,157 inodes available, so IFree=54610157. And Inodes = IUsed + IFree = 73832475. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org