On Apr 15, 2019, at 12:56, Mohr Jr, Richard Frank (Rick Mohr) <rm...@utk.edu> wrote: > > >> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar <youssefelda...@gmail.com> >> wrote: >> >> For one Lustre filesystem, inode count in the summary is notably less than >> what the individual OST inode counts would add up to: > > The first thing to understand is that every Lustre file will consume one > inode on the MDT, and this inode uses attributes to store information about > which OSTs the file is striped over. Then for each file stripe, there will > also be an inode consumed on the corresponding OSTs. For example, a file > with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs > (one inode on each OST the file is striped over). > >> # lfs df -i /lfs01 >> UUID Inodes IUsed IFree IUse% Mounted on >> lustrefs-MDT0000_UUID 2402287616 46560885 2355726731 2% >> /share/lfs01[MDT:0] >> lustrefs-OST0001_UUID 24117248 22883788 1233460 95% >> /share/lfs01[OST:1] >> lustrefs-OST0003_UUID 24117248 22903308 1213940 95% >> /share/lfs01[OST:3] >> lustrefs-OST0004_UUID 24117248 22895442 1221806 95% >> /share/lfs01[OST:4] >> lustrefs-OST0006_UUID 24117248 22890201 1227047 95% >> /share/lfs01[OST:6] >> >> filesystem_summary: 51457138 46560885 4896253 90% /share/lfs01 > > On this file system, there are already 46,560,885 files which also consume > the same number of inodes on the MDT (so IUsed=46560885). However, even > though the MDT has over 2 billion inodes free, every file created in the > future will use at least one inode on an OST. If you add up all the free > inodes on all the OSTs, you get 4896253. So at best, there is only space for > 4,896,253 more files. That is why IFree=4896253. Then, Inodes = IUsed + > IFree = 46,560,885 + 4,896,253 = 51,457,138. > >> On another filesystem, this is not the case: >> >> # lfs df -i /lfs02 >> UUID Inodes IUsed IFree IUse% Mounted on >> lustrefs-MDT0000_UUID 1288503296 19222318 1269280978 1% >> /share/lfs02[MDT:0] >> lustrefs-OST0001_UUID 24117248 5942156 18175092 25% >> /share/lfs02[OST:1] >> lustrefs-OST0002_UUID 24117248 5816469 18300779 24% >> /share/lfs02[OST:2] >> lustrefs-OST0003_UUID 24117248 5982962 18134286 25% >> /share/lfs02[OST:3] >> >> filesystem_summary: 73832475 19222318 54610157 26% /share/lfs02 > > Again, there are already 19,222,318 files on the file system, so > IUsed=19222318. All the OSTs together only have 18,175,092 + 18,300,779 + > 18,134,286 = 54,610,157 inodes available, so IFree=54610157. And Inodes = > IUsed + IFree = 73832475.
Thanks to Rick for the good explanation here. One thing to add is that it appears that the /lfs01 filesystem has a default stripe_count=2, since there are 46560885 inodes used on MDT0000 and 91572739 total objects used on the four OSTs, and 91572739/46560885 = 1.96 OST objects per MDT inode. If you have a large number of small files, you don't need a high stripe count. Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
