Re: [lustre-discuss] inodes not adding up
On Apr 15, 2019, at 12:56, Mohr Jr, Richard Frank (Rick Mohr) wrote: > > >> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar >> wrote: >> >> For one Lustre filesystem, inode count in the summary is notably less than >> what the individual OST inode counts would add up to: > > The first thing to understand is that every Lustre file will consume one > inode on the MDT, and this inode uses attributes to store information about > which OSTs the file is striped over. Then for each file stripe, there will > also be an inode consumed on the corresponding OSTs. For example, a file > with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs > (one inode on each OST the file is striped over). > >> # lfs df -i /lfs01 >> UUID Inodes IUsed IFree IUse% Mounted on >> lustrefs-MDT_UUID 240228761646560885 2355726731 2% >> /share/lfs01[MDT:0] >> lustrefs-OST0001_UUID2411724822883788 1233460 95% >> /share/lfs01[OST:1] >> lustrefs-OST0003_UUID2411724822903308 1213940 95% >> /share/lfs01[OST:3] >> lustrefs-OST0004_UUID2411724822895442 1221806 95% >> /share/lfs01[OST:4] >> lustrefs-OST0006_UUID2411724822890201 1227047 95% >> /share/lfs01[OST:6] >> >> filesystem_summary: 5145713846560885 4896253 90% /share/lfs01 > > On this file system, there are already 46,560,885 files which also consume > the same number of inodes on the MDT (so IUsed=46560885). However, even > though the MDT has over 2 billion inodes free, every file created in the > future will use at least one inode on an OST. If you add up all the free > inodes on all the OSTs, you get 4896253. So at best, there is only space for > 4,896,253 more files. That is why IFree=4896253. Then, Inodes = IUsed + > IFree = 46,560,885 + 4,896,253 = 51,457,138. > >> On another filesystem, this is not the case: >> >> # lfs df -i /lfs02 >> UUID Inodes IUsed IFree IUse% Mounted on >> lustrefs-MDT_UUID 128850329619222318 1269280978 1% >> /share/lfs02[MDT:0] >> lustrefs-OST0001_UUID24117248 594215618175092 25% >> /share/lfs02[OST:1] >> lustrefs-OST0002_UUID24117248 581646918300779 24% >> /share/lfs02[OST:2] >> lustrefs-OST0003_UUID24117248 598296218134286 25% >> /share/lfs02[OST:3] >> >> filesystem_summary: 738324751922231854610157 26% /share/lfs02 > > Again, there are already 19,222,318 files on the file system, so > IUsed=19222318. All the OSTs together only have 18,175,092 + 18,300,779 + > 18,134,286 = 54,610,157 inodes available, so IFree=54610157. And Inodes = > IUsed + IFree = 73832475. Thanks to Rick for the good explanation here. One thing to add is that it appears that the /lfs01 filesystem has a default stripe_count=2, since there are 46560885 inodes used on MDT and 91572739 total objects used on the four OSTs, and 91572739/46560885 = 1.96 OST objects per MDT inode. If you have a large number of small files, you don't need a high stripe count. Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] inodes not adding up
> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar wrote: > > For one Lustre filesystem, inode count in the summary is notably less than > what the individual OST inode counts would add up to: The first thing to understand is that every Lustre file will consume one inode on the MDT, and this inode uses attributes to store information about which OSTs the file is striped over. Then for each file stripe, there will also be an inode consumed on the corresponding OSTs. For example, a file with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs (one inode on each OST the file is striped over). > # lfs df -i /lfs01 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT_UUID 240228761646560885 2355726731 2% > /share/lfs01[MDT:0] > lustrefs-OST0001_UUID2411724822883788 1233460 95% > /share/lfs01[OST:1] > lustrefs-OST0003_UUID2411724822903308 1213940 95% > /share/lfs01[OST:3] > lustrefs-OST0004_UUID2411724822895442 1221806 95% > /share/lfs01[OST:4] > lustrefs-OST0006_UUID2411724822890201 1227047 95% > /share/lfs01[OST:6] > > filesystem_summary: 5145713846560885 4896253 90% /share/lfs01 On this file system, there are already 46,560,885 files which also consume the same number of inodes on the MDT (so IUsed=46560885). However, even though the MDT has over 2 billion inodes free, every file created in the future will use at least one inode on an OST. If you add up all the free inodes on all the OSTs, you get 4896253. So at best, there is only space for 4,896,253 more files. That is why IFree=4896253. Then, Inodes = IUsed + IFree = 46,560,885 + 4,896,253 = 51,457,138. > On another filesystem, this is not the case: > > # lfs df -i /lfs02 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT_UUID 128850329619222318 1269280978 1% > /share/lfs02[MDT:0] > lustrefs-OST0001_UUID24117248 594215618175092 25% > /share/lfs02[OST:1] > lustrefs-OST0002_UUID24117248 581646918300779 24% > /share/lfs02[OST:2] > lustrefs-OST0003_UUID24117248 598296218134286 25% > /share/lfs02[OST:3] > > filesystem_summary: 738324751922231854610157 26% /share/lfs02 Again, there are already 19,222,318 files on the file system, so IUsed=19222318. All the OSTs together only have 18,175,092 + 18,300,779 + 18,134,286 = 54,610,157 inodes available, so IFree=54610157. And Inodes = IUsed + IFree = 73832475. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] inodes not adding up
Likely you're seeing the discrepancy as a result of precreated objects. Cheers On Sat, Apr 13, 2019, 2:58 AM Youssef Eldakar wrote: > For one Lustre filesystem, inode count in the summary is notably less than > what the individual OST inode counts would add up to: > > # lfs df -i /lfs01 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT_UUID 240228761646560885 2355726731 2% > /share/lfs01[MDT:0] > lustrefs-OST0001_UUID2411724822883788 1233460 95% > /share/lfs01[OST:1] > lustrefs-OST0003_UUID2411724822903308 1213940 95% > /share/lfs01[OST:3] > lustrefs-OST0004_UUID2411724822895442 1221806 95% > /share/lfs01[OST:4] > lustrefs-OST0006_UUID2411724822890201 1227047 95% > /share/lfs01[OST:6] > > filesystem_summary: 5145713846560885 4896253 90% /share/lfs01 > > On another filesystem, this is not the case: > > # lfs df -i /lfs02 > UUID Inodes IUsed IFree IUse% Mounted on > lustrefs-MDT_UUID 128850329619222318 1269280978 1% > /share/lfs02[MDT:0] > lustrefs-OST0001_UUID24117248 594215618175092 25% > /share/lfs02[OST:1] > lustrefs-OST0002_UUID24117248 581646918300779 24% > /share/lfs02[OST:2] > lustrefs-OST0003_UUID24117248 598296218134286 25% > /share/lfs02[OST:3] > > filesystem_summary: 738324751922231854610157 26% /share/lfs02 > > In fact, in the latter case, the whole is slightly greater than the sum of > its parts. > > Any pointers regarding where I would look for an explanation would be > appreciated. > > Youssef Eldakar > Bibliotheca Alexandrina > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] inodes not adding up
For one Lustre filesystem, inode count in the summary is notably less than what the individual OST inode counts would add up to: # lfs df -i /lfs01 UUID Inodes IUsed IFree IUse% Mounted on lustrefs-MDT_UUID 240228761646560885 2355726731 2% /share/lfs01[MDT:0] lustrefs-OST0001_UUID2411724822883788 1233460 95% /share/lfs01[OST:1] lustrefs-OST0003_UUID2411724822903308 1213940 95% /share/lfs01[OST:3] lustrefs-OST0004_UUID2411724822895442 1221806 95% /share/lfs01[OST:4] lustrefs-OST0006_UUID2411724822890201 1227047 95% /share/lfs01[OST:6] filesystem_summary: 5145713846560885 4896253 90% /share/lfs01 On another filesystem, this is not the case: # lfs df -i /lfs02 UUID Inodes IUsed IFree IUse% Mounted on lustrefs-MDT_UUID 128850329619222318 1269280978 1% /share/lfs02[MDT:0] lustrefs-OST0001_UUID24117248 594215618175092 25% /share/lfs02[OST:1] lustrefs-OST0002_UUID24117248 581646918300779 24% /share/lfs02[OST:2] lustrefs-OST0003_UUID24117248 598296218134286 25% /share/lfs02[OST:3] filesystem_summary: 738324751922231854610157 26% /share/lfs02 In fact, in the latter case, the whole is slightly greater than the sum of its parts. Any pointers regarding where I would look for an explanation would be appreciated. Youssef Eldakar Bibliotheca Alexandrina ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
