Re: [lustre-discuss] confused about mdt space
On Apr 1, 2020, at 08:59, Mohr Jr, Richard Frank mailto:rm...@utk.edu>> wrote: On Apr 1, 2020, at 10:07 AM, Mohr Jr, Richard Frank mailto:rm...@utk.edu>> wrote: On Apr 1, 2020, at 3:55 AM, 肖正刚 mailto:guru.nov...@gmail.com>> wrote: For " the recent lustre versions use a 1KB inode size by default and the default format options create 1 inodes for every 2.5 KB of MDT space" : I checked the inode size is 1KB and in my online systems, as you said , about 40~41% of mdt disk space consumed by inodes. but from the manual I found the default "inode ratio" is 2K, so where the additional 0.5KB comes from ? I was basing this on info I found in this email thread: http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-February/015302.html I think the 2K/inode ratio listed in the manual may be incorrect (or perhaps someone can point out my mistake). Looking at the lustre source (utils/libmount_utils_ldiskfs.c), the default behavior is determined by this line: bytes_per_inode = inode_size + 1536 So the default bytes per inode is 1K (inode size) + 1.5K = 2.5K Right. In 2.10 the ldiskfs inode size was increased from 512 bytes to 1024 bytes in order to fit the more complex PFL layouts. That increased the total amount of space per inode from 2048 bytes to 2560 bytes. I guess the manual needs to be updated. Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
Now, I am clear. Thanks Richard ! Mohr Jr, Richard Frank 于2020年4月1日周三 下午10:59写道： > > > > On Apr 1, 2020, at 10:07 AM, Mohr Jr, Richard Frank > wrote: > > > > > > > >> On Apr 1, 2020, at 3:55 AM, 肖正刚 wrote: > >> > >> For " the recent lustre versions use a 1KB inode size by default and > the default format options create 1 inodes for every 2.5 KB of MDT space" : > >> I checked the inode size is 1KB and in my online systems, as you said > , about 40~41% of mdt disk space consumed by inodes. > >> but from the manual I found the default "inode ratio" is 2K, so where > the additional 0.5KB comes from ? > >> > > > > I was basing this on info I found in this email thread: > > > > > http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-February/015302.html > > > > I think the 2K/inode ratio listed in the manual may be incorrect (or > perhaps someone can point out my mistake). > > Looking at the lustre source (utils/libmount_utils_ldiskfs.c), the default > behavior is determined by this line: > > bytes_per_inode = inode_size + 1536 > > So the default bytes per inode is 1K (inode size) + 1.5K = 2.5K > > — > Rick Mohr > Senior HPC System Administrator > Joint Institute for Computational Sciences > University of Tennessee > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
> On Apr 1, 2020, at 10:07 AM, Mohr Jr, Richard Frank wrote: > > > >> On Apr 1, 2020, at 3:55 AM, 肖正刚 wrote: >> >> For " the recent lustre versions use a 1KB inode size by default and the >> default format options create 1 inodes for every 2.5 KB of MDT space" : >> I checked the inode size is 1KB and in my online systems, as you said , >> about 40~41% of mdt disk space consumed by inodes. >> but from the manual I found the default "inode ratio" is 2K, so where the >> additional 0.5KB comes from ? >> > > I was basing this on info I found in this email thread: > > http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-February/015302.html > > I think the 2K/inode ratio listed in the manual may be incorrect (or perhaps > someone can point out my mistake). Looking at the lustre source (utils/libmount_utils_ldiskfs.c), the default behavior is determined by this line: bytes_per_inode = inode_size + 1536 So the default bytes per inode is 1K (inode size) + 1.5K = 2.5K — Rick Mohr Senior HPC System Administrator Joint Institute for Computational Sciences University of Tennessee ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
> On Apr 1, 2020, at 3:55 AM, 肖正刚 wrote: > > For " the recent lustre versions use a 1KB inode size by default and the > default format options create 1 inodes for every 2.5 KB of MDT space" : > I checked the inode size is 1KB and in my online systems, as you said , > about 40~41% of mdt disk space consumed by inodes. > but from the manual I found the default "inode ratio" is 2K, so where the > additional 0.5KB comes from ? > I was basing this on info I found in this email thread: http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-February/015302.html I think the 2K/inode ratio listed in the manual may be incorrect (or perhaps someone can point out my mistake). — Rick Mohr Senior HPC System Administrator Joint Institute for Computational Sciences University of Tennessee ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
Hi, Please forget my first question, I made a mistake. For " the recent lustre versions use a 1KB inode size by default and the default format options create 1 inodes for every 2.5 KB of MDT space" : I checked the inode size is 1KB and in my online systems, as you said , about 40~41% of mdt disk space consumed by inodes. but from the manual I found the default "inode ratio" is 2K, so where the additional 0.5KB comes from ? Thanks. 肖正刚 于2020年4月1日周三 下午1:00写道： > Thanks a lot. > I have two more questions: > 1) Assume I consider the mdt space use the method described in lustre > manual, by calculation, the metadata space is 400GB. > After format(default option), about 160GB(40% of 400GB) preallocated for > inodes, so the avalaible inodes number is less than estimated, right ? > 2) mds need additional space for other use, like log,acls,xattrs；how to > estimate these space ? > > Thanks! > > Mohr Jr, Richard Frank 于2020年3月31日周二 下午9:57写道： > >> >> >> > On Mar 30, 2020, at 10:56 PM, 肖正刚 wrote: >> > >> > Hello, I have some question about metadata space. >> > 1) I have ten 960GB SAS SSDs for mdt,after done raid10,we have 4.7TB >> space free. >> > after formated as mdt,we only have 2.6TB space free; so where the 2.1TB >> space go ? >> > 2) for the 2.6TB space, what's it used for? >> >> That space is used by inodes. I believe the recent lustre versions use a >> 1KB inode size by default and the default format options create 1 inodes >> for every 2.5 KB of MDT space. So about 40% of your disk space will be >> consumed by inodes. >> >> — >> Rick Mohr >> Senior HPC System Administrator >> Joint Institute for Computational Sciences >> University of Tennessee >> >> >> >> >> ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
Thanks a lot. I have two more questions: 1) Assume I consider the mdt space use the method described in lustre manual, by calculation, the metadata space is 400GB. After format(default option), about 160GB(40% of 400GB) preallocated for inodes, so the avalaible inodes number is less than estimated, right ? 2) mds need additional space for other use, like log,acls,xattrs；how to estimate these space ? Thanks! Mohr Jr, Richard Frank 于2020年3月31日周二 下午9:57写道： > > > > On Mar 30, 2020, at 10:56 PM, 肖正刚 wrote: > > > > Hello, I have some question about metadata space. > > 1) I have ten 960GB SAS SSDs for mdt,after done raid10,we have 4.7TB > space free. > > after formated as mdt,we only have 2.6TB space free; so where the 2.1TB > space go ? > > 2) for the 2.6TB space, what's it used for? > > That space is used by inodes. I believe the recent lustre versions use a > 1KB inode size by default and the default format options create 1 inodes > for every 2.5 KB of MDT space. So about 40% of your disk space will be > consumed by inodes. > > — > Rick Mohr > Senior HPC System Administrator > Joint Institute for Computational Sciences > University of Tennessee > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] confused about mdt space
> On Mar 30, 2020, at 10:56 PM, 肖正刚 wrote: > > Hello, I have some question about metadata space. > 1) I have ten 960GB SAS SSDs for mdt,after done raid10,we have 4.7TB space > free. > after formated as mdt,we only have 2.6TB space free; so where the 2.1TB space > go ? > 2) for the 2.6TB space, what's it used for? That space is used by inodes. I believe the recent lustre versions use a 1KB inode size by default and the default format options create 1 inodes for every 2.5 KB of MDT space. So about 40% of your disk space will be consumed by inodes. — Rick Mohr Senior HPC System Administrator Joint Institute for Computational Sciences University of Tennessee ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] confused about mdt space
Hello, I have some question about metadata space. 1) I have ten 960GB SAS SSDs for mdt,after done raid10,we have 4.7TB space free. after formated as mdt,we only have 2.6TB space free; so where the 2.1TB space go ? 2) for the 2.6TB space, what's it used for? ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
