On Sep 8, 2020, at 9:13 PM, Tung-Han Hsieh <thhs...@twcp1.phys.ntu.edu.tw> wrote: > > I would like to ask whether Lustre file system has implemented the > function to optimize for large sparse data files ? > > For example, a 3GB data file but with more than 80% bytes zero, can > Lustre file system optimize the storage not actually taking the whole > 3GB of disk space ?
Could you please explain your usage further? Lustre definitely has support for sparse files - if they are written by an application with "seek" or by multiple threads in parallel, then only the blocks that are written will use space on the OST. For ldiskfs the block size is 4KB. For ZFS the OST block size is up to 1MB, if the file size is 1MB or larger. That is why compression on ZFS can help reduce space usage on the OST, because it can effectively compress the 1MB blocks that are nearly full of zeroes, if your sparse writes are smaller than the blocksize. If you are *copying* a sparse file, that depends on the tool that is doing the copy. For example, "cp --sparse=always" will generate a sparse file. We are also working on adding SEEK_HOLE and SEEK_DATA, which will help tools to copy sparse files. Cheers, Andreas > I know that some file systems (e.g., ZFS) has this function. If Lustre > does not have it, is there a roadmap to implement it in the future ? > > Thanks for your reply in advance. > > Best Regards, > > T.H.Hsieh > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Cheers, Andreas
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
