Hello Roch, Wednesday, August 22, 2007, 10:13:10 AM, you wrote:
RP> £ukasz K writes: >> > Is ZFS efficient at handling huge populations of tiny-to-small files - >> > for example, 20 million TIFF images in a collection, each between 5 >> > and 500k in size? >> > >> > I am asking because I could have sworn that I read somewhere that it >> > isn't, but I can't find the reference. >> >> It depends, what type of I/O you will do. If only reads, there is no >> problem. Writting small files ( and removing ) will fragmentate pool >> and it will be a huge problem. >> You can set recordsize to 32k ( or 16k ) and it will help for some time. >> RP> Comparing recordsize of 16K with 128K. RP> Files in the range of [0,16K] : no difference. RP> Files in the range of [16K,128K] : more efficient to use 128K RP> Files in the range of [128K,500K] : more efficient to use 16K RP> In the [16K,128K] range the actual filesize is rounded up to RP> 16K with 16K recordsize and to the nearest 512B boundary RP> with 128K recordsize. This will be fairly catastrophic for RP> files slightly above 16K (rounded up to 32K vs 16K+512B). RP> In the [128K, 500K] range we're hurt by this RP> 5003563 use smaller "tail block" for last block of object RP> until it is fixed, then yes , files stored using 16K RP> records are rounded up more tightly. metadata probably RP> east parts of the gains. Roch, I guess Lukasz was talking about some problems we're seeing here which are partly caused by utilizing all 128KB slabs, so forcing file system to 16KB helps here (for CPU) - workaround. Sure, we're talking about lots and lots of files, really small. Perhaps someone could work with Lukasz and investigate it more closely. Lukasz posted so more detailed info not so long ago - unfortunately there was no feedback. -- Best regards, Robert Milkowski mailto:[EMAIL PROTECTED] http://milek.blogspot.com _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
