-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/06/14 05:03, Tom Harvill wrote:
> We've found that a large majority of our files (~40MM of ~50MM) > are less than 10KB. We believe our filesystem (lustre) is > bottlenecked with IOPs and locking related to jobs running against > these files. We have ~700TB usable storage with ~500TB consumed, > almost all consumption is by a relatively small number of very very > large files. Coincidentally I was talking to some people involved in Lustre yesterday about it and I asked them how well it coped. There was a bit of shuffling of feet and the answer was along the lines that it wasn't really their target use case. Our GPFS filesystem has a lot of fairly small files, but nothing like the proportion that you are seeing (4/5ths < 10kB). We have had odd edge cases, like people running OpenFOAM creating a directory hierarchy of millions of directories, each with 5 files in them. That was fun... All the best! Chris - -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOanGIACgkQO2KABBYQAh/ZSQCcCsHfUwWLt5H4mPBij+IAqQsz XVcAnAtFPoLVF53qymI8KwieqGo7cQMl =590L -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
