Just a quick addition to this thread (for now), here's the presentation I think Killian refers to: http://www.opensfs.org/wp-content/uploads/2014/04/D2_S16_PLFSandLustrePerformanceComparison.pdf
And here's *all* the slides from LUG 2014: http://opensfs.org/lug-2014-presos/ Some of them are incredibly informative, and I have high hopes for some of the future changes referenced in Andreas Dilger's presentation ( http://opensfs.org/wp-content/uploads/2014/04/D1_S9_LustreFutureFeatures.pdf ) - specifically, I think the combination of DNE (phase 1 & 2) and small files on MDTs should help immensely with both IOPS and overall throughput on those sorts of small-file workloads. Cheers, - Brian On Fri, Jun 13, 2014 at 2:02 PM, Kilian Cavalotti < [email protected]> wrote: > On Fri, Jun 13, 2014 at 7:03 AM, Ellis H. Wilson III <[email protected]> > wrote: > > a) Fix it transparently with automatic policies/FS's in the back-end. (I > > know of at least one FS that packs small files with metadata > transparently > > on SSDs to expedite small file IOPS, but message me off-list for that as > I > > start work for that shop soon and don't want to so blatantly advertise). > > There are limits to how much these policies/FS's can fix though. Bad I/O > > will still be Bad I/O after a point. > > Speaking of which, there's this project from Los Alamos, PLFS > (https://github.com/plfs/plfs-core), that tries to aggregate random > I/O from applications into sequential operations that better suit the > underlying parallel filesystem. Never used it myself, but I've seen a > presentation from Brett Kettering at LUG'14 which outlined nice > performance improvements for some use cases. > > Cheers, > -- > Kilian > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf >
_______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
