On Wed, Oct 21, 2009 at 12:31 PM, Kyle Schochenmaier <[email protected]> wrote: > Chris - > > A lot of work has been done with pvfs2 and infiniband. > Pete Wyckoff has many publications with it - (I cant locate them > right now from my old link to his publications) > There are some documents from my work at AmesLab floating around, and > a careful search of the mailing list archives here will yield some > user-contributed data. > > If you specify what type of performance metrics you are looking for I > might be able to help further. We can take that offline.
I've got drives that do 800MB/s locally, and scale linearly (locally) up to 6GB/s as you add drives. Exporting these via SRP from one target, I can get 2.7GB/s on the initiator using one QDR IB port, and 3.7GB/s using two QDR IB ports (off two different HCA's on the same target). How would PVFS compare and scale? Thanks, Chris P.S. thanks to Rob for the sanity check. > > Hope that helps. > > Kyle Schochenmaier > > > > On Wed, Oct 21, 2009 at 1:22 PM, Rob Ross <[email protected]> wrote: >> Hi Chris, >> >> Yes, you can certainly run PVFS servers on compute nodes. That's what we >> used to do all the time back in "the day", but it's still possible. >> >> The file system does not migrate files. In part this is because it would >> dramatically complicate metadata management, in part because we are >> frequently running in environments where there's no locality to exploit. >> >> You can choose to distribute over fewer (or just one) node, and there is >> functionality to discover where a file is stored, so you can map your >> computation onto the storage. >> >> Obviously yes, doing work on some other node will slow down computation on >> that node, how much depends on many things. >> >> Rob >> >> On Oct 21, 2009, at 12:40 PM, Chris Worley wrote: >> >>> I haven't looked at PVFS for nearly a decade, so my understanding >>> could be way out of date. Sorry if this is a stupid question; I'm >>> hoping PVFS might fill some requirements I'm looking at. >>> >>> Is it typical to use compute nodes as PVFS file servers? Is it even >>> possible (i.e. it isn't w/ Lustre)? >>> >>> If so, while striping across I/O nodes looks to be the default, is >>> there any provision for file affinity... i.e. files migrating to the >>> nodes using them? >>> >>> If striping is the only choice, and compute nodes serve the I/O too, >>> then a job running on one set of nodes could be using files striped >>> across another set of nodes in use by another job, causing both jobs >>> to loose performance, correct? >>> >>> Are there any links to PVFS performance over IB? >>> >>> Thanks, >>> >>> Chris >> >> _______________________________________________ >> Pvfs2-users mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
