On 05/26/2011 03:29 PM, Greg Keller wrote: > Agreed. Just finished telling another vendor, "It's not high speed > storage unless it has an IB/RDMA interface". They love that. Except
Heh ... love it! > for some really edge cases, I can't imagine running IO over GbE for > anything more than trivial IO loads. Lots of our customers do, when they have a large legacy GbE network, and upgrading is expensive. We can have a very large fan in to our units, but IB (even SDR!) is really nice to move data over for storage. > I am Curious if anyone is doing IO over IB to SRP targets or some > similar "Block Device" approach. The Integration into the filesystem by Both block and file targets. SRPT on our units, and fronted by OSSes for Lustre and similar like things. Can do iSCSI as well (over IB using iSER, or over 10GbE ... works really nicely in either case). > Lustre/GPFS and others may be the best way to go, but we are not 100% > convinced yet. Any stories to share? If you do this with Lustre, make sure your OSSes are in HA pairs using pacemaker/ucarp, and use DRBD between backend units, or MD on the OSS to mirror the storage. Unfortunately IB doesn't virtualize well (last I checked), so these have to be physical OSSes. I presume something similar on GPFS. GlusterFS, PVFS2/OrangeFS, etc. go fine without the block devices, and Gluster does mirroring at the file level. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: [email protected] web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
