[Gluster-users] working inifiniband HW ??
Dear List, I am currently testing glusterfs on a small scale non-production env with ordinary nics. I would like to purchase a couple of inifiniband nics in order to connect 3 servers in point to point mode that is with no switches in between. Since I've noticed that some of you have this kind of H/W Any info on brands/models and a good known to work setup will be highly appreciated. Regards, Harry ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] working inifiniband HW ??
On 08/07/2013, at 10:20 AM, HL wrote: I am currently testing glusterfs on a small scale non-production env with ordinary nics. I would like to purchase a couple of inifiniband nics in order to connect 3 servers in point to point mode that is with no switches in between. Since I've noticed that some of you have this kind of H/W Any info on brands/models and a good known to work setup will be highly appreciated. Depends on the kind of performance you're after. :) If you're just wanting something better than 10GbE at minimal cost, these work fine under Linux: http://www.ebay.co.uk/itm/360657396651 ~US$40 each, not counting postage They're rebadged Mellanox MHGH28-XTC cards. Completely ok to be flashed with standard Mellanox firmware. For cables, I use these: http://www.ebay.co.uk/itm/251200441924 US$14 each, not counting postage Note, I'm comfortable getting stuff off eBay where the seller seems ok. So far, so good. ;) If you do decide to use the above stuff, and later on want a switch, try and find a Voltaire ISR 9024D-M. They're DDR infiniband (20Gb/s per port), and can run fanless (completely silent). http://www.ebay.co.uk/itm/350827346490 They can be had pretty cheaply if you're willing to wait for good pricing. I got mine for ~US$270. Hope that helps. :) Regards and best wishes, Justin Clift Regards, Harry ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Open Source and Standards @ Red Hat twitter.com/realjustinclift ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] working inifiniband HW ??
I concur - I used the same cards on our test cluster - just make sure you upgrade the firmware to the latest revision: http://www.mellanox.com/content/pages.php?pg=custom_firmware_table The fw upgrade is not trivial - well, it IS trivial, but bringing all the info together was not - if you want some rough notes, let me know. the other harry On Mon, Jul 8, 2013 at 5:23 AM, Justin Clift jcl...@redhat.com wrote: On 08/07/2013, at 10:20 AM, HL wrote: I am currently testing glusterfs on a small scale non-production env with ordinary nics. I would like to purchase a couple of inifiniband nics in order to connect 3 servers in point to point mode that is with no switches in between. Since I've noticed that some of you have this kind of H/W Any info on brands/models and a good known to work setup will be highly appreciated. Depends on the kind of performance you're after. :) If you're just wanting something better than 10GbE at minimal cost, these work fine under Linux: http://www.ebay.co.uk/itm/360657396651 ~US$40 each, not counting postage They're rebadged Mellanox MHGH28-XTC cards. Completely ok to be flashed with standard Mellanox firmware. For cables, I use these: http://www.ebay.co.uk/itm/251200441924 US$14 each, not counting postage Note, I'm comfortable getting stuff off eBay where the seller seems ok. So far, so good. ;) If you do decide to use the above stuff, and later on want a switch, try and find a Voltaire ISR 9024D-M. They're DDR infiniband (20Gb/s per port), and can run fanless (completely silent). http://www.ebay.co.uk/itm/350827346490 They can be had pretty cheaply if you're willing to wait for good pricing. I got mine for ~US$270. Hope that helps. :) Regards and best wishes, Justin Clift Regards, Harry ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Open Source and Standards @ Red Hat twitter.com/realjustinclift ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Data backup by the NSA - your tax dollars at work. -- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] working inifiniband HW ??
On 08/07/2013, at 6:02 PM, Harry Mangalam wrote: I concur - I used the same cards on our test cluster - just make sure you upgrade the firmware to the latest revision: http://www.mellanox.com/content/pages.php?pg=custom_firmware_table The fw upgrade is not trivial - well, it IS trivial, but bringing all the info together was not - if you want some rough notes, let me know. Nowdays there should be copious info on the new Mellanox Community website (recently launched): http://community.mellanox.com Saying that because I've written out the step-by-step instructions for people on the forums there a bunch of times over the last month or two. :) Searching for flint or mstflint should pull up appropriate stuff for anyone else who needs it. Hopefully useful. :) Regards and best wishes, Justin Clift the other harry -- Open Source and Standards @ Red Hat twitter.com/realjustinclift ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
