IIRC, one thing to look out for is that there are two ways to do IP over Infiniband. You can either do IP over Infiniband directly (IPoIB), or encapsulate Ethernet in Infiniband (EoIB), and then do IP over the fake Ethernet network.
IPoIB is more common, but I'd assume that IB<->Ethernet bridges really only bridge EoIB. On Tue Oct 07 2014 at 5:34:57 PM Christian Balzer <ch...@gol.com> wrote: > On Tue, 07 Oct 2014 20:40:31 +0000 Scott Laird wrote: > > > I've done this two ways in the past. Either I'll give each machine an > > Infiniband network link and a 1000baseT link and use the Infiniband one > > as the private network for Ceph, or I'll throw an Infiniband card into a > > PC and run something like Vyatta/VyOS on it and make it a router, so IP > > traffic can get out of the IB network. Of course, those have both been > > for test labs. YMMV. > > > > That. > > Of course in a production environment you would want something with 2 > routers in a failover configuration. > And there are switches/gateways that combine IB and Ethernet, but they > tend to be not so cheap. ^^ > > More below. > > > On Tue Oct 07 2014 at 11:05:23 AM Massimiliano Cuttini > > <m...@phoenixweb.it> wrote: > > > > > Hi Christian, > > > > > > When you say "10 gig infiniband", do you mean QDRx4 Infiniband > > > (usually flogged as 40Gb/s even though it is 32Gb/s, but who's > > > counting), which tends to be the same basic hardware as the 10Gb/s > > > Ethernet offerings from Mellanox? > > > > > > A brand new 18 port switch of that caliber will only cost about 180$ > > > per port, too. > > > > > > > > > > > > I investigate about infiniband but i didn't found affordable prices at > > > all. > > Then you're doing it wrong or comparing apples to oranges (you of course > need to compare IB switches to similar 10GbE ones). > And the prices of HCA (aka network cards in the servers) and cabling. > > > > Moreover how do you connect your *l**egacy node servers* to your > > > *brand new storages* if you have Infiniband only on storages & > > > switches? Is there any mixed switch that allow you both to connect > > > with Infiniband and Ethernet? > > > > > > If there is, please send specs because i cannot find just by google it. > > > > The moment you type in "infiniband et" google will already predict amongst > other pertinent things "infiniband ethernet gateway" and "infiniband > ethernet bridge". > But even "infiniband ethernet switch" has a link telling you pretty much > what was said here now at the 6th position: > http://www.tomshardware.com/forum/44997-42-connect- > infiniband-switch-ethernet > > Christian > > > Thanks, > > > Max > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Fusion Communications > http://www.gol.com/ > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
