Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Fri, 2016-12-16 at 09:52 +0100, Robert Sander wrote: > On 15.12.2016 16:49, Bjoern Laessig wrote: > > > What does your Cluster do? Where is your data. What happens now? > > You could configure the interfaces between the nodes as pointopoint > links and run OSPF on them. The cluster nodes then would have their > node IP on a dummy interface. OSPF would sort out the routing. yes, this is worth a try! Thanks. Bird should be suitable for me. Björn ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On 15.12.2016 16:49, Bjoern Laessig wrote: > What does your Cluster do? Where is your data. What happens now? You could configure the interfaces between the nodes as pointopoint links and run OSPF on them. The cluster nodes then would have their node IP on a dummy interface. OSPF would sort out the routing. If a link between two nodes go down the traffic is routed via the third node. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Do, 2016-12-15 at 14:31 +0100, ulem...@polarzone.de wrote: > Hi Björn, > i think he use something like this: > http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server i looked into it. This was the first idea for a config i throw away. Imagine you have 3 Systems. A,B,C Your cable between B and C is faulty. -> for A: everything is working. for B: Server A is up, Server C is down for C: Server A is up, Server B is down What does your Cluster do? Where is your data. What happens now? At the moment i think of a way ebtables preventing broadcast loops. Testing it soon. kind regards, Björn Lässig ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Hi Björn, i think he use something like this: http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server Udo Am 2016-12-15 11:10, schrieb Bjoern Laessig: On Do, 2016-10-27 at 15:47 +0200, mj wrote: Hi Jelle, On 10/27/2016 03:04 PM, Jelle de Jong wrote: > Hello everybody, > > I want to upgrade my small ceph cluster to 10Gbit networking and would > like some recommendation from your experience. > > What is your recommend budget 10Gbit switch suitable for Ceph? We are running a 3-node cluster, with _direct_ 10G cable connections (quasi crosslink) between the three hosts. This is very low-budget, as it gives you 10G speed, without a (relatively) expensive 10G switch. Working fine here, with each host having a double 10G intel nic, plus a regular 1G interface. Could you please send me details how you configured your network? I am planning this configuration too, but didn't test it yet. My special interest is how you prevented ethernet loops und what did you changed on the default ceph settings? Thanks a lot, Björn Lässig ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Do, 2016-10-27 at 15:47 +0200, mj wrote: > Hi Jelle, > > On 10/27/2016 03:04 PM, Jelle de Jong wrote: > > Hello everybody, > > > > I want to upgrade my small ceph cluster to 10Gbit networking and would > > like some recommendation from your experience. > > > > What is your recommend budget 10Gbit switch suitable for Ceph? > > We are running a 3-node cluster, with _direct_ 10G cable connections > (quasi crosslink) between the three hosts. This is very low-budget, as > it gives you 10G speed, without a (relatively) expensive 10G switch. > > Working fine here, with each host having a double 10G intel nic, plus a > regular 1G interface. Could you please send me details how you configured your network? I am planning this configuration too, but didn't test it yet. My special interest is how you prevented ethernet loops und what did you changed on the default ceph settings? Thanks a lot, Björn Lässig ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
>>We use Mellanox SX1036 and SX1012, which can function in 10 and 56GbE modes. >>It uses QSFP, Twinax or MPO, which terminates with LC fiber connections. >>While not dirt cheap, or entry >>level, we like these as being considerably >>cheaper than even a decent SDN solution. We have been able to build MLAG and >>leaf and spine solutions pretty easily with these. We use sx1012 too , pretty happy with them (in production since 2 years now). We are going to use new SN2100 (16 ports 40GB or 16 ports 100GB, with breakout cables that's 48 ports 10G or 48ports 25G). around 6000€ for the 40GB, and 12000€ with 100GB. (with mlx-os, but you can use also cumulus with theses new mellanox switches :) - Mail original - De: "Simon Leinen" À: "Erik McCormick" Cc: "ceph-users" Envoyé: Lundi 31 Octobre 2016 11:28:07 Objet: Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade Erik McCormick writes: > We use Edge-Core 5712-54x running Cumulus Linux. Anything off their > compatibility list would be good though. The switch is 48 10G sfp+ > ports. We just use copper cables with attached sfp. It also had 6 40G > ports. The switch cost around $4800 and the cumulus license is about > 3k for a perpetual license. Similar here, except we use Quanta switches (T5032-LY6). SFP+ slots and DAC cables. Actually our switches are 32*40GE, and we use "fan-out" DAC cables (QSFP on one side, 4 SFP+ on the other). Compared to 10GBaseT (RJ45), DAC cables are thicker, which may complicate cable management a little. On the other hand I think DAC still needs less power than 10GBaseT. And with the 40G setup, we have good port density and a smooth migration path to 40GE. We already use 40GE for our leaf-spine uplinks. Another advantage for us is that we can use a single SKU for both leaf and spine switches. The Cumulus licenses are a bit more expensive for those 40GE switches (as are the switches themselves), but it's still a good deal for us. Maybe these days it makes sense to look at 100GE switches in preference to 40GE; 100GE ports can normally be used as 2*50GE, 4*25GE, 1*40GE or 4*10GE as well, so the upgrade paths seem even nicer. And the prices are getting competitive I think. -- Simon. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Erik McCormick writes: > We use Edge-Core 5712-54x running Cumulus Linux. Anything off their > compatibility list would be good though. The switch is 48 10G sfp+ > ports. We just use copper cables with attached sfp. It also had 6 40G > ports. The switch cost around $4800 and the cumulus license is about > 3k for a perpetual license. Similar here, except we use Quanta switches (T5032-LY6). SFP+ slots and DAC cables. Actually our switches are 32*40GE, and we use "fan-out" DAC cables (QSFP on one side, 4 SFP+ on the other). Compared to 10GBaseT (RJ45), DAC cables are thicker, which may complicate cable management a little. On the other hand I think DAC still needs less power than 10GBaseT. And with the 40G setup, we have good port density and a smooth migration path to 40GE. We already use 40GE for our leaf-spine uplinks. Another advantage for us is that we can use a single SKU for both leaf and spine switches. The Cumulus licenses are a bit more expensive for those 40GE switches (as are the switches themselves), but it's still a good deal for us. Maybe these days it makes sense to look at 100GE switches in preference to 40GE; 100GE ports can normally be used as 2*50GE, 4*25GE, 1*40GE or 4*10GE as well, so the upgrade paths seem even nicer. And the prices are getting competitive I think. -- Simon. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Thursday, October 27, 2016, Jelle de Jong wrote: > Hello everybody, > > I want to upgrade my small ceph cluster to 10Gbit networking and would > like some recommendation from your experience. > > What is your recommend budget 10Gbit switch suitable for Ceph? We use Mellanox SX1036 and SX1012, which can function in 10 and 56GbE modes. It uses QSFP, Twinax or MPO, which terminates with LC fiber connections. While not dirt cheap, or entry level, we like these as being considerably cheaper than even a decent SDN solution. We have been able to build MLAG and leaf and spine solutions pretty easily with these. > > I would like to use X550-T1 intel adapters in my nodes. > > Or is fibre recommended? > X520-DA2 > X520-SR1 > > Kind regards, > > Jelle de Jong > GNU/Linux Consultant > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- -- Alex Gorbachev Storcium ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Hi Jelle, we do Arista 7050TX 10GbE Switching as TORs. We cluster them with VARP HA, active-active which works very well here. The host nics are X540 copper. Interfaces are all 2x10G LACP bonds. While some may argue Fiber is nice, and will lower latency expecially for longer distances, for us copper is more practical and cheaper too. thx will On Thu, Oct 27, 2016 at 9:04 PM, Jelle de Jong wrote: > Hello everybody, > > I want to upgrade my small ceph cluster to 10Gbit networking and would like > some recommendation from your experience. > > What is your recommend budget 10Gbit switch suitable for Ceph? > > I would like to use X550-T1 intel adapters in my nodes. > > Or is fibre recommended? > X520-DA2 > X520-SR1 > > Kind regards, > > Jelle de Jong > GNU/Linux Consultant > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Oct 27, 2016 3:16 PM, "Oliver Dzombic" wrote: > > Hi, > > i can recommand > > X710-DA2 > Weer also use this nice for everything. > 10G Switch is going over our bladeinfrastructure, so i can't recommand > something for you there. > > I assume that the usual juniper/cisco will do a good job. I think, for > ceph, the switch is not the major point of a setup. > We use Edge-Core 5712-54x running Cumulus Linux. Anything off their compatibility list would be good though. The switch is 48 10G sfp+ ports. We just use copper cables with attached sfp. It also had 6 40G ports. The switch cost around $4800 and the cumulus license is about 3k for a perpetual license. > -- > Mit freundlichen Gruessen / Best regards > > Oliver Dzombic > IP-Interactive > > mailto:i...@ip-interactive.de > > Anschrift: > > IP Interactive UG ( haftungsbeschraenkt ) > Zum Sonnenberg 1-3 > 63571 Gelnhausen > > HRB 93402 beim Amtsgericht Hanau > Geschäftsführung: Oliver Dzombic > > Steuer Nr.: 35 236 3622 1 > UST ID: DE274086107 > > > Am 27.10.2016 um 15:04 schrieb Jelle de Jong: > > Hello everybody, > > > > I want to upgrade my small ceph cluster to 10Gbit networking and would > > like some recommendation from your experience. > > > > What is your recommend budget 10Gbit switch suitable for Ceph? > > > > I would like to use X550-T1 intel adapters in my nodes. > > > > Or is fibre recommended? > > X520-DA2 > > X520-SR1 > > > > Kind regards, > > > > Jelle de Jong > > GNU/Linux Consultant > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -Erik ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Thu, Oct 27, 2016 at 9:59 AM, Erick Perez - Quadrian Enterprises < epe...@quadrianweb.com> wrote: > > > On Thu, Oct 27, 2016 at 9:19 AM, Richard Chan < > rich...@treeboxsolutions.com> wrote: > >> Dell N40xx is cheap if you buy the RJ-45 versions.It will save a lot on >> optics. >> Our NICs are all RJ-45. We use CAT7 and the switch hasn't given any >> problems. >> >> We have one with an SFP+ uplink, and used a Dell-branded DAC cable that >> has also worked flawlessly. >> >> (Not to be confused with Dell S-series which are is much more expensive.) >> >> You do want to be careful with SFP+: many(all?) vendors create expensive >> lock-in by requiring their house brand optics: 48 x SFP+ can easily exceed >> the cost of the switch by several times. The switch port will not come up >> if you use aftermarket optics. >> >> >> - Richard >> >> >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > I have two options for my networks. > A budget-wise switch Netgear XS716T, > and a not so budget wise switch HPE 5700 (around 8 thousand if you push > your sales rep hard enough) > -- > > - > Erick Perez > > So sorry, I meant Netgear XS728T. In amazon is listed as Netgear ProSAFE XS728T 28-Port 10-Gigabit Ethernet Smart Managed Switch (XS728T-100NES) for 2thousand dollars. -- - Erick Perez ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
On Thu, Oct 27, 2016 at 9:19 AM, Richard Chan wrote: > Dell N40xx is cheap if you buy the RJ-45 versions.It will save a lot on > optics. > Our NICs are all RJ-45. We use CAT7 and the switch hasn't given any > problems. > > We have one with an SFP+ uplink, and used a Dell-branded DAC cable that > has also worked flawlessly. > > (Not to be confused with Dell S-series which are is much more expensive.) > > You do want to be careful with SFP+: many(all?) vendors create expensive > lock-in by requiring their house brand optics: 48 x SFP+ can easily exceed > the cost of the switch by several times. The switch port will not come up > if you use aftermarket optics. > > > - Richard > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > I have two options for my networks. A budget-wise switch Netgear XS716T, and a not so budget wise switch HPE 5700 (around 8 thousand if you push your sales rep hard enough) -- - Erick Perez ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Dell N40xx is cheap if you buy the RJ-45 versions.It will save a lot on optics. Our NICs are all RJ-45. We use CAT7 and the switch hasn't given any problems. We have one with an SFP+ uplink, and used a Dell-branded DAC cable that has also worked flawlessly. (Not to be confused with Dell S-series which are is much more expensive.) You do want to be careful with SFP+: many(all?) vendors create expensive lock-in by requiring their house brand optics: 48 x SFP+ can easily exceed the cost of the switch by several times. The switch port will not come up if you use aftermarket optics. - Richard ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Hi Jelle, On 10/27/2016 03:04 PM, Jelle de Jong wrote: Hello everybody, I want to upgrade my small ceph cluster to 10Gbit networking and would like some recommendation from your experience. What is your recommend budget 10Gbit switch suitable for Ceph? We are running a 3-node cluster, with _direct_ 10G cable connections (quasi crosslink) between the three hosts. This is very low-budget, as it gives you 10G speed, without a (relatively) expensive 10G switch. Working fine here, with each host having a double 10G intel nic, plus a regular 1G interface. MJ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Arista has been good to us. Netgear or eBay if you have to scrape bottom. Rob Sent from my iPhone > On Oct 27, 2016, at 9:16 AM, Oliver Dzombic wrote: > > Hi, > > i can recommand > > X710-DA2 > > 10G Switch is going over our bladeinfrastructure, so i can't recommand > something for you there. > > I assume that the usual juniper/cisco will do a good job. I think, for > ceph, the switch is not the major point of a setup. > > -- > Mit freundlichen Gruessen / Best regards > > Oliver Dzombic > IP-Interactive > > mailto:i...@ip-interactive.de > > Anschrift: > > IP Interactive UG ( haftungsbeschraenkt ) > Zum Sonnenberg 1-3 > 63571 Gelnhausen > > HRB 93402 beim Amtsgericht Hanau > Geschäftsführung: Oliver Dzombic > > Steuer Nr.: 35 236 3622 1 > UST ID: DE274086107 > > >> Am 27.10.2016 um 15:04 schrieb Jelle de Jong: >> Hello everybody, >> >> I want to upgrade my small ceph cluster to 10Gbit networking and would >> like some recommendation from your experience. >> >> What is your recommend budget 10Gbit switch suitable for Ceph? >> >> I would like to use X550-T1 intel adapters in my nodes. >> >> Or is fibre recommended? >> X520-DA2 >> X520-SR1 >> >> Kind regards, >> >> Jelle de Jong >> GNU/Linux Consultant >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Oliver Dzombic > Sent: 27 October 2016 14:16 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade > > Hi, > > i can recommand > > X710-DA2 > > 10G Switch is going over our bladeinfrastructure, so i can't recommand > something for you there. > > I assume that the usual juniper/cisco will do a good job. I think, for ceph, > the switch is not the major point of a setup. > > -- > Mit freundlichen Gruessen / Best regards > > Oliver Dzombic > IP-Interactive > > mailto:i...@ip-interactive.de > > Anschrift: > > IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3 > 63571 Gelnhausen > > HRB 93402 beim Amtsgericht Hanau > Geschäftsführung: Oliver Dzombic > > Steuer Nr.: 35 236 3622 1 > UST ID: DE274086107 > > > Am 27.10.2016 um 15:04 schrieb Jelle de Jong: > > Hello everybody, > > > > I want to upgrade my small ceph cluster to 10Gbit networking and would > > like some recommendation from your experience. > > > > What is your recommend budget 10Gbit switch suitable for Ceph? > > > > I would like to use X550-T1 intel adapters in my nodes. > > > > Or is fibre recommended? > > X520-DA2 > > X520-SR1 > > > > Kind regards, > > > > Jelle de Jong > > GNU/Linux Consultant I personally like 10G-T for TOR switching as it simplifies cabling and all the servers we buy now have these ports on-board. We currently use Dell N4000 as we got them for a very good price and are fine for basic L2/L3 networking. However, we are looking to probably replace them in the near future with Cumulus Linux switches to gain advantages around OSPF/BGP on OSD nodes and better manageability. > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Hi, i can recommand X710-DA2 10G Switch is going over our bladeinfrastructure, so i can't recommand something for you there. I assume that the usual juniper/cisco will do a good job. I think, for ceph, the switch is not the major point of a setup. -- Mit freundlichen Gruessen / Best regards Oliver Dzombic IP-Interactive mailto:i...@ip-interactive.de Anschrift: IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3 63571 Gelnhausen HRB 93402 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic Steuer Nr.: 35 236 3622 1 UST ID: DE274086107 Am 27.10.2016 um 15:04 schrieb Jelle de Jong: > Hello everybody, > > I want to upgrade my small ceph cluster to 10Gbit networking and would > like some recommendation from your experience. > > What is your recommend budget 10Gbit switch suitable for Ceph? > > I would like to use X550-T1 intel adapters in my nodes. > > Or is fibre recommended? > X520-DA2 > X520-SR1 > > Kind regards, > > Jelle de Jong > GNU/Linux Consultant > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 10Gbit switch advice for small ceph cluster upgrade
Hello everybody, I want to upgrade my small ceph cluster to 10Gbit networking and would like some recommendation from your experience. What is your recommend budget 10Gbit switch suitable for Ceph? I would like to use X550-T1 intel adapters in my nodes. Or is fibre recommended? X520-DA2 X520-SR1 Kind regards, Jelle de Jong GNU/Linux Consultant ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
