Re: [casper] Network Switches
hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) i think jason has also tested this switch. best wishes, dan On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: > Hi all. Has anyone tested the Mellanox SX-1024 series of switches with > ROACH-2 and 10 and/or 40 gb NICs? > > These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. > > Thanks! > > John > > > >
Re: [casper] Network Switches
> hi john, > > jack hickish tested a mellanox SX1012 > (12x40Gbe, or 48x10Gbe, or mixture), > when he was visiting berkeley. > > the SX1012 worked beautifully on roach2, after jack upgraded > the switch firmware. and it's a great price. ($6K) Yeah, we're looking at a 1024, or a 1036. They all use the same silicon. Thanks! I'll give it a try and let you all know how it works out. John > > i think jason has also tested this switch. > > best wishes, > > dan > > > > > On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: > >> Hi all. Has anyone tested the Mellanox SX-1024 series of switches with >> ROACH-2 and 10 and/or 40 gb NICs? >> >> These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. >> >> Thanks! >> >> John >> >> >> >> >
Re: [casper] Network Switches
Hi John, As Dan said I've tested (to some extent) the SX1012. I just used one ROACH2 and corner turned data through 8 x 10GbE ports. It worked well basically up to line rate, with no CRC errors after a few hours of operation, but only after - Jason put me in touch with some Mellanox guys who provided updated firmware - using branded Mellanox 3m QSFP -> 4xSFP cables Before the upgrade transmissions from the switch to the ROACH2 would frequently fail. Of course this may or may not be remotely relevant for the SX1024 :) Good luck! J On 2 September 2014 16:46, Dan Werthimer wrote: > > hi john, > > jack hickish tested a mellanox SX1012 > (12x40Gbe, or 48x10Gbe, or mixture), > when he was visiting berkeley. > > the SX1012 worked beautifully on roach2, after jack upgraded > the switch firmware. and it's a great price. ($6K) > > i think jason has also tested this switch. > > best wishes, > > dan > > > > > On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: >> >> Hi all. Has anyone tested the Mellanox SX-1024 series of switches with >> ROACH-2 and 10 and/or 40 gb NICs? >> >> These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. >> >> Thanks! >> >> John >> >> >> >
Re: [casper] Network Switches
Hi John, LEDA uses the SX1024 w/ 40GbE MCX313A-BCCT and 10GbE ROACH2 daughter cards. We use 11 40GbE ports (21.4 Gb/s) and 32 10 GbE ports (7.4 Gb/s). Half duplex traffic is ~ 235 Gb/s sustained. Routing setup took << 1 day. Management has required nearly zero time in the last 12 months. Our tests have included higher rates on individual ports, maxing out the 10 and 40 GbE ports at their line rates, but the aggregate during tests was 'small.' We obtained an academic discount to around $11K if I remember correctly. If you would like to learn specifics beyond the above, please speak to Jonathon Kocz. He is the correlator's father. Best, Lincoln On 9/2/14, 11:46 AM, Dan Werthimer wrote: hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) i think jason has also tested this switch. best wishes, dan On Tue, Sep 2, 2014 at 8:36 AM, John Ford mailto:jf...@nrao.edu>> wrote: Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John -- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX:1 617-495-7345 greenh...@cfa.harvard.edu Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln
Re: [casper] Network Switches
Hi John, Email is passing in the ether. I have eyed the SX1035 and SX1036 w/ upgrades in mind. What is your understanding of the difference? Lincoln On 9/2/14, 11:54 AM, John Ford wrote: hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) Yeah, we're looking at a 1024, or a 1036. They all use the same silicon. Thanks! I'll give it a try and let you all know how it works out. John i think jason has also tested this switch. best wishes, dan On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John -- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX:1 617-495-7345 greenh...@cfa.harvard.edu Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln
Re: [casper] Network Switches
Hi Jack (and John), Interesting report. Difficult to puzzle out differences in our mutual experience unless the 1012 is unlike the 1024 in subtle ways. We (LEDA) did not require an upgrade or Mellanox cables for the 10/40GbE links. Again - I would look forward to discussion people may have with Jonathon (Cc me if not conducted via the Casper list). Best, Lincoln On 9/2/14, 11:54 AM, Jack Hickish wrote: Hi John, As Dan said I've tested (to some extent) the SX1012. I just used one ROACH2 and corner turned data through 8 x 10GbE ports. It worked well basically up to line rate, with no CRC errors after a few hours of operation, but only after - Jason put me in touch with some Mellanox guys who provided updated firmware - using branded Mellanox 3m QSFP -> 4xSFP cables Before the upgrade transmissions from the switch to the ROACH2 would frequently fail. Of course this may or may not be remotely relevant for the SX1024 :) Good luck! J On 2 September 2014 16:46, Dan Werthimer wrote: hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) i think jason has also tested this switch. best wishes, dan On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John -- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX:1 617-495-7345 greenh...@cfa.harvard.edu Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln
Re: [casper] Network Switches
I found before the firmware upgrade the problems were very hardware dependent. Eg, all ports would have errors with mellanox 3m cables, whilst with shorter (2m, non-mellanox) cables, most ROACH2 ports would work, though some would have occassional errors. After the firmware upgrade, the mellanox 3m cables worked everywhere, but the unbranded 2m still gave errors in certain ports. As far as I could tell, (though I didn't pursue this as vigorously as I could, since Jason's report was that the switch wouldn't work straight out the box) there were no problems when sending ROACH2->switch, only the other way round. I'm trying to buy an SX1012 in the UK at the moment, and will need to test again with higher traffic rates. I'll let the list know how things go... Jack On 2 September 2014 17:01, Lincoln Greenhill wrote: > Hi Jack (and John), > > Interesting report. Difficult to puzzle out > differences in our mutual experience unless the > 1012 is unlike the 1024 in subtle ways. > > We (LEDA) did not require an upgrade or Mellanox cables > for the 10/40GbE links. Again - I would look > forward to discussion people may have with Jonathon > (Cc me if not conducted via the Casper list). > > Best, > Lincoln > > > On 9/2/14, 11:54 AM, Jack Hickish wrote: >> >> Hi John, >> >> As Dan said I've tested (to some extent) the SX1012. I just used one >> ROACH2 and corner >> turned data through 8 x 10GbE ports. It worked well basically up to >> line rate, with no CRC errors after a few hours of operation, but only >> after >> >> - Jason put me in touch with some Mellanox guys who provided updated >> firmware >> - using branded Mellanox 3m QSFP -> 4xSFP cables >> >> Before the upgrade transmissions from the switch to the ROACH2 would >> frequently fail. >> >> Of course this may or may not be remotely relevant for the SX1024 :) >> >> Good luck! >> >> J >> >> On 2 September 2014 16:46, Dan Werthimer wrote: >>> >>> >>> hi john, >>> >>> jack hickish tested a mellanox SX1012 >>> (12x40Gbe, or 48x10Gbe, or mixture), >>> when he was visiting berkeley. >>> >>> the SX1012 worked beautifully on roach2, after jack upgraded >>> the switch firmware. and it's a great price. ($6K) >>> >>> i think jason has also tested this switch. >>> >>> best wishes, >>> >>> dan >>> >>> >>> >>> >>> On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John >>> >> > > -- > Lincoln J. Greenhill Harvard-Smithsonian CfA > Office: 1 617-495-7194 60 Garden St, Mail Stop 42 > Cell: 1 650 722-7798 Cambridge, MA 02138 > FAX:1 617-495-7345 greenh...@cfa.harvard.edu > Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln
Re: [casper] Network Switches
The SX1012 and SX1036 definitely use the same ASIC series and both suffer from the PHY link problem. I've got seven of them here, and tested them with 20 different ROACH-2 boards. As Jack points out, some links are worse than others and swapping cables and ROACH2 mezzanine cards often sorts the problem out. To be honest, the issue is probably on the ROACH-2 side, as we haven't tuned the PHY's RX parameters. Mellanox claim to have tested and complied to IEEE standards. To get around this, Mellanox have a "patch" for CASPER users, which adjusts the TX drive on the switch so that a bigger signal hits the ROACH2. This fixes all the problems, and the links are then very very reliable. They actually borrowed a ROACH-2 for the purpose of testing and tuning their PHYs. Very kind of them! This tuning was done using their own 3m cables, so you should expect the best performance using those copper links. Another option is to use AOC cables, which are cost-effective and loss-less. Lincoln, not to alarm you, but are you actually checking for bit errors on your SX1024 links? I suspect you're just not noticing any faults. Packets don't always get dropped... oftentimes a few bits in the data is just corrupted, and if you're looking at correlator noise, you might not even notice it! ...the "RX error" line from the 10G core is hard-coded to zero, so if this is what you're watching, you'd never know. Likewise the ROACH2->switch links work fine, it's only the switch->ROACH2 direction that's problematic, so you'd never see the error counters on the switch climbing. We (SKA-SA) add checksums to our packets to be sure. Mostly, we were seeing BER numbers like 10e-10 on most of the Mellanox kit out-the-box, but a few links had bad links with BER ~10E-6. After patching, it's better than 10e-13. Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 02 Sep 2014, at 18:01, Lincoln Greenhill wrote: > Hi Jack (and John), > > Interesting report. Difficult to puzzle out > differences in our mutual experience unless the > 1012 is unlike the 1024 in subtle ways. > > We (LEDA) did not require an upgrade or Mellanox cables > for the 10/40GbE links. Again - I would look > forward to discussion people may have with Jonathon > (Cc me if not conducted via the Casper list). > > Best, > Lincoln > > On 9/2/14, 11:54 AM, Jack Hickish wrote: >> Hi John, >> >> As Dan said I've tested (to some extent) the SX1012. I just used one >> ROACH2 and corner >> turned data through 8 x 10GbE ports. It worked well basically up to >> line rate, with no CRC errors after a few hours of operation, but only >> after >> >> - Jason put me in touch with some Mellanox guys who provided updated firmware >> - using branded Mellanox 3m QSFP -> 4xSFP cables >> >> Before the upgrade transmissions from the switch to the ROACH2 would >> frequently fail. >> >> Of course this may or may not be remotely relevant for the SX1024 :) >> >> Good luck! >> >> J >> >> On 2 September 2014 16:46, Dan Werthimer wrote: >>> >>> hi john, >>> >>> jack hickish tested a mellanox SX1012 >>> (12x40Gbe, or 48x10Gbe, or mixture), >>> when he was visiting berkeley. >>> >>> the SX1012 worked beautifully on roach2, after jack upgraded >>> the switch firmware. and it's a great price. ($6K) >>> >>> i think jason has also tested this switch. >>> >>> best wishes, >>> >>> dan >>> >>> >>> >>> >>> On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John >>> >> > > -- > Lincoln J. Greenhill Harvard-Smithsonian CfA > Office: 1 617-495-7194 60 Garden St, Mail Stop 42 > Cell: 1 650 722-7798 Cambridge, MA 02138 > FAX:1 617-495-7345 greenh...@cfa.harvard.edu > Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln >
Re: [casper] Network Switches
hi jason, thanks for all this info. in the leda correlator, the 10/40Gbe links are unidirectional, roach2-->switch and switch-->gpuserver so i don't think leda would encounter the switch-->roach2 bit errors, although it wouldn't hurt to upgrade the firmware. best wishes, dan On Tue, Sep 2, 2014 at 9:33 AM, Jason Manley wrote: > The SX1012 and SX1036 definitely use the same ASIC series and both suffer > from the PHY link problem. I've got seven of them here, and tested them > with 20 different ROACH-2 boards. As Jack points out, some links are worse > than others and swapping cables and ROACH2 mezzanine cards often sorts the > problem out. To be honest, the issue is probably on the ROACH-2 side, as we > haven't tuned the PHY's RX parameters. Mellanox claim to have tested and > complied to IEEE standards. > > To get around this, Mellanox have a "patch" for CASPER users, which > adjusts the TX drive on the switch so that a bigger signal hits the ROACH2. > This fixes all the problems, and the links are then very very reliable. > They actually borrowed a ROACH-2 for the purpose of testing and tuning > their PHYs. Very kind of them! This tuning was done using their own 3m > cables, so you should expect the best performance using those copper links. > Another option is to use AOC cables, which are cost-effective and loss-less. > > Lincoln, not to alarm you, but are you actually checking for bit errors on > your SX1024 links? I suspect you're just not noticing any faults. Packets > don't always get dropped... oftentimes a few bits in the data is just > corrupted, and if you're looking at correlator noise, you might not even > notice it! ...the "RX error" line from the 10G core is hard-coded to zero, > so if this is what you're watching, you'd never know. Likewise the > ROACH2->switch links work fine, it's only the switch->ROACH2 direction > that's problematic, so you'd never see the error counters on the switch > climbing. We (SKA-SA) add checksums to our packets to be sure. Mostly, we > were seeing BER numbers like 10e-10 on most of the Mellanox kit > out-the-box, but a few links had bad links with BER ~10E-6. After patching, > it's better than 10e-13. > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 02 Sep 2014, at 18:01, Lincoln Greenhill > wrote: > > > Hi Jack (and John), > > > > Interesting report. Difficult to puzzle out > > differences in our mutual experience unless the > > 1012 is unlike the 1024 in subtle ways. > > > > We (LEDA) did not require an upgrade or Mellanox cables > > for the 10/40GbE links. Again - I would look > > forward to discussion people may have with Jonathon > > (Cc me if not conducted via the Casper list). > > > > Best, > > Lincoln > > > > On 9/2/14, 11:54 AM, Jack Hickish wrote: > >> Hi John, > >> > >> As Dan said I've tested (to some extent) the SX1012. I just used one > >> ROACH2 and corner > >> turned data through 8 x 10GbE ports. It worked well basically up to > >> line rate, with no CRC errors after a few hours of operation, but only > >> after > >> > >> - Jason put me in touch with some Mellanox guys who provided updated > firmware > >> - using branded Mellanox 3m QSFP -> 4xSFP cables > >> > >> Before the upgrade transmissions from the switch to the ROACH2 would > >> frequently fail. > >> > >> Of course this may or may not be remotely relevant for the SX1024 :) > >> > >> Good luck! > >> > >> J > >> > >> On 2 September 2014 16:46, Dan Werthimer wrote: > >>> > >>> hi john, > >>> > >>> jack hickish tested a mellanox SX1012 > >>> (12x40Gbe, or 48x10Gbe, or mixture), > >>> when he was visiting berkeley. > >>> > >>> the SX1012 worked beautifully on roach2, after jack upgraded > >>> the switch firmware. and it's a great price. ($6K) > >>> > >>> i think jason has also tested this switch. > >>> > >>> best wishes, > >>> > >>> dan > >>> > >>> > >>> > >>> > >>> On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: > > Hi all. Has anyone tested the Mellanox SX-1024 series of switches > with > ROACH-2 and 10 and/or 40 gb NICs? > > These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. > > Thanks! > > John > > > > >>> > >> > > > > -- > > Lincoln J. Greenhill Harvard-Smithsonian CfA > > Office: 1 617-495-7194 60 Garden St, Mail Stop 42 > > Cell: 1 650 722-7798 Cambridge, MA 02138 > > FAX:1 617-495-7345 greenh...@cfa.harvard.edu > > Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln > > > >
Re: [casper] Network Switches
Not to steer this conversation too far off track, but does a patch exist to enable the ethernet CRC check within the 10Gb yellow block? When I tested the switch I used Dave MacMahon's CRC generator block, but it would be nice if there was the option to have the 10 gig core use the UDP crc -- or get rid of the port that pretends that it's doing this :) Cheers, J On 2 September 2014 17:33, Jason Manley wrote: > The SX1012 and SX1036 definitely use the same ASIC series and both suffer > from the PHY link problem. I've got seven of them here, and tested them with > 20 different ROACH-2 boards. As Jack points out, some links are worse than > others and swapping cables and ROACH2 mezzanine cards often sorts the problem > out. To be honest, the issue is probably on the ROACH-2 side, as we haven't > tuned the PHY's RX parameters. Mellanox claim to have tested and complied to > IEEE standards. > > To get around this, Mellanox have a "patch" for CASPER users, which adjusts > the TX drive on the switch so that a bigger signal hits the ROACH2. This > fixes all the problems, and the links are then very very reliable. They > actually borrowed a ROACH-2 for the purpose of testing and tuning their PHYs. > Very kind of them! This tuning was done using their own 3m cables, so you > should expect the best performance using those copper links. Another option > is to use AOC cables, which are cost-effective and loss-less. > > Lincoln, not to alarm you, but are you actually checking for bit errors on > your SX1024 links? I suspect you're just not noticing any faults. Packets > don't always get dropped... oftentimes a few bits in the data is just > corrupted, and if you're looking at correlator noise, you might not even > notice it! ...the "RX error" line from the 10G core is hard-coded to zero, so > if this is what you're watching, you'd never know. Likewise the > ROACH2->switch links work fine, it's only the switch->ROACH2 direction that's > problematic, so you'd never see the error counters on the switch climbing. We > (SKA-SA) add checksums to our packets to be sure. Mostly, we were seeing BER > numbers like 10e-10 on most of the Mellanox kit out-the-box, but a few links > had bad links with BER ~10E-6. After patching, it's better than 10e-13. > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 02 Sep 2014, at 18:01, Lincoln Greenhill > wrote: > >> Hi Jack (and John), >> >> Interesting report. Difficult to puzzle out >> differences in our mutual experience unless the >> 1012 is unlike the 1024 in subtle ways. >> >> We (LEDA) did not require an upgrade or Mellanox cables >> for the 10/40GbE links. Again - I would look >> forward to discussion people may have with Jonathon >> (Cc me if not conducted via the Casper list). >> >> Best, >> Lincoln >> >> On 9/2/14, 11:54 AM, Jack Hickish wrote: >>> Hi John, >>> >>> As Dan said I've tested (to some extent) the SX1012. I just used one >>> ROACH2 and corner >>> turned data through 8 x 10GbE ports. It worked well basically up to >>> line rate, with no CRC errors after a few hours of operation, but only >>> after >>> >>> - Jason put me in touch with some Mellanox guys who provided updated >>> firmware >>> - using branded Mellanox 3m QSFP -> 4xSFP cables >>> >>> Before the upgrade transmissions from the switch to the ROACH2 would >>> frequently fail. >>> >>> Of course this may or may not be remotely relevant for the SX1024 :) >>> >>> Good luck! >>> >>> J >>> >>> On 2 September 2014 16:46, Dan Werthimer wrote: hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) i think jason has also tested this switch. best wishes, dan On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: > > Hi all. Has anyone tested the Mellanox SX-1024 series of switches with > ROACH-2 and 10 and/or 40 gb NICs? > > These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. > > Thanks! > > John > > > >>> >> >> -- >> Lincoln J. Greenhill Harvard-Smithsonian CfA >> Office: 1 617-495-7194 60 Garden St, Mail Stop 42 >> Cell: 1 650 722-7798 Cambridge, MA 02138 >> FAX:1 617-495-7345 greenh...@cfa.harvard.edu >> Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln >> >
Re: [casper] Network Switches
> Hi John, > > Email is passing in the ether. > > I have eyed the SX1035 and SX1036 > w/ upgrades in mind. What is your > understanding of the difference? I can't see any, really. The sx1035 has slightly lower latency, and is way more expensive... I don't really understand what "L2 Lite Switching" is, which seems to be the difference! Does anyone know? John > > Lincoln > > > On 9/2/14, 11:54 AM, John Ford wrote: >>> hi john, >>> >>> jack hickish tested a mellanox SX1012 >>> (12x40Gbe, or 48x10Gbe, or mixture), >>> when he was visiting berkeley. >>> >>> the SX1012 worked beautifully on roach2, after jack upgraded >>> the switch firmware. and it's a great price. ($6K) >> >> Yeah, we're looking at a 1024, or a 1036. They all use the same >> silicon. >> >> Thanks! I'll give it a try and let you all know how it works out. >> >> John >> >>> >>> i think jason has also tested this switch. >>> >>> best wishes, >>> >>> dan >>> >>> >>> >>> >>> On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: >>> Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John >>> >> >> >> > > -- > Lincoln J. Greenhill Harvard-Smithsonian CfA > Office: 1 617-495-7194 60 Garden St, Mail Stop 42 > Cell: 1 650 722-7798 Cambridge, MA 02138 > FAX:1 617-495-7345 greenh...@cfa.harvard.edu > Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln >
Re: [casper] Network Switches
Hi all. After looking further into this, I think for our purposes the MSX1024 is the way to go. It has 48 10 gbe ports with SFP+ that we can use with optical transcievers. I don't think there's a 4-1 splitter cable with the ability to use optical SFP+ transcievers, so we'll have to limit our use of the 40 gig ports to computers with 40 gig interfaces. Or does someone know of a 40-10 optical cable? John > hi john, > > jack hickish tested a mellanox SX1012 > (12x40Gbe, or 48x10Gbe, or mixture), > when he was visiting berkeley. > > the SX1012 worked beautifully on roach2, after jack upgraded > the switch firmware. and it's a great price. ($6K) > > i think jason has also tested this switch. > > best wishes, > > dan > > > > > On Tue, Sep 2, 2014 at 8:36 AM, John Ford wrote: > >> Hi all. Has anyone tested the Mellanox SX-1024 series of switches with >> ROACH-2 and 10 and/or 40 gb NICs? >> >> These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. >> >> Thanks! >> >> John >> >> >> >> >
Re: [casper] Network Switches
> Or does someone know of a 40-10 optical cable? Yes, and mellanox have them, including the fibre cable itself in short distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so you're limited to ~100m or so. It uses an MTP connector on the 40G side, and an LC connector on the 10G side. We use these in our pedestal switches for MeerKAT. Jason
Re: [casper] Network Switches
> >> Or does someone know of a 40-10 optical cable? > Yes, and mellanox have them, including the fibre cable itself in short > distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so > you're limited to ~100m or so. It uses an MTP connector on the 40G side, > and an LC connector on the 10G side. We use these in our pedestal switches > for MeerKAT. Thanks. Maybe the 1036 is an option, then. It has way better internal interconnect bandwidth. I couldn't find them in the mellanox web site. > > Jason >
Re: [casper] Network Switches
Just be careful with the 1036 and breaking the 40g ports out to 10g. you can't break out all the ports, and when you break one out, you sometimes lose another 40g port! Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 03 Sep 2014, at 15:07, John Ford wrote: >> >>> Or does someone know of a 40-10 optical cable? >> Yes, and mellanox have them, including the fibre cable itself in short >> distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so >> you're limited to ~100m or so. It uses an MTP connector on the 40G side, >> and an LC connector on the 10G side. We use these in our pedestal switches >> for MeerKAT. > > Thanks. Maybe the 1036 is an option, then. It has way better internal > interconnect bandwidth. I couldn't find them in the mellanox web site. > >> >> Jason >> > >
Re: [casper] Network Switches
John - Mellanox hides this information in the back of their manuals in my experience. Best to download. L. On 9/3/14, 9:11 AM, Jason Manley wrote: Just be careful with the 1036 and breaking the 40g ports out to 10g. you can't break out all the ports, and when you break one out, you sometimes lose another 40g port! Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 03 Sep 2014, at 15:07, John Ford wrote: Or does someone know of a 40-10 optical cable? Yes, and mellanox have them, including the fibre cable itself in short distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so you're limited to ~100m or so. It uses an MTP connector on the 40G side, and an LC connector on the 10G side. We use these in our pedestal switches for MeerKAT. Thanks. Maybe the 1036 is an option, then. It has way better internal interconnect bandwidth. I couldn't find them in the mellanox web site. Jason -- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX:1 617-495-7345 greenh...@cfa.harvard.edu Skype: ljgreenhillwww.cfa.harvard.edu/~lincoln
Re: [casper] Network Switches
> Just be careful with the 1036 and breaking the 40g ports out to 10g. you > can't break out all the ports, and when you break one out, you sometimes > lose another 40g port! I just noticed that! It's true also with the 1024. > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 03 Sep 2014, at 15:07, John Ford wrote: > >>> Or does someone know of a 40-10 optical cable? >>> Yes, and mellanox have them, including the fibre cable itself in short >>> distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so >>> you're limited to ~100m or so. It uses an MTP connector on the 40G >>> side, >>> and an LC connector on the 10G side. We use these in our pedestal >>> switches >>> for MeerKAT. >> >> Thanks. Maybe the 1036 is an option, then. It has way better internal >> interconnect bandwidth. I couldn't find them in the mellanox web site. >> >>> >>> Jason >>> >> >> > >
