I found before the firmware upgrade the problems were very hardware dependent.
Eg, all ports would have errors with mellanox 3m cables, whilst with shorter (2m, non-mellanox) cables, most ROACH2 ports would work, though some would have occassional errors. After the firmware upgrade, the mellanox 3m cables worked everywhere, but the unbranded 2m still gave errors in certain ports. As far as I could tell, (though I didn't pursue this as vigorously as I could, since Jason's report was that the switch wouldn't work straight out the box) there were no problems when sending ROACH2->switch, only the other way round. I'm trying to buy an SX1012 in the UK at the moment, and will need to test again with higher traffic rates. I'll let the list know how things go... Jack On 2 September 2014 17:01, Lincoln Greenhill <lgreenh...@cfa.harvard.edu> wrote: > Hi Jack (and John), > > Interesting report. Difficult to puzzle out > differences in our mutual experience unless the > 1012 is unlike the 1024 in subtle ways. > > We (LEDA) did not require an upgrade or Mellanox cables > for the 10/40GbE links. Again - I would look > forward to discussion people may have with Jonathon > (Cc me if not conducted via the Casper list). > > Best, > Lincoln > > > On 9/2/14, 11:54 AM, Jack Hickish wrote: >> >> Hi John, >> >> As Dan said I've tested (to some extent) the SX1012. I just used one >> ROACH2 and corner >> turned data through 8 x 10GbE ports. It worked well basically up to >> line rate, with no CRC errors after a few hours of operation, but only >> after >> >> - Jason put me in touch with some Mellanox guys who provided updated >> firmware >> - using branded Mellanox 3m QSFP -> 4xSFP cables >> >> Before the upgrade transmissions from the switch to the ROACH2 would >> frequently fail. >> >> Of course this may or may not be remotely relevant for the SX1024 :) >> >> Good luck! >> >> J >> >> On 2 September 2014 16:46, Dan Werthimer <d...@ssl.berkeley.edu> wrote: >>> >>> >>> hi john, >>> >>> jack hickish tested a mellanox SX1012 >>> (12x40Gbe, or 48x10Gbe, or mixture), >>> when he was visiting berkeley. >>> >>> the SX1012 worked beautifully on roach2, after jack upgraded >>> the switch firmware. and it's a great price. ($6K) >>> >>> i think jason has also tested this switch. >>> >>> best wishes, >>> >>> dan >>> >>> >>> >>> >>> On Tue, Sep 2, 2014 at 8:36 AM, John Ford <jf...@nrao.edu> wrote: >>>> >>>> >>>> Hi all. Has anyone tested the Mellanox SX-1024 series of switches with >>>> ROACH-2 and 10 and/or 40 gb NICs? >>>> >>>> These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. >>>> >>>> Thanks! >>>> >>>> John >>>> >>>> >>>> >>> >> > > -- > Lincoln J. Greenhill Harvard-Smithsonian CfA > Office: 1 617-495-7194 60 Garden St, Mail Stop 42 > Cell: 1 650 722-7798 Cambridge, MA 02138 > FAX: 1 617-495-7345 greenh...@cfa.harvard.edu > Skype: ljgreenhill www.cfa.harvard.edu/~lincoln
