On 09-May-12 7:26 PM, Jeff Squyres wrote:
> On May 9, 2012, at 11:33 AM,<ludovic.hab...@ext.bull.net> wrote:
>
>> Well, looking at the libverbs sources, it seems that there is a patch for
>> devinfo (cf below).
>>
>> Thus, the correct speeds, according to the libibverbs patch. And, if I
>> understand well, they must be
>> multiplied by the coding ratio to get the real link speed.
>>
>> I joined another patch. I don't know if we should remove the case 4 and fall
>> through case 8 or just keep it as I do.
>
> Your patch looks good to me... but what do I know? :-)
>
> Mellanox: please comment.
>
>> Regards,
>>
>> Ludovic
>>
>> libibverbs/fixes/add_support_for_extended_speends.patch
>>
>> [PATCH]libibverbs: Add support for extended speeds
>>
>> 8: FDR-10 is a proprietary link speed which is 10.3125 Gbps at 64/66
>> encoding rather than 8b10b encoding.
>> 16: FDR - 14.0625 Gbps
>> 32: EDR - 25.78125 Gbps
>>
>> +--- a/examples/devinfo.c
>> ++++ b/examples/devinfo.c
>> +@@ -139,7 +139,12 @@ static const char *speed_str(uint8_t speed)
>> + switch (speed) {
>> + case 1: return "2.5 Gbps";
>> + case 2: return "5.0 Gbps";
>> +- case 4: return "10.0 Gbps";
>> ++
>> ++ case 4: /* fall through */
>> ++ case 8: return "10.0 Gbps";
>> ++
>> ++ case 16: return "14.0 Gbps";
>> ++ case 32: return "25.0 Gbps";
>> + default: return "invalid speed";
>> + }
>> + }
>>
>>
>>
>> -----devel-boun...@open-mpi.org a écrit : -----
>> A : de...@open-mpi.org
>> De : Peter Kjellström
>> Envoyé par : devel-boun...@open-mpi.org
>> Date : 09/05/2012 16:35
>> Objet : Re: [OMPI devel] FDR: bad patch? (was: 1.6rc3 is out)
>>
>> On Wednesday 09 May 2012 10.20.47 Jeff Squyres wrote:
>> ...
>>> I don't have any QDR or FDR equipment -- what's the typical active_width for
>>> QDR and FDR?
>>>
>>> QDR is 32000, so is that active_speed of 4 and active_width of 4? (8000 * 4
>>> = 32000)?
>>>
>>> FDR is 56000 (right?
>>
>> Pretty much all IB is 4x to host.
>>
>> 4x QDR is 40 Gbps at link level => 8 Gbps data per link (8/10 coding)
>> 4x FDR10 is 40 Gbps at link level => ~9.7 Gbps data per link (64/66 coding)
>> 4x FDR is 56 Gbps at link level => ~13.5 Gbps data per link (64/66 coding)
Here are the ActiveSpeed values:
1: SDR - 2.5 Gb/s * (8/10 coding)
2: DDR - 5 Gb/s * (8/10 coding)
4: QDR - 10 Gb/s * (8/10 coding)
8: FDR10 - 10 Gb/s * (64/66 coding)
16: FDR - 14 Gbps * (64/66 coding)
32: EDR - 25 Gbs * (64/66 coding)
EDR is not out yet, so we can ignore it for the sake of this patch.
It is also true that *most* of IB cables are 4x.
There are 12x clusters out there (like TACC's Ranger), and sometimes
there are 1x links (mostly because the cable is not perfect or too long).
So cases of 4 and 8 are not the same for us, because MPI needs actual
*data* rate instead of *bit* rate, which is different in these cases.
I'll send a patch for review in a minute...
-- YK
>>
>> /Peter
>>
>>> I haven't kept up with IB specs since Cisco killed
>>> IB...), so what are its active_speed and active_width values?
>>
>> --
>> -= Peter Kjellström
>> -= National Supercomputer Centre
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> [pièce jointe "signature.asc" supprimée par Ludovic Hablot/EXT/FR/BULL]
>> <OpenMPI_FDR.patch>_______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
