Jeff,
I think you will have to revert this patch as the btl_bandwidth __IS__ supposed
to be in Mbs and not MBs. We usually talk about networks in Mbs (there is a
pattern in Ethernet 1G/10G, Myricom 10G). In addition the original design of
the multi-rail was based on this assumption, and the multi-rail handling code
deal with these values (at that level I don't think it really matters, but at
least it needs consistent values from all BTLs).
However, going over the existing BTLs I can see that some BTLs do not correctly
set this value:
BTL Bandwidth Auto-detect Status
Elan 2000 NO Correct
GM 250 NO Doubtful
MX 2000/10000 YES (Mbs) Correct (before the patch)
OFUD 800 NO Doubtful
OpenIB 2000/4000/8000 YES (Mbs) Correct (multiplied by the
active_width)
Portals 1000 NO Doubtful
SCTP 100 NO Conservative value (correct)
Self 100 XXX Correct (doesn't matter anyway)
SM 9000 NO Correct
TCP 100 NO Conservative value (correct)
UDAPL 225 NO Incorrect
Some of these BTL values do not make sense, neither in Mbs or MBs. Here is a
list of such BTLs: OFUD, Portals, UDAPL. If the corresponding developers can
provide the default bandwidth (in Mbs) I will update their values.
For SCTP, TCP I don't know how to detect it reliably in a portable way, so I
expect to let them set to this very conservative value. Moreover, the BTL TCP
is only used for multi-rail if the available high performance network allows
it, so it doesn't really matter.
george.
On Sep 3, 2010, at 08:03 , Jeff Squyres wrote:
> Thanks; committed in r23712.
>
> Can you file CMRs for 1.4 and 1.5? Thanks.
>
>
> On Sep 3, 2010, at 3:53 AM, Brice Goglin wrote:
>
>> For some reason, the MX btl sets btl_bandwidth in megabits/s instead of
>> megabytes/s. So we get crazy btl_weights in case of heterogeneous
>> multirail. And --mca btl_mx_bandwidth <width> cannot work around the
>> problem (it probably doesn't help because it's overriden by the runtime
>> link width detection anyway?).
>>
>> Signed-off-by: Brice Goglin <brice.gog...@inria.fr>
>>
>> Index: ompi/mca/btl/mx/btl_mx_component.c
>> ===================================================================
>> --- ompi/mca/btl/mx/btl_mx_component.c (révision 23711)
>> +++ ompi/mca/btl/mx/btl_mx_component.c (copie de travail)
>> @@ -159,7 +159,7 @@
>> MCA_BTL_FLAGS_PUT |
>> MCA_BTL_FLAGS_SEND |
>> MCA_BTL_FLAGS_RDMA_MATCHED);
>> - mca_btl_mx_module.super.btl_bandwidth = 2000;
>> + mca_btl_mx_module.super.btl_bandwidth = 250;
>> mca_btl_mx_module.super.btl_latency = 5;
>> mca_btl_base_param_register(&mca_btl_mx_component.super.btl_version,
>> &mca_btl_mx_module.super);
>> @@ -357,7 +357,7 @@
>> mx_btl->mx_endpoint = mx_endpoint;
>> mx_btl->mx_endpoint_addr = mx_endpoint_addr;
>>
>> - mx_btl->super.btl_bandwidth = 2000; /* whatever */
>> + mx_btl->super.btl_bandwidth = 250; /* whatever */
>> mx_btl->super.btl_latency = 10;
>> #if defined(MX_HAS_NET_TYPE)
>> {
>> @@ -370,11 +370,11 @@
>> } else {
>> if( MX_SPEED_2G == value ) {
>> mx_unique_network_id |= 0xaa000000;
>> - mx_btl->super.btl_bandwidth = 2000;
>> + mx_btl->super.btl_bandwidth = 250;
>> mx_btl->super.btl_latency = 5;
>> } else if( MX_SPEED_10G == value ) {
>> mx_unique_network_id |= 0xbb000000;
>> - mx_btl->super.btl_bandwidth = 10000;
>> + mx_btl->super.btl_bandwidth = 1250;
>> mx_btl->super.btl_latency = 3;
>> } else {
>> mx_unique_network_id |= 0xcc000000;
>>
>>
>> <fix_btl_bandwidths.patch>_______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
