Jeff, I think you will have to revert this patch as the btl_bandwidth __IS__ supposed to be in Mbs and not MBs. We usually talk about networks in Mbs (there is a pattern in Ethernet 1G/10G, Myricom 10G). In addition the original design of the multi-rail was based on this assumption, and the multi-rail handling code deal with these values (at that level I don't think it really matters, but at least it needs consistent values from all BTLs).
However, going over the existing BTLs I can see that some BTLs do not correctly set this value: BTL Bandwidth Auto-detect Status Elan 2000 NO Correct GM 250 NO Doubtful MX 2000/10000 YES (Mbs) Correct (before the patch) OFUD 800 NO Doubtful OpenIB 2000/4000/8000 YES (Mbs) Correct (multiplied by the active_width) Portals 1000 NO Doubtful SCTP 100 NO Conservative value (correct) Self 100 XXX Correct (doesn't matter anyway) SM 9000 NO Correct TCP 100 NO Conservative value (correct) UDAPL 225 NO Incorrect Some of these BTL values do not make sense, neither in Mbs or MBs. Here is a list of such BTLs: OFUD, Portals, UDAPL. If the corresponding developers can provide the default bandwidth (in Mbs) I will update their values. For SCTP, TCP I don't know how to detect it reliably in a portable way, so I expect to let them set to this very conservative value. Moreover, the BTL TCP is only used for multi-rail if the available high performance network allows it, so it doesn't really matter. george. On Sep 3, 2010, at 08:03 , Jeff Squyres wrote: > Thanks; committed in r23712. > > Can you file CMRs for 1.4 and 1.5? Thanks. > > > On Sep 3, 2010, at 3:53 AM, Brice Goglin wrote: > >> For some reason, the MX btl sets btl_bandwidth in megabits/s instead of >> megabytes/s. So we get crazy btl_weights in case of heterogeneous >> multirail. And --mca btl_mx_bandwidth <width> cannot work around the >> problem (it probably doesn't help because it's overriden by the runtime >> link width detection anyway?). >> >> Signed-off-by: Brice Goglin <[email protected]> >> >> Index: ompi/mca/btl/mx/btl_mx_component.c >> =================================================================== >> --- ompi/mca/btl/mx/btl_mx_component.c (révision 23711) >> +++ ompi/mca/btl/mx/btl_mx_component.c (copie de travail) >> @@ -159,7 +159,7 @@ >> MCA_BTL_FLAGS_PUT | >> MCA_BTL_FLAGS_SEND | >> MCA_BTL_FLAGS_RDMA_MATCHED); >> - mca_btl_mx_module.super.btl_bandwidth = 2000; >> + mca_btl_mx_module.super.btl_bandwidth = 250; >> mca_btl_mx_module.super.btl_latency = 5; >> mca_btl_base_param_register(&mca_btl_mx_component.super.btl_version, >> &mca_btl_mx_module.super); >> @@ -357,7 +357,7 @@ >> mx_btl->mx_endpoint = mx_endpoint; >> mx_btl->mx_endpoint_addr = mx_endpoint_addr; >> >> - mx_btl->super.btl_bandwidth = 2000; /* whatever */ >> + mx_btl->super.btl_bandwidth = 250; /* whatever */ >> mx_btl->super.btl_latency = 10; >> #if defined(MX_HAS_NET_TYPE) >> { >> @@ -370,11 +370,11 @@ >> } else { >> if( MX_SPEED_2G == value ) { >> mx_unique_network_id |= 0xaa000000; >> - mx_btl->super.btl_bandwidth = 2000; >> + mx_btl->super.btl_bandwidth = 250; >> mx_btl->super.btl_latency = 5; >> } else if( MX_SPEED_10G == value ) { >> mx_unique_network_id |= 0xbb000000; >> - mx_btl->super.btl_bandwidth = 10000; >> + mx_btl->super.btl_bandwidth = 1250; >> mx_btl->super.btl_latency = 3; >> } else { >> mx_unique_network_id |= 0xcc000000; >> >> >> <fix_btl_bandwidths.patch>_______________________________________________ >> devel mailing list >> [email protected] >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > [email protected] > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > devel mailing list > [email protected] > http://www.open-mpi.org/mailman/listinfo.cgi/devel
