Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r15041

[Jeff Squyres]

Hey Gleb --

Can you explain the rationale for this change?  Is there a reason why  
the bandwidths reported by the IBV API are not sufficient?  Are you  
trying to do creative things with multi-LID scenarios (perhaps QOS- 
like things)?  If so, this looks like a good idea, but I'm not sure  
that MCA params are the right vehicle for it.

I'm not particularly fond of creating variable MCA parameters after  
the btl open call because they won't show up in ompi_info.  Should we  
do something else if you want to override bandwidths, perhaps  
something similar to the HCA params file?  If you recall, we made the  
HCA params stuff a file because of exactly this issue: we didn't want  
variable MCA parameters and we didn't want to have to make the user  
type out a potentially long set of complex MCA parameters that were  
dependent upon the exact environment that the user was running in.



On Jun 13, 2007, at 8:47 AM, g...@osl.iu.edu wrote:

Author: gleb
Date: 2007-06-13 08:47:38 EDT (Wed, 13 Jun 2007)
New Revision: 15041
URL: https://svn.open-mpi.org/trac/ompi/changeset/15041

Log:
Allow to configure bandwidth and latency with finer granularity.

Set bandwidth for all ports of mthca0:
--mca btl_openib_bandwidth_mthca0 1000

Set bandwidth for port 1 of mthca1:
--mca btl_openib_bandwidth_mthca1:1 1000

Set latency for port 2 lid 123 on mthca0:
--mca btl_openib_latency_mthca0:2:123 20

Text files modified:
   trunk/ompi/mca/btl/openib/btl_openib_component.c |    43 ++++++++ 
++++++++++++++++++++++++++++++++
   1 files changed, 43 insertions(+), 0 deletions(-)

Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c
====================================================================== 
========
--- trunk/ompi/mca/btl/openib/btl_openib_component.c    (original)
+++ trunk/ompi/mca/btl/openib/btl_openib_component.c	2007-06-13  
08:47:38 EDT (Wed, 13 Jun 2007)
@@ -299,6 +299,14 @@
     openib_reg->mr = NULL;
     return OMPI_SUCCESS;
 }
+static inline int param_register_int(const char* param_name, int  
default_value)
+{
+    int param_value = default_value;
+    int id = mca_base_param_register_int("btl", "openib",  
param_name, NULL,
+            default_value);
+    mca_base_param_lookup_int(id, &param_value);
+    return param_value;
+}

 static int init_one_port(opal_list_t *btl_list,  
mca_btl_openib_hca_t *hca,
                          uint8_t port_num, uint16_t pkey_index,
@@ -331,6 +339,7 @@
     for(lid = ib_port_attr->lid;
             lid < ib_port_attr->lid + lmc; lid++){
         for(i = 0; i < mca_btl_openib_component.btls_per_lid; i++){
+            char param[40];
             openib_btl = malloc(sizeof(mca_btl_openib_module_t));
             if(NULL == openib_btl) {
                 BTL_ERROR(("Failed malloc: %s:%d\n", __FILE__,  
__LINE__));
@@ -353,6 +362,40 @@
             openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbfunc =  
btl_openib_control;
             openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbdata = NULL;

+            /* Check bandwidth configured for this HCA */
+            sprintf(param, "bandwidth_%s", ibv_get_device_name(hca- 
>ib_dev));
+            openib_btl->super.btl_bandwidth =
+                param_register_int(param, openib_btl- 
>super.btl_bandwidth);
+
+            /* Check bandwidth configured for this HCA/port */
+            sprintf(param, "bandwidth_%s:%d", ibv_get_device_name 
(hca->ib_dev),
+                    port_num);
+            openib_btl->super.btl_bandwidth =
+                param_register_int(param, openib_btl- 
>super.btl_bandwidth);
+
+            /* Check bandwidth configured for this HCA/port/LID */
+            sprintf(param, "bandwidth_%s:%d:%d",
+                    ibv_get_device_name(hca->ib_dev), port_num, lid);
+            openib_btl->super.btl_bandwidth =
+                param_register_int(param, openib_btl- 
>super.btl_bandwidth);
+
+            /* Check latency configured for this HCA */
+            sprintf(param, "latency_%s", ibv_get_device_name(hca- 
>ib_dev));
+            openib_btl->super.btl_latency =
+                param_register_int(param, openib_btl- 
>super.btl_latency);
+
+            /* Check latency configured for this HCA/port */
+            sprintf(param, "latency_%s:%d", ibv_get_device_name 
(hca->ib_dev),
+                    port_num);
+            openib_btl->super.btl_latency =
+                param_register_int(param, openib_btl- 
>super.btl_latency);
+
+            /* Check latency configured for this HCA/port/LID */
+            sprintf(param, "latency_%s:%d:%d", ibv_get_device_name 
(hca->ib_dev),
+                    port_num, lid);
+            openib_btl->super.btl_latency =
+                param_register_int(param, openib_btl- 
>super.btl_latency);
+
             /* Auto-detect the port bandwidth */
             if (0 == openib_btl->super.btl_bandwidth) {
                 /* To calculate the bandwidth available on this port,
_______________________________________________
svn-full mailing list
svn-f...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/svn-full


--
Jeff Squyres
Cisco Systems