Re: [OMPI devel] Q: Using a hostfile in managed environment?

[Thomas Naughton]

Hi Ralph,

OK, that's pretty much what I thought but wanted to get a sanity check. :-)
I'll see if I can reproduce the issue in a more precise manner and open an
issue if I find something off in the mapping.

Thanks,
--tjn

 _
  Thomas Naughton  naught...@ornl.gov
  Research Associate   (865) 576-4184


On Fri, 24 Feb 2017, r...@open-mpi.org wrote:




On Feb 24, 2017, at 11:57 AM, Thomas Naughton  wrote:

Hi,

We're trying to track down some curious behavior and decided to take a step
back and check a base assumption.

When running within a managed environment (job allocation):

   Q: Should you be able to use `--hostfile` or `--host` options to
  operate on a subset of the resources in the allocation?
  (Example: within 4 node SLURM allocation, run on just 2 nodes in
   allocation.)


Yes - those options are used to “filter” the allocation prior to launch



   Q: Additionally, should this be the same when launching the DVM in
  order to run on a subset of resources using subsequent
  'mpirun --hnp ...' commands?
  (Only 'orte-dvm' would need to have `--hostfile` or `--host` args.)


Yes - only the DVM needs to know the filter. When operating with a DVM, “mpirun 
--hnp...” only packages up the cmd line and sends it to the DVM. All the 
mapping occurs in orte-dvm.



There are a variety of interactions with ess/ras/rmaps and the resource
manager, but the thought was that you "should" be able to use a hostfile to
operate on a subset of the allocation. Is that a flawed assumption?

Thanks,
--tjn

_
 Thomas Naughton  naught...@ornl.gov
 Research Associate   (865) 576-4184

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel


___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Q: Using a hostfile in managed environment?

[r...@open-mpi.org]

> On Feb 24, 2017, at 11:57 AM, Thomas Naughton  wrote:
> 
> Hi,
> 
> We're trying to track down some curious behavior and decided to take a step
> back and check a base assumption.
> 
> When running within a managed environment (job allocation):
> 
>Q: Should you be able to use `--hostfile` or `--host` options to
>   operate on a subset of the resources in the allocation?
>   (Example: within 4 node SLURM allocation, run on just 2 nodes in
>allocation.)

Yes - those options are used to “filter” the allocation prior to launch

> 
>Q: Additionally, should this be the same when launching the DVM in
>   order to run on a subset of resources using subsequent
>   'mpirun --hnp ...' commands?
>   (Only 'orte-dvm' would need to have `--hostfile` or `--host` args.)

Yes - only the DVM needs to know the filter. When operating with a DVM, “mpirun 
--hnp...” only packages up the cmd line and sends it to the DVM. All the 
mapping occurs in orte-dvm.

> 
> There are a variety of interactions with ess/ras/rmaps and the resource
> manager, but the thought was that you "should" be able to use a hostfile to
> operate on a subset of the allocation. Is that a flawed assumption?
> 
> Thanks,
> --tjn
> 
> _
>  Thomas Naughton  naught...@ornl.gov
>  Research Associate   (865) 576-4184
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Jeff Squyres (jsquyres)]

Please re-read my prior email (MCA parameters are not always set as environment 
variables):

https://www.mail-archive.com/devel@lists.open-mpi.org/msg19958.html

If you want to read the value of MCA parameters in MPI applications, you should 
be using the MPI_T API. That will work regardless of whether you pass MCA 
parameters on the mpirun command line, in the openmpi-mca-params.conf file, or 
other mechanism.



> On Feb 24, 2017, at 1:36 PM, Dahai Guo  wrote:
> 
> Basically, I like to set apath in the param conf file, and let OMPI broadcast 
> it to each MPI rank.  
> 
> In the following example, it prints the value as NULL, if I just set it in 
> the conf file. So My question is how to update some ompi base code to make it 
> right?
> 
>   If I set it in mpirun -mca apath=sth, I can update a bit "setup_folk" and 
> get it right for each rank. 
> 
> Dahai
> 
> 
> #include 
> #include 
> #include 
> #include 
> 
> int main(int argc, char *argv[] )
> {
> 
>   intprovided, rank, Size, ierr;
> 
>   ierr=MPI_Init_thread(,,MPI_THREAD_FUNNELED, );
>   ierr=MPI_Comm_rank(MPI_COMM_WORLD, );
>   ierr=MPI_Comm_size(MPI_COMM_WORLD, );
> 
>char *value = getenv("OMPI_MCA_apath");
> 
>   printf(" --- Hi from rank = %d, path0 = %s  \n", rank, value );
> 
>   ierr=MPI_Barrier(MPI_COMM_WORLD);
>   ierr=MPI_Finalize();
> 
>   return 0;
> }
> 
> 
> From: Jeff Squyres (jsquyres) 
> To: Dahai Guo ; Open MPI Developers List 
>  
> Cc: Dahai Guo 
> Sent: Friday, February 24, 2017 12:27 PM
> Subject: Re: [OMPI devel] define a new ENV variable in 
> etc/openmpi-mca-params.conf
> 
> It's still not clear what you're trying to do.
> 
> Are you setting an MCA parameter, or are you trying to forward arbitrary 
> environment variables?  Are you working in MPI processes (or executables 
> launched by MPI processes), or in the Open MPI code base itself?
> 
> Also, Ralph and I chatted: what he said is not correct for the v2.x series 
> (my original answer was correct).  We think what Ralph said may have been 
> true in prior releases, but without further detailed information on what 
> you're trying to do, we didn't dig any further.
> 
> 
> 
> 
> > On Feb 24, 2017, at 11:35 AM, Dahai Guo via devel 
> >  wrote:
> > 
> > 
> > I need this parameter used in my other codes related to ompi, 
> > do I need to register "OMPI_MCA_apath" in some codes in OMPI? if so, 
> > usually in which codes?  
> > 
> > Dahai 
> > 
> > 
> > 
> > 
> > From: "r...@open-mpi.org" 
> > To: OpenMPI Devel  
> > Sent: Friday, February 24, 2017 9:44 AM
> > Subject: Re: [OMPI devel] define a new ENV variable in 
> > etc/openmpi-mca-params.conf
> > 
> > I think Jeff got lost in the weeds here. If you define a new MCA param in 
> > the default param file, we will automatically pick it up and it will be in 
> > the environment of your application. You don’t need to do anything. 
> > However, you checked for the wrong envar. Anything you provide is going to 
> > have an “OMPI_MCA_” attached to the front of it. So for your “apath” 
> > example, the envar will be
> > 
> > OMPI_MCA_apath
> > 
> > HTH
> > Ralph
> > 
> > > On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> > > wrote:
> > > 
> > > On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
> > >> 
> > >> oops, I should use the word "MCA parameters". If I define a MCA 
> > >> parameter (apath, for example) in etc/openmpi-mca-params.conf, how can 
> > >> function setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its 
> > >> value?  I tried to call getenv("apath") there, which return null. 
> > > 
> > > Settings MCA parameters does not necessarily set environment variables.  
> > > More below.
> > > 
> > >> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then 
> > >> I could get it in setup_fork.
> > > 
> > > There are multiple mechanisms in which MCA params are passed to the 
> > > back-end MPI processes.
> > > 
> > > 1. If they are set via "--mca a b" on the mpirun command line, they are 
> > > passed back to the launching orteds as part of the launch schema for the 
> > > job.  I.e., the MCA param names and values are bundled up and sent in the 
> > > network message to each of the orteds that are involved in the launching 
> > > of the target processes.  IIRC, the launching orteds will fork, setenv a 
> > > corresponding environment variable for each of the MCA param values in 
> > > the launch schema, and then exec the MPI process.  The MPI process then 
> > > picks up those MCA param values from the environment variables.
> > > 
> > > 2. If they are set via the openmpi-mca-params.conf file, they are not 
> > > sent via network message in the launch schema.  Instead, each Open MPI 
> > > process (including MPI processes and orteds) reads that 

[OMPI devel] Q: Using a hostfile in managed environment?

[Thomas Naughton]

Hi,

We're trying to track down some curious behavior and decided to take a step
back and check a base assumption.

When running within a managed environment (job allocation):

Q: Should you be able to use `--hostfile` or `--host` options to
   operate on a subset of the resources in the allocation?
   (Example: within 4 node SLURM allocation, run on just 2 nodes in
allocation.)

Q: Additionally, should this be the same when launching the DVM in
   order to run on a subset of resources using subsequent
   'mpirun --hnp ...' commands?
   (Only 'orte-dvm' would need to have `--hostfile` or `--host` args.)

There are a variety of interactions with ess/ras/rmaps and the resource
manager, but the thought was that you "should" be able to use a hostfile to
operate on a subset of the allocation. Is that a flawed assumption?

Thanks,
--tjn

 _
  Thomas Naughton  naught...@ornl.gov
  Research Associate   (865) 576-4184

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Dahai Guo via devel]

Basically, I like to set apath in the param conf file, and let OMPI broadcast 
it to each MPI rank.  
In the following example, it prints the value as NULL, if I just set it in the 
conf file. So My question is how to update some ompi base code to make it right?
  If I set it in mpirun -mca apath=sth, I can update a bit "setup_folk" and get 
it right for each rank. 
Dahai

#include #include #include #include 
int main(int argc, char *argv[] ){
  int    provided, rank, Size, ierr;
  ierr=MPI_Init_thread(,,MPI_THREAD_FUNNELED, );  
ierr=MPI_Comm_rank(MPI_COMM_WORLD, );  ierr=MPI_Comm_size(MPI_COMM_WORLD, 
);
   char *value = getenv("OMPI_MCA_apath");
  printf(" --- Hi from rank = %d, path0 = %s  \n", rank, value );
  ierr=MPI_Barrier(MPI_COMM_WORLD);  ierr=MPI_Finalize();
  return 0;}

  From: Jeff Squyres (jsquyres) 
 To: Dahai Guo ; Open MPI Developers List 
 
Cc: Dahai Guo 
 Sent: Friday, February 24, 2017 12:27 PM
 Subject: Re: [OMPI devel] define a new ENV variable in 
etc/openmpi-mca-params.conf
   
It's still not clear what you're trying to do.

Are you setting an MCA parameter, or are you trying to forward arbitrary 
environment variables?  Are you working in MPI processes (or executables 
launched by MPI processes), or in the Open MPI code base itself?

Also, Ralph and I chatted: what he said is not correct for the v2.x series (my 
original answer was correct).  We think what Ralph said may have been true in 
prior releases, but without further detailed information on what you're trying 
to do, we didn't dig any further.




> On Feb 24, 2017, at 11:35 AM, Dahai Guo via devel  
> wrote:
> 
> 
> I need this parameter used in my other codes related to ompi, 
> do I need to register "OMPI_MCA_apath" in some codes in OMPI? if so, usually 
> in which codes?  
> 
> Dahai 
> 
> 
> 
> 
> From: "r...@open-mpi.org" 
> To: OpenMPI Devel  
> Sent: Friday, February 24, 2017 9:44 AM
> Subject: Re: [OMPI devel] define a new ENV variable in 
> etc/openmpi-mca-params.conf
> 
> I think Jeff got lost in the weeds here. If you define a new MCA param in the 
> default param file, we will automatically pick it up and it will be in the 
> environment of your application. You don’t need to do anything. However, you 
> checked for the wrong envar. Anything you provide is going to have an 
> “OMPI_MCA_” attached to the front of it. So for your “apath” example, the 
> envar will be
> 
> OMPI_MCA_apath
> 
> HTH
> Ralph
> 
> > On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> > wrote:
> > 
> > On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
> >> 
> >> oops, I should use the word "MCA parameters". If I define a MCA parameter 
> >> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
> >> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I 
> >> tried to call getenv("apath") there, which return null. 
> > 
> > Settings MCA parameters does not necessarily set environment variables.  
> > More below.
> > 
> >> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
> >> could get it in setup_fork.
> > 
> > There are multiple mechanisms in which MCA params are passed to the 
> > back-end MPI processes.
> > 
> > 1. If they are set via "--mca a b" on the mpirun command line, they are 
> > passed back to the launching orteds as part of the launch schema for the 
> > job.  I.e., the MCA param names and values are bundled up and sent in the 
> > network message to each of the orteds that are involved in the launching of 
> > the target processes.  IIRC, the launching orteds will fork, setenv a 
> > corresponding environment variable for each of the MCA param values in the 
> > launch schema, and then exec the MPI process.  The MPI process then picks 
> > up those MCA param values from the environment variables.
> > 
> > 2. If they are set via the openmpi-mca-params.conf file, they are not sent 
> > via network message in the launch schema.  Instead, each Open MPI process 
> > (including MPI processes and orteds) reads that file upon startup to get 
> > the values (more specifically: all Open MPI processes *always* read this 
> > file before processing whatever MCA param values are sent in the launch 
> > schema).  A consequence: if openmpi-mca-params.conf is not on a network 
> > filesystem (and visible to all Open MPI processes on all servers), you need 
> > to copy the change you made to the one openmpi-mca-params.conf file to all 
> > copies of openmpi-mca-params.conf.
> > 
> > -
> > 
> > What are you trying to do?
> > 
> > If you're working in an MPI application, you can use the MPI_T API to 
> > retrieve MCA parameter names and their current values (regardless of 
> > whether mechanism #1 or #2 -- or another mechanism -- is used).
> > 
> > If you're 

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Jeff Squyres (jsquyres)]

It's still not clear what you're trying to do.

Are you setting an MCA parameter, or are you trying to forward arbitrary 
environment variables?  Are you working in MPI processes (or executables 
launched by MPI processes), or in the Open MPI code base itself?

Also, Ralph and I chatted: what he said is not correct for the v2.x series (my 
original answer was correct).  We think what Ralph said may have been true in 
prior releases, but without further detailed information on what you're trying 
to do, we didn't dig any further.




> On Feb 24, 2017, at 11:35 AM, Dahai Guo via devel  
> wrote:
> 
> 
> I need this parameter used in my other codes related to ompi, 
> do I need to register "OMPI_MCA_apath" in some codes in OMPI? if so, usually 
> in which codes?  
> 
> Dahai 
> 
> 
> 
> 
> From: "r...@open-mpi.org" 
> To: OpenMPI Devel  
> Sent: Friday, February 24, 2017 9:44 AM
> Subject: Re: [OMPI devel] define a new ENV variable in 
> etc/openmpi-mca-params.conf
> 
> I think Jeff got lost in the weeds here. If you define a new MCA param in the 
> default param file, we will automatically pick it up and it will be in the 
> environment of your application. You don’t need to do anything. However, you 
> checked for the wrong envar. Anything you provide is going to have an 
> “OMPI_MCA_” attached to the front of it. So for your “apath” example, the 
> envar will be
> 
> OMPI_MCA_apath
> 
> HTH
> Ralph
> 
> > On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> > wrote:
> > 
> > On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
> >> 
> >> oops, I should use the word "MCA parameters". If I define a MCA parameter 
> >> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
> >> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I 
> >> tried to call getenv("apath") there, which return null. 
> > 
> > Settings MCA parameters does not necessarily set environment variables.  
> > More below.
> > 
> >> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
> >> could get it in setup_fork.
> > 
> > There are multiple mechanisms in which MCA params are passed to the 
> > back-end MPI processes.
> > 
> > 1. If they are set via "--mca a b" on the mpirun command line, they are 
> > passed back to the launching orteds as part of the launch schema for the 
> > job.  I.e., the MCA param names and values are bundled up and sent in the 
> > network message to each of the orteds that are involved in the launching of 
> > the target processes.  IIRC, the launching orteds will fork, setenv a 
> > corresponding environment variable for each of the MCA param values in the 
> > launch schema, and then exec the MPI process.  The MPI process then picks 
> > up those MCA param values from the environment variables.
> > 
> > 2. If they are set via the openmpi-mca-params.conf file, they are not sent 
> > via network message in the launch schema.  Instead, each Open MPI process 
> > (including MPI processes and orteds) reads that file upon startup to get 
> > the values (more specifically: all Open MPI processes *always* read this 
> > file before processing whatever MCA param values are sent in the launch 
> > schema).  A consequence: if openmpi-mca-params.conf is not on a network 
> > filesystem (and visible to all Open MPI processes on all servers), you need 
> > to copy the change you made to the one openmpi-mca-params.conf file to all 
> > copies of openmpi-mca-params.conf.
> > 
> > -
> > 
> > What are you trying to do?
> > 
> > If you're working in an MPI application, you can use the MPI_T API to 
> > retrieve MCA parameter names and their current values (regardless of 
> > whether mechanism #1 or #2 -- or another mechanism -- is used).
> > 
> > If you're working in the Open MPI code base, you should be using the 
> > internal opal_mca_var API to retrieve and/or set MCA param values.  See 
> > https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_var.h 
> > for all the functions available in this API.
> > 
> > -- 
> > Jeff Squyres
> > jsquy...@cisco.com
> 
> > 
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel


-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Dahai Guo via devel]

I need this parameter used in my other codes related to ompi, 
do I need to register "OMPI_MCA_apath" in some codes in OMPI? if so, usually in 
which codes?  
Dahai 



  From: "r...@open-mpi.org" 
 To: OpenMPI Devel  
 Sent: Friday, February 24, 2017 9:44 AM
 Subject: Re: [OMPI devel] define a new ENV variable in 
etc/openmpi-mca-params.conf
   
I think Jeff got lost in the weeds here. If you define a new MCA param in the 
default param file, we will automatically pick it up and it will be in the 
environment of your application. You don’t need to do anything. However, you 
checked for the wrong envar. Anything you provide is going to have an 
“OMPI_MCA_” attached to the front of it. So for your “apath” example, the envar 
will be

OMPI_MCA_apath

HTH
Ralph

> On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
>> 
>> oops, I should use the word "MCA parameters". If I define a MCA parameter 
>> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
>> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I 
>> tried to call getenv("apath") there, which return null. 
> 
> Settings MCA parameters does not necessarily set environment variables.  More 
> below.
> 
>> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
>> could get it in setup_fork.
> 
> There are multiple mechanisms in which MCA params are passed to the back-end 
> MPI processes.
> 
> 1. If they are set via "--mca a b" on the mpirun command line, they are 
> passed back to the launching orteds as part of the launch schema for the job. 
>  I.e., the MCA param names and values are bundled up and sent in the network 
> message to each of the orteds that are involved in the launching of the 
> target processes.  IIRC, the launching orteds will fork, setenv a 
> corresponding environment variable for each of the MCA param values in the 
> launch schema, and then exec the MPI process.  The MPI process then picks up 
> those MCA param values from the environment variables.
> 
> 2. If they are set via the openmpi-mca-params.conf file, they are not sent 
> via network message in the launch schema.  Instead, each Open MPI process 
> (including MPI processes and orteds) reads that file upon startup to get the 
> values (more specifically: all Open MPI processes *always* read this file 
> before processing whatever MCA param values are sent in the launch schema).  
> A consequence: if openmpi-mca-params.conf is not on a network filesystem (and 
> visible to all Open MPI processes on all servers), you need to copy the 
> change you made to the one openmpi-mca-params.conf file to all copies of 
> openmpi-mca-params.conf.
> 
> -
> 
> What are you trying to do?
> 
> If you're working in an MPI application, you can use the MPI_T API to 
> retrieve MCA parameter names and their current values (regardless of whether 
> mechanism #1 or #2 -- or another mechanism -- is used).
> 
> If you're working in the Open MPI code base, you should be using the internal 
> opal_mca_var API to retrieve and/or set MCA param values.  See 
> https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_var.h for 
> all the functions available in this API.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

   ___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[r...@open-mpi.org]

I think Jeff got lost in the weeds here. If you define a new MCA param in the 
default param file, we will automatically pick it up and it will be in the 
environment of your application. You don’t need to do anything. However, you 
checked for the wrong envar. Anything you provide is going to have an 
“OMPI_MCA_” attached to the front of it. So for your “apath” example, the envar 
will be

OMPI_MCA_apath

HTH
Ralph

> On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
>> 
>> oops, I should use the word "MCA parameters". If I define a MCA parameter 
>> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
>> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I 
>> tried to call getenv("apath") there, which return null. 
> 
> Settings MCA parameters does not necessarily set environment variables.  More 
> below.
> 
>> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
>> could get it in setup_fork.
> 
> There are multiple mechanisms in which MCA params are passed to the back-end 
> MPI processes.
> 
> 1. If they are set via "--mca a b" on the mpirun command line, they are 
> passed back to the launching orteds as part of the launch schema for the job. 
>  I.e., the MCA param names and values are bundled up and sent in the network 
> message to each of the orteds that are involved in the launching of the 
> target processes.  IIRC, the launching orteds will fork, setenv a 
> corresponding environment variable for each of the MCA param values in the 
> launch schema, and then exec the MPI process.  The MPI process then picks up 
> those MCA param values from the environment variables.
> 
> 2. If they are set via the openmpi-mca-params.conf file, they are not sent 
> via network message in the launch schema.  Instead, each Open MPI process 
> (including MPI processes and orteds) reads that file upon startup to get the 
> values (more specifically: all Open MPI processes *always* read this file 
> before processing whatever MCA param values are sent in the launch schema).  
> A consequence: if openmpi-mca-params.conf is not on a network filesystem (and 
> visible to all Open MPI processes on all servers), you need to copy the 
> change you made to the one openmpi-mca-params.conf file to all copies of 
> openmpi-mca-params.conf.
> 
> -
> 
> What are you trying to do?
> 
> If you're working in an MPI application, you can use the MPI_T API to 
> retrieve MCA parameter names and their current values (regardless of whether 
> mechanism #1 or #2 -- or another mechanism -- is used).
> 
> If you're working in the Open MPI code base, you should be using the internal 
> opal_mca_var API to retrieve and/or set MCA param values.  See 
> https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_var.h for 
> all the functions available in this API.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Jeff Squyres (jsquyres)]

On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
> 
> oops, I should use the word "MCA parameters". If I define a MCA parameter 
> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I tried 
> to call getenv("apath") there, which return null. 

Settings MCA parameters does not necessarily set environment variables.  More 
below.

> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
> could get it in setup_fork.

There are multiple mechanisms in which MCA params are passed to the back-end 
MPI processes.

1. If they are set via "--mca a b" on the mpirun command line, they are passed 
back to the launching orteds as part of the launch schema for the job.  I.e., 
the MCA param names and values are bundled up and sent in the network message 
to each of the orteds that are involved in the launching of the target 
processes.  IIRC, the launching orteds will fork, setenv a corresponding 
environment variable for each of the MCA param values in the launch schema, and 
then exec the MPI process.  The MPI process then picks up those MCA param 
values from the environment variables.

2. If they are set via the openmpi-mca-params.conf file, they are not sent via 
network message in the launch schema.  Instead, each Open MPI process 
(including MPI processes and orteds) reads that file upon startup to get the 
values (more specifically: all Open MPI processes *always* read this file 
before processing whatever MCA param values are sent in the launch schema).  A 
consequence: if openmpi-mca-params.conf is not on a network filesystem (and 
visible to all Open MPI processes on all servers), you need to copy the change 
you made to the one openmpi-mca-params.conf file to all copies of 
openmpi-mca-params.conf.

-

What are you trying to do?

If you're working in an MPI application, you can use the MPI_T API to retrieve 
MCA parameter names and their current values (regardless of whether mechanism 
#1 or #2 -- or another mechanism -- is used).

If you're working in the Open MPI code base, you should be using the internal 
opal_mca_var API to retrieve and/or set MCA param values.  See 
https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_var.h for 
all the functions available in this API.

-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Dahai Guo via devel]

I mean getenv("OMPI_MCA_apath") in setup_fork.

  From: Dahai Guo via devel 
 To: Jeff Squyres (jsquyres) ; Dahai Guo 
; Open MPI Developers List  
Cc: Dahai Guo 
 Sent: Friday, February 24, 2017 9:11 AM
 Subject: Re: [OMPI devel] define a new ENV variable in 
etc/openmpi-mca-params.conf
   
oops, I should use the word "MCA parameters". If I define a MCA parameter 
(apath, for example) in etc/openmpi-mca-params.conf, how can function 
setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I tried 
to call getenv("apath") there, which return null. 
However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
could get it in setup_fork.
Did I miss something?
Dahai

  From: Jeff Squyres (jsquyres) 
 To: Dahai Guo ; Open MPI Developers List 
 
Cc: Dahai Guo 
 Sent: Friday, February 24, 2017 8:57 AM
 Subject: Re: [OMPI devel] define a new ENV variable in 
etc/openmpi-mca-params.conf
  
On Feb 23, 2017, at 5:44 PM, Dahai Guo via devel  
wrote:
> 
> If I define a new ENV variable in etc/openmpi-mca-params.conf, what OMPI 
> codes should I modify  in order for this parameter to be delivered to each 
> rank?

I'm not quite sure what you're asking.  The openmpi-mca-params.conf does not 
define environment variables; it only sets values for Open MPI's MCA parameters 
(i.e., run-time parameters).

If you want to pass environment variables from the environment of mpirun to the 
MPI processes that it starts, use the -x CLI option to mpirun.  For example:

export FOO=bar
mpirun -x FOO --host a,b -np 2 my_mpi_program

-- 
Jeff Squyres
jsquy...@cisco.com


   
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

   ___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Dahai Guo via devel]

oops, I should use the word "MCA parameters". If I define a MCA parameter 
(apath, for example) in etc/openmpi-mca-params.conf, how can function 
setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I tried 
to call getenv("apath") there, which return null. 
However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
could get it in setup_fork.
Did I miss something?
Dahai

  From: Jeff Squyres (jsquyres) 
 To: Dahai Guo ; Open MPI Developers List 
 
Cc: Dahai Guo 
 Sent: Friday, February 24, 2017 8:57 AM
 Subject: Re: [OMPI devel] define a new ENV variable in 
etc/openmpi-mca-params.conf
   
On Feb 23, 2017, at 5:44 PM, Dahai Guo via devel  
wrote:
> 
> If I define a new ENV variable in etc/openmpi-mca-params.conf, what OMPI 
> codes should I modify  in order for this parameter to be delivered to each 
> rank?

I'm not quite sure what you're asking.  The openmpi-mca-params.conf does not 
define environment variables; it only sets values for Open MPI's MCA parameters 
(i.e., run-time parameters).

If you want to pass environment variables from the environment of mpirun to the 
MPI processes that it starts, use the -x CLI option to mpirun.  For example:

export FOO=bar
mpirun -x FOO --host a,b -np 2 my_mpi_program

-- 
Jeff Squyres
jsquy...@cisco.com


   ___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[Jeff Squyres (jsquyres)]

On Feb 23, 2017, at 5:44 PM, Dahai Guo via devel  
wrote:
> 
> If I define a new ENV variable in etc/openmpi-mca-params.conf, what OMPI 
> codes should I modify  in order for this parameter to be delivered to each 
> rank?

I'm not quite sure what you're asking.  The openmpi-mca-params.conf does not 
define environment variables; it only sets values for Open MPI's MCA parameters 
(i.e., run-time parameters).

If you want to pass environment variables from the environment of mpirun to the 
MPI processes that it starts, use the -x CLI option to mpirun.  For example:

export FOO=bar
mpirun -x FOO --host a,b -np 2 my_mpi_program

-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] ABI Navigator for OpenMPI

[Jeff Squyres (jsquyres)]

That's pretty cool!

One minor request: our project's name is "Open MPI", not "OpenMPI".  Could you 
update your listings to reflect the correct name?

Thank you!

> On Feb 23, 2017, at 11:17 PM, Andrey Ponomarenko 
>  wrote:
> 
> Hello,
> 
> I'd like to present a new project called "ABI Navigator" to search for binary 
> symbols (functions, global data, etc.) in OpenMPI and other open-source 
> libraries: https://abi-laboratory.pro/index.php?view=navigator
> 
> The project allows to find out in which versions of the library some symbol 
> is defined, added, removed or changed. The data is taken from the ABI Tracker 
> project: https://abi-laboratory.pro/tracker/timeline/openmpi/
> 
> Example for symbol MPI_T_category_get_index from libmpi.so: 
> https://abi-laboratory.pro/index.php?view=navigator=MPI_T_category_get_index#result
> 
> The project aims to help library users and maintainers to resolve issues with 
> missed symbols and navigate through the reports in the ABI Tracker.
> 
> Have you ever encountered the "undefined reference" error or want to know 
> whether the symbol is _stable_ enough to import by your code? Try to find it 
> in the ABI Navigator!
> 
> Enjoy!
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel


-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel