In the event that I need to get this up-and-running soon (I do need something working within 2 weeks), can you recommend an older version where this is expected to work?
Thanks, Brian On Tue, Aug 28, 2012 at 4:58 PM, Brian Budge <brian.bu...@gmail.com> wrote: > Thanks! > > On Tue, Aug 28, 2012 at 4:57 PM, Ralph Castain <r...@open-mpi.org> wrote: >> Yeah, I'm seeing the hang as well when running across multiple machines. Let >> me dig a little and get this fixed. >> >> Thanks >> Ralph >> >> On Aug 28, 2012, at 4:51 PM, Brian Budge <brian.bu...@gmail.com> wrote: >> >>> Hmmm, I went to the build directories of openmpi for my two machines, >>> went into the orte/test/mpi directory and made the executables on both >>> machines. I set the hostsfile in the env variable on the "master" >>> machine. >>> >>> Here's the output: >>> >>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile >>> ./simple_spawn >>> Parent [pid 97504] starting up! >>> 0 completed MPI_Init >>> Parent [pid 97504] about to spawn! >>> Parent [pid 97507] starting up! >>> Parent [pid 97508] starting up! >>> Parent [pid 30626] starting up! >>> ^C >>> zsh: interrupt OMPI_MCA_orte_default_hostfile= ./simple_spawn >>> >>> I had to ^C to kill the hung process. >>> >>> When I run using mpirun: >>> >>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile >>> mpirun -np 1 ./simple_spawn >>> Parent [pid 97511] starting up! >>> 0 completed MPI_Init >>> Parent [pid 97511] about to spawn! >>> Parent [pid 97513] starting up! >>> Parent [pid 30762] starting up! >>> Parent [pid 30764] starting up! >>> Parent done with spawn >>> Parent sending message to child >>> 1 completed MPI_Init >>> Hello from the child 1 of 3 on host budgeb-sandybridge pid 97513 >>> 0 completed MPI_Init >>> Hello from the child 0 of 3 on host budgeb-interlagos pid 30762 >>> 2 completed MPI_Init >>> Hello from the child 2 of 3 on host budgeb-interlagos pid 30764 >>> Child 1 disconnected >>> Child 0 received msg: 38 >>> Child 0 disconnected >>> Parent disconnected >>> Child 2 disconnected >>> 97511: exiting >>> 97513: exiting >>> 30762: exiting >>> 30764: exiting >>> >>> As you can see, I'm using openmpi v 1.6.1. I just barely freshly >>> installed on both machines using the default configure options. >>> >>> Thanks for all your help. >>> >>> Brian >>> >>> On Tue, Aug 28, 2012 at 4:39 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>> Looks to me like it didn't find your executable - could be a question of >>>> where it exists relative to where you are running. If you look in your >>>> OMPI source tree at the orte/test/mpi directory, you'll see an example >>>> program "simple_spawn.c" there. Just "make simple_spawn" and execute that >>>> with your default hostfile set - does it work okay? >>>> >>>> It works fine for me, hence the question. >>>> >>>> Also, what OMPI version are you using? >>>> >>>> On Aug 28, 2012, at 4:25 PM, Brian Budge <brian.bu...@gmail.com> wrote: >>>> >>>>> I see. Okay. So, I just tried removing the check for universe size, >>>>> and set the universe size to 2. Here's my output: >>>>> >>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib >>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe >>>>> [budgeb-interlagos:29965] [[4156,0],0] ORTE_ERROR_LOG: Fatal in file >>>>> base/plm_base_receive.c at line 253 >>>>> [budgeb-interlagos:29963] [[4156,1],0] ORTE_ERROR_LOG: The specified >>>>> application failed to start in file dpm_orte.c at line 785 >>>>> >>>>> The corresponding run with mpirun still works. >>>>> >>>>> Thanks, >>>>> Brian >>>>> >>>>> On Tue, Aug 28, 2012 at 2:46 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>> I see the issue - it's here: >>>>>> >>>>>>> MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize, &flag); >>>>>>> >>>>>>> if(!flag) { >>>>>>> std::cerr << "no universe size" << std::endl; >>>>>>> return -1; >>>>>>> } >>>>>>> universeSize = *puniverseSize; >>>>>>> if(universeSize == 1) { >>>>>>> std::cerr << "cannot start slaves... not enough nodes" << std::endl; >>>>>>> } >>>>>> >>>>>> The universe size is set to 1 on a singleton because the attribute gets >>>>>> set at the beginning of time - we haven't any way to go back and change >>>>>> it. The sequence of events explains why. The singleton starts up and >>>>>> sets its attributes, including universe_size. It also spins off an orte >>>>>> daemon to act as its own private "mpirun" in case you call comm_spawn. >>>>>> At this point, however, no hostfile has been read - the singleton is >>>>>> just an MPI proc doing its own thing, and the orte daemon is just >>>>>> sitting there on "stand-by". >>>>>> >>>>>> When your app calls comm_spawn, then the orte daemon gets called to >>>>>> launch the new procs. At that time, it (not the original singleton!) >>>>>> reads the hostfile to find out how many nodes are around, and then does >>>>>> the launch. >>>>>> >>>>>> You are trying to check the number of nodes from within the singleton, >>>>>> which won't work - it has no way of discovering that info. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Aug 28, 2012, at 2:38 PM, Brian Budge <brian.bu...@gmail.com> wrote: >>>>>> >>>>>>>> echo hostsfile >>>>>>> localhost >>>>>>> budgeb-sandybridge >>>>>>> >>>>>>> Thanks, >>>>>>> Brian >>>>>>> >>>>>>> On Tue, Aug 28, 2012 at 2:36 PM, Ralph Castain <r...@open-mpi.org> >>>>>>> wrote: >>>>>>>> Hmmm...what is in your "hostsfile"? >>>>>>>> >>>>>>>> On Aug 28, 2012, at 2:33 PM, Brian Budge <brian.bu...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Ralph - >>>>>>>>> >>>>>>>>> Thanks for confirming this is possible. I'm trying this and currently >>>>>>>>> failing. Perhaps there's something I'm missing in the code to make >>>>>>>>> this work. Here are the two instantiations and their outputs: >>>>>>>>> >>>>>>>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib >>>>>>>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe >>>>>>>>> cannot start slaves... not enough nodes >>>>>>>>> >>>>>>>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib >>>>>>>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile mpirun -n 1 >>>>>>>>>> ./master_exe >>>>>>>>> master spawned 1 slaves... >>>>>>>>> slave responding... >>>>>>>>> >>>>>>>>> >>>>>>>>> The code: >>>>>>>>> >>>>>>>>> //master.cpp >>>>>>>>> #include <mpi.h> >>>>>>>>> #include <boost/filesystem.hpp> >>>>>>>>> #include <iostream> >>>>>>>>> >>>>>>>>> int main(int argc, char **args) { >>>>>>>>> int worldSize, universeSize, *puniverseSize, flag; >>>>>>>>> >>>>>>>>> MPI_Comm everyone; //intercomm >>>>>>>>> boost::filesystem::path curPath = >>>>>>>>> boost::filesystem::absolute(boost::filesystem::current_path()); >>>>>>>>> >>>>>>>>> std::string toRun = (curPath / "slave_exe").string(); >>>>>>>>> >>>>>>>>> int ret = MPI_Init(&argc, &args); >>>>>>>>> >>>>>>>>> if(ret != MPI_SUCCESS) { >>>>>>>>> std::cerr << "failed init" << std::endl; >>>>>>>>> return -1; >>>>>>>>> } >>>>>>>>> >>>>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &worldSize); >>>>>>>>> >>>>>>>>> if(worldSize != 1) { >>>>>>>>> std::cerr << "too many masters" << std::endl; >>>>>>>>> } >>>>>>>>> >>>>>>>>> MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize, >>>>>>>>> &flag); >>>>>>>>> >>>>>>>>> if(!flag) { >>>>>>>>> std::cerr << "no universe size" << std::endl; >>>>>>>>> return -1; >>>>>>>>> } >>>>>>>>> universeSize = *puniverseSize; >>>>>>>>> if(universeSize == 1) { >>>>>>>>> std::cerr << "cannot start slaves... not enough nodes" << >>>>>>>>> std::endl; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> char *buf = (char*)alloca(toRun.size() + 1); >>>>>>>>> memcpy(buf, toRun.c_str(), toRun.size()); >>>>>>>>> buf[toRun.size()] = '\0'; >>>>>>>>> >>>>>>>>> MPI_Comm_spawn(buf, MPI_ARGV_NULL, universeSize-1, MPI_INFO_NULL, >>>>>>>>> 0, MPI_COMM_SELF, &everyone, >>>>>>>>> MPI_ERRCODES_IGNORE); >>>>>>>>> >>>>>>>>> std::cerr << "master spawned " << universeSize-1 << " slaves..." >>>>>>>>> << std::endl; >>>>>>>>> >>>>>>>>> MPI_Finalize(); >>>>>>>>> >>>>>>>>> return 0; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> //slave.cpp >>>>>>>>> #include <mpi.h> >>>>>>>>> >>>>>>>>> int main(int argc, char **args) { >>>>>>>>> int size; >>>>>>>>> MPI_Comm parent; >>>>>>>>> MPI_Init(&argc, &args); >>>>>>>>> >>>>>>>>> MPI_Comm_get_parent(&parent); >>>>>>>>> >>>>>>>>> if(parent == MPI_COMM_NULL) { >>>>>>>>> std::cerr << "slave has no parent" << std::endl; >>>>>>>>> } >>>>>>>>> MPI_Comm_remote_size(parent, &size); >>>>>>>>> if(size != 1) { >>>>>>>>> std::cerr << "parent size is " << size << std::endl; >>>>>>>>> } >>>>>>>>> >>>>>>>>> std::cerr << "slave responding..." << std::endl; >>>>>>>>> >>>>>>>>> MPI_Finalize(); >>>>>>>>> >>>>>>>>> return 0; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Any ideas? Thanks for any help. >>>>>>>>> >>>>>>>>> Brian >>>>>>>>> >>>>>>>>> On Wed, Aug 22, 2012 at 9:03 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>>> It really is just that simple :-) >>>>>>>>>> >>>>>>>>>> On Aug 22, 2012, at 8:56 AM, Brian Budge <brian.bu...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Okay. Is there a tutorial or FAQ for setting everything up? Or is >>>>>>>>>>> it >>>>>>>>>>> really just that simple? I don't need to run a copy of the orte >>>>>>>>>>> server somewhere? >>>>>>>>>>> >>>>>>>>>>> if my current ip is 192.168.0.1, >>>>>>>>>>> >>>>>>>>>>> 0 > echo 192.168.0.11 > /tmp/hostfile >>>>>>>>>>> 1 > echo 192.168.0.12 >> /tmp/hostfile >>>>>>>>>>> 2 > export OMPI_MCA_orte_default_hostfile=/tmp/hostfile >>>>>>>>>>> 3 > ./mySpawningExe >>>>>>>>>>> >>>>>>>>>>> At this point, mySpawningExe will be the master, running on >>>>>>>>>>> 192.168.0.1, and I can have spawned, for example, childExe on >>>>>>>>>>> 192.168.0.11 and 192.168.0.12? Or childExe1 on 192.168.0.11 and >>>>>>>>>>> childExe2 on 192.168.0.12? >>>>>>>>>>> >>>>>>>>>>> Thanks for the help. >>>>>>>>>>> >>>>>>>>>>> Brian >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 22, 2012 at 7:15 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>>> wrote: >>>>>>>>>>>> Sure, that's still true on all 1.3 or above releases. All you need >>>>>>>>>>>> to do is set the hostfile envar so we pick it up: >>>>>>>>>>>> >>>>>>>>>>>> OMPI_MCA_orte_default_hostfile=<foo> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Aug 21, 2012, at 7:23 PM, Brian Budge <brian.bu...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi. I know this is an old thread, but I'm curious if there are >>>>>>>>>>>>> any >>>>>>>>>>>>> tutorials describing how to set this up? Is this still available >>>>>>>>>>>>> on >>>>>>>>>>>>> newer open mpi versions? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Brian >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jan 4, 2008 at 7:57 AM, Ralph Castain <r...@lanl.gov> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Hi Elena >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm copying this to the user list just to correct a >>>>>>>>>>>>>> mis-statement on my part >>>>>>>>>>>>>> in an earlier message that went there. I had stated that a >>>>>>>>>>>>>> singleton could >>>>>>>>>>>>>> comm_spawn onto other nodes listed in a hostfile by setting an >>>>>>>>>>>>>> environmental >>>>>>>>>>>>>> variable that pointed us to the hostfile. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is incorrect in the 1.2 code series. That series does not >>>>>>>>>>>>>> allow >>>>>>>>>>>>>> singletons to read a hostfile at all. Hence, any comm_spawn done >>>>>>>>>>>>>> by a >>>>>>>>>>>>>> singleton can only launch child processes on the singleton's >>>>>>>>>>>>>> local host. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This situation has been corrected for the upcoming 1.3 code >>>>>>>>>>>>>> series. For the >>>>>>>>>>>>>> 1.2 series, though, you will have to do it via an mpirun command >>>>>>>>>>>>>> line. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for the confusion - I sometimes have too many code >>>>>>>>>>>>>> families to keep >>>>>>>>>>>>>> straight in this old mind! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ralph >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 1/4/08 5:10 AM, "Elena Zhebel" <ezhe...@fugro-jason.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello Ralph, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you very much for the explanations. >>>>>>>>>>>>>>> But I still do not get it running... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For the case >>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host >>>>>>>>>>>>>>> my_master.exe >>>>>>>>>>>>>>> everything works. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For the case >>>>>>>>>>>>>>> ./my_master.exe >>>>>>>>>>>>>>> it does not. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I did: >>>>>>>>>>>>>>> - create my_hostfile and put it in the >>>>>>>>>>>>>>> $HOME/.openmpi/components/ >>>>>>>>>>>>>>> my_hostfile : >>>>>>>>>>>>>>> bollenstreek slots=2 max_slots=3 >>>>>>>>>>>>>>> octocore01 slots=8 max_slots=8 >>>>>>>>>>>>>>> octocore02 slots=8 max_slots=8 >>>>>>>>>>>>>>> clstr000 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr001 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr002 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr003 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr004 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr005 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr006 slots=2 max_slots=3 >>>>>>>>>>>>>>> clstr007 slots=2 max_slots=3 >>>>>>>>>>>>>>> - setenv OMPI_MCA_rds_hostfile_path my_hostfile (I put it in >>>>>>>>>>>>>>> .tcshrc and >>>>>>>>>>>>>>> then source .tcshrc) >>>>>>>>>>>>>>> - in my_master.cpp I did >>>>>>>>>>>>>>> MPI_Info info1; >>>>>>>>>>>>>>> MPI_Info_create(&info1); >>>>>>>>>>>>>>> char* hostname = >>>>>>>>>>>>>>> "clstr002,clstr003,clstr005,clstr006,clstr007,octocore01,octocore02"; >>>>>>>>>>>>>>> MPI_Info_set(info1, "host", hostname); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _intercomm = intracomm.Spawn("./childexe", argv1, _nProc, >>>>>>>>>>>>>>> info1, 0, >>>>>>>>>>>>>>> MPI_ERRCODES_IGNORE); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - After I call the executable, I've got this error message >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> bollenstreek: > ./my_master >>>>>>>>>>>>>>> number of processes to run: 1 >>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>> Some of the requested hosts are not included in the current >>>>>>>>>>>>>>> allocation for >>>>>>>>>>>>>>> the application: >>>>>>>>>>>>>>> ./childexe >>>>>>>>>>>>>>> The requested hosts were: >>>>>>>>>>>>>>> clstr002,clstr003,clstr005,clstr006,clstr007,octocore01,octocore02 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Verify that you have mapped the allocated resources properly >>>>>>>>>>>>>>> using the >>>>>>>>>>>>>>> --host specification. >>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of resource in >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>> base/rmaps_base_support_fns.c at line 225 >>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of resource in >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>> rmaps_rr.c at line 478 >>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of resource in >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>> base/rmaps_base_map_job.c at line 210 >>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of resource in >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>> rmgr_urm.c at line 372 >>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of resource in >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>> communicator/comm_dyn.c at line 608 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Did I miss something? >>>>>>>>>>>>>>> Thanks for help! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Elena >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>> From: Ralph H Castain [mailto:r...@lanl.gov] >>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2007 3:50 PM >>>>>>>>>>>>>>> To: Elena Zhebel; Open MPI Users <us...@open-mpi.org> >>>>>>>>>>>>>>> Cc: Ralph H Castain >>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and cluster >>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 12/18/07 7:35 AM, "Elena Zhebel" <ezhe...@fugro-jason.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks a lot! Now it works! >>>>>>>>>>>>>>>> The solution is to use mpirun -n 1 -hostfile my.hosts *.exe >>>>>>>>>>>>>>>> and pass >>>>>>>>>>>>>>> MPI_Info >>>>>>>>>>>>>>>> Key to the Spawn function! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> One more question: is it necessary to start my "master" >>>>>>>>>>>>>>>> program with >>>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host >>>>>>>>>>>>>>>> my_master.exe ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No, it isn't necessary - assuming that my_master_host is the >>>>>>>>>>>>>>> first host >>>>>>>>>>>>>>> listed in your hostfile! If you are only executing one >>>>>>>>>>>>>>> my_master.exe (i.e., >>>>>>>>>>>>>>> you gave -n 1 to mpirun), then we will automatically map that >>>>>>>>>>>>>>> process onto >>>>>>>>>>>>>>> the first host in your hostfile. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you want my_master.exe to go on someone other than the first >>>>>>>>>>>>>>> host in the >>>>>>>>>>>>>>> file, then you have to give us the -host option. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Are there other possibilities for easy start? >>>>>>>>>>>>>>>> I would say just to run ./my_master.exe , but then the master >>>>>>>>>>>>>>>> process >>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>> know about the available in the network hosts. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can set the hostfile parameter in your environment instead >>>>>>>>>>>>>>> of on the >>>>>>>>>>>>>>> command line. Just set OMPI_MCA_rds_hostfile_path = my.hosts. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can then just run ./my_master.exe on the host where you >>>>>>>>>>>>>>> want the master >>>>>>>>>>>>>>> to reside - everything should work the same. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just as an FYI: the name of that environmental variable is >>>>>>>>>>>>>>> going to change >>>>>>>>>>>>>>> in the 1.3 release, but everything will still work the same. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hope that helps >>>>>>>>>>>>>>> Ralph >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks and regards, >>>>>>>>>>>>>>>> Elena >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>> From: Ralph H Castain [mailto:r...@lanl.gov] >>>>>>>>>>>>>>>> Sent: Monday, December 17, 2007 5:49 PM >>>>>>>>>>>>>>>> To: Open MPI Users <us...@open-mpi.org>; Elena Zhebel >>>>>>>>>>>>>>>> Cc: Ralph H Castain >>>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and cluster >>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12/17/07 8:19 AM, "Elena Zhebel" <ezhe...@fugro-jason.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello Ralph, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you for your answer. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm using OpenMPI 1.2.3. , compiler glibc232, Linux Suse 10.0. >>>>>>>>>>>>>>>>> My "master" executable runs only on the one local host, then >>>>>>>>>>>>>>>>> it spawns >>>>>>>>>>>>>>>>> "slaves" (with MPI::Intracomm::Spawn). >>>>>>>>>>>>>>>>> My question was: how to determine the hosts where these >>>>>>>>>>>>>>>>> "slaves" will be >>>>>>>>>>>>>>>>> spawned? >>>>>>>>>>>>>>>>> You said: "You have to specify all of the hosts that can be >>>>>>>>>>>>>>>>> used by >>>>>>>>>>>>>>>>> your job >>>>>>>>>>>>>>>>> in the original hostfile". How can I specify the host file? I >>>>>>>>>>>>>>>>> can not >>>>>>>>>>>>>>>>> find it >>>>>>>>>>>>>>>>> in the documentation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hmmm...sorry about the lack of documentation. I always assumed >>>>>>>>>>>>>>>> that the MPI >>>>>>>>>>>>>>>> folks in the project would document such things since it has >>>>>>>>>>>>>>>> little to do >>>>>>>>>>>>>>>> with the underlying run-time, but I guess that fell through >>>>>>>>>>>>>>>> the cracks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are two parts to your question: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. how to specify the hosts to be used for the entire job. I >>>>>>>>>>>>>>>> believe that >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> somewhat covered here: >>>>>>>>>>>>>>>> http://www.open-mpi.org/faq/?category=running#simple-spmd-run >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That FAQ tells you what a hostfile should look like, though >>>>>>>>>>>>>>>> you may already >>>>>>>>>>>>>>>> know that. Basically, we require that you list -all- of the >>>>>>>>>>>>>>>> nodes that both >>>>>>>>>>>>>>>> your master and slave programs will use. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. how to specify which nodes are available for the master, >>>>>>>>>>>>>>>> and which for >>>>>>>>>>>>>>>> the slave. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You would specify the host for your master on the mpirun >>>>>>>>>>>>>>>> command line with >>>>>>>>>>>>>>>> something like: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host >>>>>>>>>>>>>>>> my_master.exe >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This directs Open MPI to map that specified executable on the >>>>>>>>>>>>>>>> specified >>>>>>>>>>>>>>> host >>>>>>>>>>>>>>>> - note that my_master_host must have been in my_hostfile. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Inside your master, you would create an MPI_Info key "host" >>>>>>>>>>>>>>>> that has a >>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>> consisting of a string "host1,host2,host3" identifying the >>>>>>>>>>>>>>>> hosts you want >>>>>>>>>>>>>>>> your slave to execute upon. Those hosts must have been >>>>>>>>>>>>>>>> included in >>>>>>>>>>>>>>>> my_hostfile. Include that key in the MPI_Info array passed to >>>>>>>>>>>>>>>> your Spawn. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We don't currently support providing a hostfile for the slaves >>>>>>>>>>>>>>>> (as opposed >>>>>>>>>>>>>>>> to the host-at-a-time string above). This may become available >>>>>>>>>>>>>>>> in a future >>>>>>>>>>>>>>>> release - TBD. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hope that helps >>>>>>>>>>>>>>>> Ralph >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks and regards, >>>>>>>>>>>>>>>>> Elena >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>> From: users-boun...@open-mpi.org >>>>>>>>>>>>>>>>> [mailto:users-boun...@open-mpi.org] On >>>>>>>>>>>>>>>>> Behalf Of Ralph H Castain >>>>>>>>>>>>>>>>> Sent: Monday, December 17, 2007 3:31 PM >>>>>>>>>>>>>>>>> To: Open MPI Users <us...@open-mpi.org> >>>>>>>>>>>>>>>>> Cc: Ralph H Castain >>>>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and cluster >>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 12/12/07 5:46 AM, "Elena Zhebel" <ezhe...@fugro-jason.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm working on a MPI application where I'm using OpenMPI >>>>>>>>>>>>>>>>>> instead of >>>>>>>>>>>>>>>>>> MPICH. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> In my "master" program I call the function >>>>>>>>>>>>>>>>>> MPI::Intracomm::Spawn which >>>>>>>>>>>>>>>>> spawns >>>>>>>>>>>>>>>>>> "slave" processes. It is not clear for me how to spawn the >>>>>>>>>>>>>>>>>> "slave" >>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>> over the network. Currently "master" creates "slaves" on the >>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>> host. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If I use 'mpirun --hostfile openmpi.hosts' then processes >>>>>>>>>>>>>>>>>> are spawn >>>>>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> network as expected. But now I need to spawn processes over >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> network >>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>> my own executable using MPI::Intracomm::Spawn, how can I >>>>>>>>>>>>>>>>>> achieve it? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm not sure from your description exactly what you are >>>>>>>>>>>>>>>>> trying to do, >>>>>>>>>>>>>>>>> nor in >>>>>>>>>>>>>>>>> what environment this is all operating within or what version >>>>>>>>>>>>>>>>> of Open >>>>>>>>>>>>>>>>> MPI >>>>>>>>>>>>>>>>> you are using. Setting aside the environment and version >>>>>>>>>>>>>>>>> issue, I'm >>>>>>>>>>>>>>>>> guessing >>>>>>>>>>>>>>>>> that you are running your executable over some specified set >>>>>>>>>>>>>>>>> of hosts, >>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>> want to provide a different hostfile that specifies the hosts >>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>> used for >>>>>>>>>>>>>>>>> the "slave" processes. Correct? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If that is correct, then I'm afraid you can't do that in any >>>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>>>> of Open >>>>>>>>>>>>>>>>> MPI today. You have to specify all of the hosts that can be >>>>>>>>>>>>>>>>> used by >>>>>>>>>>>>>>>>> your job >>>>>>>>>>>>>>>>> in the original hostfile. You can then specify a subset of >>>>>>>>>>>>>>>>> those hosts >>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>> used by your original "master" program, and then specify a >>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>> subset >>>>>>>>>>>>>>>>> to be used by the "slaves" when calling Spawn. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> But the system requires that you tell it -all- of the hosts >>>>>>>>>>>>>>>>> that are >>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>> to be used at the beginning of the job. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> At the moment, there is no plan to remove that requirement, >>>>>>>>>>>>>>>>> though >>>>>>>>>>>>>>>>> there has >>>>>>>>>>>>>>>>> been occasional discussion about doing so at some point in >>>>>>>>>>>>>>>>> the future. >>>>>>>>>>>>>>>>> No >>>>>>>>>>>>>>>>> promises that it will happen, though - managed environments, >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> particular, >>>>>>>>>>>>>>>>> currently object to the idea of changing the allocation >>>>>>>>>>>>>>>>> on-the-fly. We >>>>>>>>>>>>>>>>> may, >>>>>>>>>>>>>>>>> though, make a provision for purely hostfile-based >>>>>>>>>>>>>>>>> environments (i.e., >>>>>>>>>>>>>>>>> unmanaged) at some time in the future. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ralph >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks in advance for any help. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Elena >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> users mailing list >>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users