Smells like a bug - I'll take a look.
On Aug 16, 2011, at 9:10 AM, Simone Pellegrini wrote: > On 08/16/2011 02:11 PM, Ralph Castain wrote: >> That should work, then. When you set the "host" property, did you give the >> same name as was in your machine file? >> >> Debug options that might help: >> >> -mca plm_base_verbose 5 -mca rmaps_base_verbose 5 >> >> You'll need to configure --enable-debug to get the output, but that should >> help tell us what is happening. > To be clear here is the code I am using to spawn the MPI job: > // create the info object > MPI_Info info; > MPI_Info_create(&info); > MPI_Info_set(info, "host", const_cast<char*>(hostname.c_str())); > LOG(ERROR) << hostname; > LOG(DEBUG) << "Invoking task ID '" << task_id <<"': '" << exec_name << "'"; > > MPI_Comm_spawn( const_cast<char*>(exec_name.c_str()), cargs, num_procs, > info, 0, MPI_COMM_SELF, &intercomm, MPI_ERRCODES_IGNORE ); > > delete[] cargs; > MPI_Info_free(&info); > > and here is the log message: > In this case the MPI_Spaw creates a job with 3 MPI processes. As you can see > MPI_Spawn doesn't care about my "host" setting, it just goes ahead and map > the processes to node b05 and node b06 which are in my machinefile. (which is > the same as before) > > is there any way to overwrite this behaviour? > > DEBUG 14628:R<0> 17:00:13] Spawning new MPI processes... > DEBUG 14628:R<0> 17:00:13] Serving event 'TASK_CREATED', (number of > registered handlers: 1) > ERROR 14628:R<0> 17:00:13] b01 > DEBUG 14628:R<0> 17:00:13] Invoking task ID '4': './simulator' > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive got message > from [[34621,1],0] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive job launch > command > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:rsh: setting up job > [34621,4] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:setup_job for job > [34621,4] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: > created new proc [[34621,4],INVALID] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot > mapping proc in job [34621,4] to node b02 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: adding node b02 > to map > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for > job [34621,4] to node b02 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: > created new proc [[34621,4],INVALID] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot > mapping proc in job [34621,4] to node b01 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: adding node b01 > to map > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for > job [34621,4] to node b01 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: > created new proc [[34621,4],INVALID] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot > mapping proc in job [34621,4] to node b02 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for > job [34621,4] to node b02 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:compute_usage > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons > existing daemon [[34621,0],2] already launched > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons > existing daemon [[34621,0],1] already launched > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:rsh: no new daemons to > launch > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch_apps for job > [34621,4] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:report_launched for > job [34621,4] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > from daemon [[34621,0],0] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > completed processing > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > reissuing non-blocking recv > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > from daemon [[34621,0],1] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched > for proc [[34621,4],1] from daemon [[34621,0],1]: pid 14646 state 2 exit 0 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > completed processing > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > reissuing non-blocking recv > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > from daemon [[34621,0],2] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched > for proc [[34621,4],0] from daemon [[34621,0],2]: pid 9803 state 2 exit 0 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched > for proc [[34621,4],2] from daemon [[34621,0],2]: pid 9804 state 2 exit 0 > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch > completed processing > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:report_launched all > apps reported > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch wiring up iof > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch completed > for job [34621,4] > [kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive job > [34621,4] launched > > cheers, Simone P. >> >> >> On Aug 16, 2011, at 5:09 AM, Simone Pellegrini wrote: >> >>> On 08/16/2011 12:30 PM, Ralph Castain wrote: >>>> What version are you using? >>> OpenMPI 1.4.3 >>> >>>> >>>> On Aug 16, 2011, at 3:19 AM, Simone Pellegrini wrote: >>>> >>>>> Dear all, >>>>> I am developing a system to manage MPI tasks on top of MPI. The >>>>> architecture is rather simple, I have a set of scheduler processes which >>>>> takes care to manage the resources of a node. The idea is to have 1 (or >>>>> more) of those scheduler allocated on each node of a cluster and then >>>>> create new MPI processes (on demand) as computation is needed. Allocation >>>>> of processes is done using MPI_Spawn. >>>>> >>>>> The system now works fine on a single node by allocating the main >>>>> scheduler using the following mpi command: >>>>> mpirun --np 1 ./scheduler ... >>>>> >>>>> Now when I scale to multiple nodes problems with default MPI behaviour >>>>> starts. For example lets assume I have 2 nodes with 8 cpu cores each. I >>>>> therefore set up a machine file in the following way: >>>>> >>>>> s01 slots=1 >>>>> s02 slots=1 >>>>> >>>>> and start the node schedulers in the following way: >>>>> mpirun --np 2 --machinefile machinefile ./scheduler ... >>>>> >>>>> This allocates the processes correctly, now the problem starts when I >>>>> invoke MPI_Spawn. basically MPI spawn also uses the informations from the >>>>> machinefile and if 4 MPI processes are spawned 2 are allocated in s01 and >>>>> 2 on s02. What I want is to allocate the processes always in the same >>>>> node. >>>>> >>>>> I tried to do this by specifying an MPI_Info object which is then passed >>>>> to the MPI_Spawn routine. I tried to set the "host" property to the >>>>> hostname of the machine where the scheduler is running but this didn't >>>>> help. >>>>> >>>>> Unfortunately there is very little documentation on this. >>>>> >>>>> Thanks for the help, >>>>> Simone >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >
