Ralph Castain ha scritto: > Hi Roberto > > There is something wrong with this cmd line - perhaps it wasn't copied > correctly? > > mpirun --verbose --debug-daemons --mca obl -np 1 -wdir `pwd` > testmaster 10000 $PBS_NODEFILE > > Specifically, the following is incomplete: --mca obl > > I'm not sure if this is the problem or not, but I am unaware of such > an option and believe it could cause mpirun to become confused. Ops! Sorry, I copied the wrong log, below there the right one:
[roberto@master TestOpenMPI]$ qsub -I testmaster.pbs qsub: waiting for job 920.master.tekno-soft.it to start qsub: job 920.master.tekno-soft.it ready [roberto@master TestMPICH2]$ cd /data/roberto/MPI/TestOpenMPI/ [roberto@master TestOpenMPI]$ mpirun --debug-daemons --mca btl tcp,self -wdir "`pwd`" -np 1 testmaster 100000 $PBS_NODEFILE [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received add_local_procs [master.tekno-soft.it:05407] [[11340,0],0] node[0].name master daemon 0 arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[1].name cluster4 daemon INVALID arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[2].name cluster3 daemon INVALID arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[3].name cluster2 daemon INVALID arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[4].name cluster1 daemon INVALID arch ffc91200 Initializing MPI ... [master.tekno-soft.it:05407] [[11340,0],0] orted_recv: received sync+nidmap from local proc [[11340,1],0] [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received collective data cmd [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received message_local_procs [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received collective data cmd [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received message_local_procs Loading the node's ring from file '/var/torque/aux//920.master.tekno-soft.it' ... adding node #1 host is 'cluster4.tekno-soft.it' ... adding node #2 host is 'cluster3.tekno-soft.it' ... adding node #3 host is 'cluster2.tekno-soft.it' ... adding node #4 host is 'cluster1.tekno-soft.it' A 4 node's ring has been made At least one node is available, let's start to distribute 100000 job across 4 nodes!!! ****************** Starting job #1 ****************** Starting job #2 ****************** Starting job #3 Setting up the host as 'cluster4.tekno-soft.it' Setting the work directory as '/data/roberto/MPI/TestOpenMPI' Spawning a task 'testslave' on node 'cluster4.tekno-soft.it' Setting up the host as 'cluster3.tekno-soft.it' Setting the work directory as '/data/roberto/MPI/TestOpenMPI' Spawning a task 'testslave' on node 'cluster3.tekno-soft.it' Setting up the host as 'cluster2.tekno-soft.it' Setting the work directory as '/data/roberto/MPI/TestOpenMPI' Spawning a task 'testslave' on node 'cluster2.tekno-soft.it' Setting up the host as 'cluster1.tekno-soft.it' Setting the work directory as '/data/roberto/MPI/TestOpenMPI' Spawning a task 'testslave' on node 'cluster1.tekno-soft.it' ****************** Starting job #4 Daemon was launched on cluster3.tekno-soft.it - beginning to initialize Daemon [[11340,0],1] checking in as pid 9487 on host cluster3.tekno-soft.it Daemon [[11340,0],1] not using static ports -------------------------------------------------------------------------- A daemon (pid unknown) died unexpectedly on signal 1 while attempting to launch so we are aborting. There may be more information reported by the environment (see above). This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes. -------------------------------------------------------------------------- [master.tekno-soft.it:05407] [[11340,0],0] ORTE_ERROR_LOG: Resource busy in file base/plm_base_receive.c at line 169 [master.tekno-soft.it:05414] [[11340,1],0] ORTE_ERROR_LOG: The specified application failed to start in file dpm_orte.c at line 677 [master.tekno-soft.it:05414] *** An error occurred in MPI_Comm_spawn [master.tekno-soft.it:05414] *** on communicator MPI_COMM_WORLD [master.tekno-soft.it:05414] *** MPI_ERR_SPAWN: could not spawn processes [master.tekno-soft.it:05414] *** MPI_ERRORS_ARE_FATAL (goodbye) [master.tekno-soft.it:05407] [[11340,0],0] ORTE_ERROR_LOG: Resource busy in file base/plm_base_receive.c at line 169 [master.tekno-soft.it:05407] [[11340,0],0] orted_cmd: received add_local_procs [master.tekno-soft.it:05407] [[11340,0],0] node[0].name master daemon 0 arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[1].name cluster4 daemon INVALID arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[2].name cluster3 daemon 1 arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[3].name cluster2 daemon INVALID arch ffc91200 [master.tekno-soft.it:05407] [[11340,0],0] node[4].name cluster1 daemon INVALID arch ffc91200 [cluster3.tekno-soft.it:09487] [[11340,0],1] orted: up and running - waiting for commands! > > Ralph > > > On Sep 30, 2008, at 8:24 AM, Roberto Fichera wrote: > >> Roberto Fichera ha scritto: >>> Hi All on the list, >>> >>> I'm trying to execute dynamic MPI applications using MPI_Comm_spawn(). >>> The application I'm using for tests, basically is >>> composed by a master, which spawn a slave in each assigned node in a >>> multithreading fashion. The master is started with a >>> number of jobs to perform and a filename, containing the list of >>> assigned nodes. The idea is to handle all the dispatching >>> logic within the application, so that the master will try to take as >>> busy as possible each assigned node. Said that, for each spawned >>> job, the master allocate a thread for spawning and handling the >>> communication, than generate a random number, send it to the >>> slave which simply send it back to the master. Finally the slave >>> terminate its job and the relative node become free for a new one. >>> The things will continue until all the requested jobs are done. >>> >>> The test program I'm using *doesn't* work flawless in mpich2 because it >>> has a ~24k spawned job limitation, due to a monotonically >>> increasing of its internal context id which basically stops the >>> application due to a library internal overflow. The internal context >>> id, >>> allocated >>> for each terminated spawned job, are never recycled at moment. The >>> unique MPI-2 implementation, so supporting MPI_Comm_spawn(), >>> which was able to complete the test is currently the HP MPI. So now I >>> would start to check OpenMPI if it's suitable for our dynamic parallel >>> applications. >>> >>> The test application is linked against OpenMPI v1.3a1r19645, running of >>> Fedora8 x86_64 + all updates. >>> >>> My first attempt end up on the error below which I basically don't know >>> where to look further. Note that I've already checked PATHs and >>> LD_LIBRARY_PATH, the application is basically configured correctly >>> since >>> it uses two scripts for starting and all the paths are set there. >>> Basically I need to start *one* master application which will handle >>> all >>> the things for managing slave applications. The communication is *only* >>> master <-> slave and never collective, at moment. >>> >>> The test program is available on request. >>> >>> Does any one have an idea what's going on? >>> >>> Thanks in advance, >>> Roberto Fichera. >>> >>> [roberto@cluster4 TestOpenMPI]$ orterun -wdir >>> /data/roberto/MPI/TestOpenMPI -np >>> 1 testmaster 10000 $PBS_NODEFILE >>> Initializing MPI ... >>> Loading the node's ring from file >>> '/var/torque/aux//909.master.tekno-soft.it' >>> ... adding node #1 host is 'cluster3.tekno-soft.it' >>> ... adding node #2 host is 'cluster2.tekno-soft.it' >>> ... adding node #3 host is 'cluster1.tekno-soft.it' >>> ... adding node #4 host is 'master.tekno-soft.it' >>> A 4 node's ring has been made >>> At least one node is available, let's start to distribute 10000 job >>> across 4 >>> nodes!!! >>> ****************** Starting job #1 >>> ****************** Starting job #2 >>> ****************** Starting job #3 >>> ****************** Starting job #4 >>> Setting up the host as 'cluster3.tekno-soft.it' >>> Setting the work directory as '/data/roberto/MPI/TestOpenMPI' >>> Spawning a task './testslave.sh' on node 'cluster3.tekno-soft.it' >>> Setting up the host as 'cluster2.tekno-soft.it' >>> Setting the work directory as '/data/roberto/MPI/TestOpenMPI' >>> Spawning a task './testslave.sh' on node 'cluster2.tekno-soft.it' >>> Setting up the host as 'cluster1.tekno-soft.it' >>> Setting the work directory as '/data/roberto/MPI/TestOpenMPI' >>> Spawning a task './testslave.sh' on node 'cluster1.tekno-soft.it' >>> Setting up the host as 'master.tekno-soft.it' >>> Setting the work directory as '/data/roberto/MPI/TestOpenMPI' >>> Spawning a task './testslave.sh' on node 'master.tekno-soft.it' >>> -------------------------------------------------------------------------- >>> >>> A daemon (pid unknown) died unexpectedly on signal 1 while >>> attempting to >>> launch so we are aborting. >>> >>> There may be more information reported by the environment (see above). >>> >>> This may be because the daemon was unable to find all the needed shared >>> libraries on the remote node. You may set your LD_LIBRARY_PATH to >>> have the >>> location of the shared libraries on the remote nodes and this will >>> automatically be forwarded to the remote nodes. >>> -------------------------------------------------------------------------- >>> >>> [cluster4.tekno-soft.it:21287] [[30014,0],0] ORTE_ERROR_LOG: >>> Resource busy in >>> file base/plm_base_receive.c at line 169 >>> [cluster4.tekno-soft.it:21287] [[30014,0],0] ORTE_ERROR_LOG: >>> Resource busy in >>> file base/plm_base_receive.c at line 169 >>>
