Try manually specifying the collective component "-mca coll tuned" You seem to be using the "sync" collective component, any stale mca param files lying around ?
--Nysal On Tue, Jan 11, 2011 at 6:28 PM, Doron Shoham <doron.o...@gmail.com> wrote: > Hi > > All machines on the setup are IDataPlex with Nehalem 12 cores per node, > 24GB memory. > > > > · *Problem 1 – OMPI 1.4.3 hangs in gather:* > > > > I’m trying to run IMB and gather operation with OMPI 1.4.3 (Vanilla). > > It happens when np >= 64 and message size exceed 4k: > > mpirun -np 64 -machinefile voltairenodes -mca btl sm,self,openib > imb/src-1.4.2/IMB-MPI1 gather –npmin 64 > > > > voltairenodes consists of 64 machines. > > > > #---------------------------------------------------------------- > > # Benchmarking Gather > > # #processes = 64 > > #---------------------------------------------------------------- > > #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] > > 0 1000 0.02 0.02 0.02 > > 1 331 14.02 14.16 14.09 > > 2 331 12.87 13.08 12.93 > > 4 331 14.29 14.43 14.34 > > 8 331 16.03 16.20 16.11 > > 16 331 17.54 17.74 17.64 > > 32 331 20.49 20.62 20.53 > > 64 331 23.57 23.84 23.70 > > 128 331 28.02 28.35 28.18 > > 256 331 34.78 34.88 34.80 > > 512 331 46.34 46.91 46.60 > > 1024 331 63.96 64.71 64.33 > > 2048 331 460.67 465.74 463.18 > > 4096 331 637.33 643.99 640.75 > > > > This the padb output: > > padb –A –x –Ormgr=mpirun –tree: > > > > =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2011.01.06 14:33:17 > =~=~=~=~=~=~=~=~=~=~=~= > > > > Warning, remote process state differs across ranks > > state : ranks > > R (running) : > [1,3-6,8,10-13,16-20,23-28,30-32,34-42,44-45,47-49,51-53,56-59,61-63] > > S (sleeping) : [0,2,7,9,14-15,21-22,29,33,43,46,50,54-55,60] > > Stack trace(s) for thread: 1 > > ----------------- > > [0-63] (64 processes) > > ----------------- > > main() at ?:? > > IMB_init_buffers_iter() at ?:? > > IMB_gather() at ?:? > > PMPI_Gather() at pgather.c:175 > > mca_coll_sync_gather() at coll_sync_gather.c:46 > > ompi_coll_tuned_gather_intra_dec_fixed() at > coll_tuned_decision_fixed.c:714 > > ----------------- > > [0,3-63] (62 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at > coll_tuned_gather.c:248 > > mca_pml_ob1_recv() at pml_ob1_irecv.c:104 > > ompi_request_wait_completion() at > ../../../../ompi/request/request.h:375 > > opal_condition_wait() at > ../../../../opal/threads/condition.h:99 > > ----------------- > > [1] (1 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at > coll_tuned_gather.c:302 > > mca_pml_ob1_send() at pml_ob1_isend.c:125 > > ompi_request_wait_completion() at > ../../../../ompi/request/request.h:375 > > opal_condition_wait() at > ../../../../opal/threads/condition.h:99 > > ----------------- > > [2] (1 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at > coll_tuned_gather.c:315 > > ompi_request_default_wait() at request/req_wait.c:37 > > ompi_request_wait_completion() at > ../ompi/request/request.h:375 > > opal_condition_wait() at ../opal/threads/condition.h:99 > > Stack trace(s) for thread: 2 > > ----------------- > > [0-63] (64 processes) > > ----------------- > > start_thread() at ?:? > > btl_openib_async_thread() at btl_openib_async.c:344 > > poll() at ?:? > > Stack trace(s) for thread: 3 > > ----------------- > > [0-63] (64 processes) > > ----------------- > > start_thread() at ?:? > > service_thread_start() at btl_openib_fd.c:427 > > select() at ?:? > > -bash-3.2$ > > > > > > When running again padb after couple of minutes, I can see that the total > number of processes remain in the same position but > > different processes are at different positions. > > For example, this is the diff between two padb outputs: > > > > Warning, remote process state differs across ranks > > state : ranks > > -R (running) : [0,2-4,6-13,16-18,20-21,28-31,33-36,38-56,58,60,62-63] > > -S (sleeping) : [1,5,14-15,19,22-27,32,37,57,59,61] > > +R (running) : [2,5-14,16-23,25,28-40,42-48,50-51,53-58,61,63] > > +S (sleeping) : [0-1,3-4,15,24,26-27,41,49,52,59-60,62] > > Stack trace(s) for thread: 1 > > ----------------- > > [0-63] (64 processes) > > @@ -13,21 +13,21 @@ > > mca_coll_sync_gather() at coll_sync_gather.c:46 > > ompi_coll_tuned_gather_intra_dec_fixed() at coll_tuned_decision_fixed.c:714 > > ----------------- > > - [0,3-63] (62 processes) > > + [0-5,8-63] (62 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:248 > > mca_pml_ob1_recv() at pml_ob1_irecv.c:104 > > ompi_request_wait_completion() at ../../../../ompi/request/request.h:375 > > opal_condition_wait() at ../../../../opal/threads/condition.h:99 > > ----------------- > > - [1] (1 processes) > > + [6] (1 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:302 > > mca_pml_ob1_send() at pml_ob1_isend.c:125 > > ompi_request_wait_completion() at ../../../../ompi/request/request.h:375 > > opal_condition_wait() at ../../../../opal/threads/condition.h:99 > > ----------------- > > - [2] (1 processes) > > + [7] (1 processes) > > ----------------- > > ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:315 > > ompi_request_default_wait() at request/req_wait.c:37 > > > > > > *Choosing different gather algorithm seems to bypass the hang.* > > I’ve used the following mca parameters: > > --mca coll_tuned_use_dynamic_rules 1 > > --mca coll_tuned_gather_algorithm 1 > > > > Actually, both dec_fixed and basic_linear works while binomial and > linear_sync doesn’t. > > > > With OMPI 1.5 it doesn’t hangs (with all gather algorithms) and it much > faster (the number of repetitions is much higher): > > #---------------------------------------------------------------- > > # Benchmarking Gather > > # #processes = 64 > > #---------------------------------------------------------------- > > #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] > > 0 1000 0.02 0.03 0.02 > > 1 1000 18.50 18.55 18.53 > > 2 1000 18.17 18.25 18.22 > > 4 1000 19.04 19.10 19.07 > > 8 1000 19.60 19.67 19.64 > > 16 1000 21.39 21.47 21.43 > > 32 1000 24.83 24.91 24.87 > > 64 1000 27.35 27.45 27.40 > > 128 1000 33.23 33.34 33.29 > > 256 1000 41.24 41.39 41.32 > > 512 1000 52.62 52.81 52.71 > > 1024 1000 73.20 73.46 73.32 > > 2048 1000 416.36 418.04 417.22 > > 4096 1000 638.54 640.70 639.65 > > 8192 1000 506.26 506.97 506.63 > > 16384 1000 600.63 601.40 601.02 > > 32768 1000 639.52 640.34 639.93 > > 65536 640 914.22 916.02 915.13 > > 131072 320 2287.37 2295.18 2291.35 > > 262144 160 4041.36 4070.58 4056.27 > > 524288 80 7292.35 7463.27 7397.14 > > 1048576 40 13647.15 14107.15 13905.29 > > 2097152 20 30625.00 32635.45 31815.36 > > 4194304 10 63543.01 70987.49 68680.48 > > > > > > · *Problem 2 – segmentation fault with OMPI 1.4.3/1.5 and IMB > gather np=768:* > > When trying to run the same command but with np=768 I get segmentation > fault: > > openmpi-1.4.3/bin/mpirun -np 768 -machinefile voltairenodes -mca btl > sm,self,openib -mca coll_tuned_use_dynamic_rules 1 -mca > coll_tuned_gather_algorithm 1 imb/src/IMB-MPI1 gather -npmin 768 -mem 1.6 > > > > This happens in OMPI 1.4.3 and 1.5 > > > > [compa163:20249] *** Process received signal *** > > [compa163:20249] Signal: Segmentation fault (11) > > [compa163:20249] Signal code: Address not mapped (1) > > [compa163:20249] Failing at address: 0x2aab4a204000 > > [compa163:20249] [ 0] /lib64/libpthread.so.0 [0x366aa0e7c0] > > [compa163:20249] [ 1] > /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libmpi.so.0(ompi_convertor_unpack+0x15f) > [0x2b077882282e] > > [compa163:20249] [ 2] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so > [0x2b077b9e1672] > > [compa163:20249] [ 3] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so > [0x2b077b9dd0b6] > > [compa163:20249] [ 4] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_btl_sm.so > [0x2b077c459d87] > > [compa163:20249] [ 5] > /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libopen-pal.so.0(opal_progress+0xbe) > [0x2b0778d845b8] > > [compa163:20249] [ 6] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so > [0x2b077b9d6d62] > > [compa163:20249] [ 7] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so > [0x2b077b9d6ba7] > > [compa163:20249] [ 8] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so > [0x2b077b9d6a90] > > [compa163:20249] [ 9] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so > [0x2b077d298dc5] > > [compa163:20249] [10] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so > [0x2b077d2990d3] > > [compa163:20249] [11] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so > [0x2b077d286e9b] > > [compa163:20249] [12] > /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_sync.so > [0x2b077d07e96c] > > [compa163:20249] [13] > /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libmpi.so.0(PMPI_Gather+0x55e) > [0x2b077883ec9a] > > [compa163:20249] [14] imb/src/IMB-MPI1(IMB_gather+0xe8) [0x40a088] > > [compa163:20249] [15] imb/src/IMB-MPI1(IMB_init_buffers_iter+0x28a) > [0x405baa] > > [compa163:20249] [16] imb/src/IMB-MPI1(main+0x30f) [0x40362f] > > [compa163:20249] [17] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x3669e1d994] > > [compa163:20249] [18] imb/src/IMB-MPI1 [0x403269] > [compa163:20249] *** End of error message *** > > > Any ideas? More debuggin tips? > > Thanks, > Doron > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
