Any hint for the previous mail? Does Open MPI-1.3.3 support only a limited versions of OFED? Or any version is ok? On Sun, Oct 11, 2009 at 3:55 PM, Sangamesh B <forum....@gmail.com> wrote:
> Hi, > > A fortran application is installed with Intel Fortran 10.1, MKL-10 and > Openmpi-1.3.3 on a Rocks-5.1 HPC Linux cluster. The jobs are not scaling > when more than one node is used. The cluster has Intel Quad core Xeon > (E5472) @ 3.00GHz Dual processor (total 8 cores per node, 16GB RAM) and > Infiniband interconnectivity. > > Here are some of the timings: > > 12 cores (Node 1: 8 cores, Node2: 4 cores) -- No progress in the job > 8 cores (Node 1: 8 cores) -- 21 hours (38 CG > move steps) > 4 cores (Node 1: 4 cores) -- 25 hours > 12 cores (Node 1, Node 2, Node 3: 4cores each) -- No progress > > > Later to check, whether Open MPI is using IB or not, I used --mca btl > openib. But the job failed with following error message: > # cat /home1/g03/apps_test/amber/test16/err.352.job16 > -------------------------------------------------------------------------- > At least one pair of MPI processes are unable to reach each other for > MPI communications. This means that no Open MPI device has indicated > that it can be used to communicate between these processes. This is > an error; Open MPI requires that all MPI processes be able to reach > each other. This error can sometimes be the result of forgetting to > specify the "self" BTL. > > Process 1 ([[23671,1],12]) is on host: compute-0-12.local > Process 2 ([[23671,1],12]) is on host: compute-0-12.local > BTLs attempted: openib > > Your MPI job is now going to abort; sorry. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > PML add procs failed > --> Returned "Unreachable" (-12) instead of "Success" (0) > -------------------------------------------------------------------------- > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > [compute-0-12.local:5496] Abort before MPI_INIT completed successfully; not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > [compute-0-5.local:6916] Abort before MPI_INIT completed successfully; not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > [compute-0-5.local:6914] Abort before MPI_INIT completed successfully; not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > [compute-0-5.local:6915] Abort before MPI_INIT completed successfully; not > able to guarantee that all other processes were killed! > [compute-0-5.local:6913] Abort before MPI_INIT completed successfully; not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > mpirun has exited due to process rank 12 with PID 5496 on > node compute-0-12.local exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > [compute-0-5.local:06910] 15 more processes have sent help message > help-mca-bml-r2.txt / unreachable proc > [compute-0-5.local:06910] Set MCA parameter "orte_base_help_aggregate" to 0 > to see all help / error messages > [compute-0-5.local:06910] 15 more processes have sent help message > help-mpi-runtime / mpi_init:startup:internal-failure > -------------------------------------------------------------------------- > At least one pair of MPI processes are unable to reach each other for > MPI communications. This means that no Open MPI device has indicated > that it can be used to communicate between these processes. This is > an error; Open MPI requires that all MPI processes be able to reach > each other. This error can sometimes be the result of forgetting to > specify the "self" BTL. > > Process 1 ([[23958,1],2]) is on host: compute-0-5.local > Process 2 ([[23958,1],2]) is on host: compute-0-5.local > BTLs attempted: openib > > Then added 'self' to --mca btl openib,. With this it started running, but I > can make sure its not using IB as I observed it from the netstat -i command. > > 1st Snap: > > Every 2.0s: netstat > -i > Sun Oct 11 15:29:29 2009 > > Kernel Interface table > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP > TX-OVR Flg > eth0 1500 0 1847619 0 0 0 2073010 0 0 0 > BMRU > ib0 65520 0 708 0 0 0 509 0 5 0 > BMRU > lo 16436 0 5731 0 0 0 5731 0 0 0 > LRU > > 2nd Snap: > > Every 2.0s: netstat > -i > Sun Oct 11 15:29:57 2009 > > Kernel Interface table > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP > TX-OVR Flg > eth0 1500 0 1847647 0 0 0 2073073 0 0 0 > BMRU > ib0 65520 0 708 0 0 0 509 0 5 0 > BMRU > lo 16436 0 5731 0 0 0 5731 0 0 0 > LRU > > Why OpenMPI is not able to use IB? > > The ldd to the executable shows, no IB libraries are linked. Is this the > reason: > ldd /opt/apps/siesta/siesta_mpi > > /opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_intel_lp64.so(0x00002aaaaaaad000) > > /opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_intel_thread.so(0x00002aaaaadc2000) > /opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_core.so(0x00002aaaab2ad000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000034a6200000) > libmpi_f90.so.0 => /opt/mpi/openmpi/1.3.3/intel/lib/libmpi_f90.so.0 > (0x00002aaaab4a0000) > libmpi_f77.so.0 => /opt/mpi/openmpi/1.3.3/intel/lib/libmpi_f77.so.0 > (0x00002aaaab6a3000) > libmpi.so.0 => /opt/mpi/openmpi/1.3.3/intel/lib/libmpi.so.0 > (0x00002aaaab8db000) > libopen-rte.so.0 => /opt/mpi/openmpi/1.3.3/intel/lib/libopen-rte.so.0 > (0x00002aaaabbaa000) > libopen-pal.so.0 => /opt/mpi/openmpi/1.3.3/intel/lib/libopen-pal.so.0 > (0x00002aaaabe07000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000034a5e00000) > libnsl.so.1 => /lib64/libnsl.so.1 (0x00000034a8200000) > libutil.so.1 => /lib64/libutil.so.1 (0x00000034a6600000) > libifport.so.5 => /opt/intel/fce/10.1.008/lib/libifport.so.5 > (0x00002aaaac09a000) > libifcoremt.so.5 => /opt/intel/fce/10.1.008/lib/libifcoremt.so.5 > (0x00002aaaac1d0000) > libimf.so => /opt/intel/cce/10.1.018/lib/libimf.so (0x00002aaaac401000) > libsvml.so => /opt/intel/cce/10.1.018/lib/libsvml.so > (0x00002aaaac766000) > libm.so.6 => /lib64/libm.so.6 (0x00000034a6e00000) > libguide.so => > /opt/intel/mkl/10.0.5.025/lib/em64t/libguide.so(0x00002aaaac8f1000) > libintlc.so.5 => /opt/intel/cce/10.1.018/lib/libintlc.so.5 > (0x00002aaaaca65000) > libc.so.6 => /lib64/libc.so.6 (0x00000034a5a00000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000034a7e00000) > /lib64/ld-linux-x86-64.so.2 (0x00000034a5600000) > > With the help of Open MPI FAQ: > > # /opt/mpi/openmpi/1.3.3/intel/bin/ompi_info --param btl openib > MCA btl: parameter "btl_base_verbose" (current value: "0", > data source: default value) > Verbosity level of the BTL framework > MCA btl: parameter "btl" (current value: <none>, data > source: default value) > Default selection set of components for the btl > framework (<none> means use all components that can be found) > MCA btl: parameter "btl_openib_verbose" (current value: > "0", data source: default value) > Output some verbose OpenIB BTL information (0 = > no output, nonzero = output) > MCA btl: parameter > "btl_openib_warn_no_device_params_found" (current value: "1", data source: > default value, synonyms: > btl_openib_warn_no_hca_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = > do not warn; any other value = warn) > MCA btl: parameter "btl_openib_warn_no_hca_params_found" > (current value: "1", data source: default value, deprecated, synonym of: > btl_openib_warn_no_device_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = > do not warn; any other value = warn) > MCA btl: parameter "btl_openib_warn_default_gid_prefix" > (current value: "1", data source: default value) > Warn when there is more than one active ports and > at least one of them connected to the network with only default GID prefix > configured (0 = do not warn; any other value = > warn) > MCA btl: parameter "btl_openib_warn_nonexistent_if" > (current value: "1", data source: default value) > Warn if non-existent devices and/or ports are > specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; > any > other value = warn) > > During Open MPI install I've used --with-openib=/usr. So I believe its > compiled with IB support. > > The IB utilities such as ibv_rc_pingpong are working fine. > > I'm not getting why its OMPI is not using IB? Please help me to resolve > this issue. > > Thanks >
