HI Jeff, by adding PATH and LD_LIBRARY_PATH, I don't see orted not found issue.
[root@vcn03 pasokan]# mpirun --allow-run-as-root -np 4 -host vcn03,vcn03,vcn04,vcn04 /mnt/lustre_client/pasokan/a.out -------------------------------------------------------------------------- WARNING: No preset parameters were found for the device that Open MPI detected: Local host: vcn03 Device name: mlx5_0 Device vendor ID: 0x02c9 Device vendor part ID: 4114 Default device parameters will be used, which may result in lower performance. You can edit any of the files specified by the btl_openib_device_param_files MCA parameter to set values for your device. NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0. -------------------------------------------------------------------------- [vcn04][[33859,1],2][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument [vcn03][[33859,1],0][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument [vcn03][[33859,1],1][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument [vcn04][[33859,1],3][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument Hello world from processor vcn03, rank 0 out of 4 processors Hello world from processor vcn03, rank 1 out of 4 processors Hello world from processor vcn04, rank 2 out of 4 processors Hello world from processor vcn04, rank 3 out of 4 processors [vcn03:05070] 3 more processes have sent help message help-mpi-btl-openib.txt / no device params found [vcn03:05070] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [root@vcn03 pasokan]# but still IOR isn't running while compiled using OpenMPI, throwing segmentation fault, which used to be very straight forward in Baremetal but not in KVM + SR-IOV ________________________________________ From: Pharthiphan Asokan Sent: Tuesday, March 13, 2018 8:42 PM To: Open MPI Developers Subject: RE: [OMPI devel] How to Build OpenMPI to support FDR over SR-IOV Thanks Jeff, OpenMPI is installed here [root@vcn03 C]# cd /mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/ bin/ etc/ include/ lib/ share/ [root@vcn03 C]# why exporting these variables not taking effect export PATH=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/bin:$PATH export LD_LIBRARY_PATH=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/lib:$LD_LIBRARY_PATH export INCLUDE=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/include:$INCLUDE but as said by providing --prefix /mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/ is working [root@vcn03 C]# mpirun --prefix /mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/ --allow-run-as-root -np 2 -host vcn03,vcn04 hostname vcn04 vcn03 [root@vcn03 C]# though my issue is IOR isn't running while compile with OpenMPI on SR-IOV envirorment [root@vcn03 C]# pwd /mnt/lustre_client/pasokan/IOR-July12/src/C [root@vcn03 C]# [root@vcn03 C]# export PATH=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/bin:$PATH [root@vcn03 C]# export LD_LIBRARY_PATH=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/lib:$LD_LIBRARY_PATH [root@vcn03 C]# export INCLUDE=/mnt/lustre_client/pasokan/openmpi-3.0.0/openmpi-3.0.0/include:$INCLUDE [root@vcn03 C]# [root@vcn03 C]# gmake posix mpiio mpicc -o IOR IOR.o utilities.o parse_options.o \ aiori-POSIX.o aiori-noMPIIO.o aiori-noHDF5.o aiori-noNCMPI.o \ -lm mpicc -o IOR IOR.o utilities.o parse_options.o \ aiori-POSIX.o aiori-MPIIO.o aiori-noHDF5.o aiori-noNCMPI.o \ -lm [root@vcn03 C]# ./IOR -------------------------------------------------------------------------- WARNING: No preset parameters were found for the device that Open MPI detected: Local host: vcn03 Device name: mlx5_0 Device vendor ID: 0x02c9 Device vendor part ID: 4114 Default device parameters will be used, which may result in lower performance. You can edit any of the files specified by the btl_openib_device_param_files MCA parameter to set values for your device. NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0. -------------------------------------------------------------------------- [vcn03][[34068,1],0][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument Segmentation fault [root@vcn03 C]# Please help ! ________________________________________ From: devel [devel-boun...@lists.open-mpi.org] on behalf of Jeff Squyres (jsquyres) [jsquy...@cisco.com] Sent: Tuesday, March 13, 2018 8:20 PM To: Open MPI Developers List Subject: Re: [OMPI devel] How to Build OpenMPI to support FDR over SR-IOV On Mar 13, 2018, at 2:08 AM, Pharthiphan Asokan <paso...@ddn.com> wrote: > > [root@vcn03 C]# mpirun --allow-run-as-root -np 2 -host vcn03,vcn04 hostname > bash: orted: command not found This is the key ^^ These FAQ items may help: * https://www.open-mpi.org/faq/?category=running#run-prereqs. * https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path * https://www.open-mpi.org/faq/?category=running#mpirun-prefix -- Jeff Squyres jsquy...@cisco.com _______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel _______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel