Hi Jeff H. Why don't you just try configuring with
./configure --prefix=my_favorite_install_dir --with-libfabric=install_dir_for_libfabric make -j 8 install and see what happens? Make sure before you configure that you have PrgEnv-gnu or PrgEnv-intel module loaded. Those were the configure/compiler options I used to do testing of ofi mtl on cori. Jeff S. - this thread has gotten intermingled with mpich setup as well, hence the suggestion for the mpich shm mechanism. Howard 2015-11-19 16:59 GMT-07:00 Jeff Hammond <jeff.scie...@gmail.com>: > >> How did you configure for Cori? You need to be using the slurm plm >> component for that system. I know this sounds like gibberish. >> >> > ../configure --with-libfabric=$HOME/OFI/install-ofi-gcc-gni-cori \ > --enable-mca-static=mtl-ofi \ > --enable-mca-no-build=btl-openib,btl-vader,btl-ugni,btl-tcp \ > --enable-static --disable-shared --disable-dlopen \ > --prefix=$HOME/MPI/install-ompi-ofi-gcc-gni-xpmem-cori \ > --with-cray-pmi --with-alps --with-cray-xpmem --with-slurm \ > --without-verbs --without-fca --without-mxm --without-ucx \ > --without-portals4 --without-psm --without-psm2 \ > --without-udreg --without-ugni --without-munge \ > --without-sge --without-loadleveler --without-tm --without-lsf \ > --without-pvfs2 --without-plfs \ > --without-cuda --disable-oshmem \ > --disable-mpi-fortran --disable-oshmem-fortran \ > LDFLAGS="-L/opt/cray/ugni/default/lib64 -lugni \ > -L/opt/cray/alps/default/lib64 -lalps -lalpslli -lalpsutil \ > -ldl -lrt" > > > This is copied from > https://github.com/jeffhammond/HPCInfo/blob/master/ofi/README.md#open-mpi, > which I note in case you want to see what changes I've made at any point in > the future. > > >> There should be a with-slurm configure option to pick up this component. >> >> Indeed there is. > > >> Doesn't mpich have the option to use sysv memory? You may want to try >> that >> >> > MPICH? Look, I may have earned my way onto Santa's naughty list more than > a few times, but at least I have the decency not to post MPICH questions to > the Open-MPI list ;-) > > If there is a way to tell Open-MPI to use shm_open without filesystem > backing (if that is even possible) at configure time, I'd love to do that. > > >> Oh for tuning params you can use env variables. For example lets say >> rather than using the gni provider in ofi mtl you want to try sockets. Then >> do >> >> Export OMPI_MCA_mtl_ofi_provider_include=sockets >> >> > Thanks. I'm glad that there is an option to set them this way. > > >> In the spirit OMPI - may the force be with you. >> >> > All I will say here is that Open-MPI has a Vader BTL :-) > >> >> > On Thu 19.11.2015 09:44:20 Jeff Hammond wrote: >> > > I have no idea what this is trying to tell me. Help? >> > > >> > > jhammond@nid00024:~/MPI/qoit/collectives> mpirun -n 2 ./driver.x 64 >> > > [nid00024:00482] [[46168,0],0] ORTE_ERROR_LOG: Not found in file >> > > ../../../../../orte/mca/plm/alps/plm_alps_module.c at line 418 >> > > >> > > I can run the same job with srun without incident: >> > > >> > > jhammond@nid00024:~/MPI/qoit/collectives> srun -n 2 ./driver.x 64 >> > > MPI was initialized. >> > > >> > > This is on the NERSC Cori Cray XC40 system. I build Open-MPI git head >> from >> > > source for OFI libfabric. >> > > >> > > I have many other issues, which I will report later. As a spoiler, if >> I >> > > cannot use your mpirun, I cannot set any of the MCA options there. Is >> > > there a method to set MCA options with environment variables? I could >> not >> > > find this documented anywhere. >> > > >> > > In particular, is there a way to cause shm to not use the global >> > > filesystem? I see this issue comes up a lot and I read the list >> archives, >> > > but the warning message ( >> > > >> https://github.com/hpc/cce-mpi-openmpi-1.6.4/blob/master/ompi/mca/common/sm/ >> > > help-mpi-common-sm.txt) suggested that I could override it by setting >> TMP, >> > > TEMP or TEMPDIR, which I did to no avail. >> > >> > From my experience on edison: the one environment variable that does >> works is TMPDIR - the one that is not listed in the error message :-) >> > > That's great. I will try that now. Is there a Github issue open already > to fix that documentation? If not... > > >> > Can't help you with your mpirun problem though ... >> >> No worries. I appreciate all the help I can get. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.scie...@gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/11/28072.php >
