On Wed, 2014-03-12 at 14:34 +0000, Dave Goodell (dgoodell) wrote: > Perhaps there's an RPATH issue here? I don't fully understand the structure > of Rmpi, but is there both an app and a library (or two separate libraries) > that are linking against MPI? > > I.e., what we want is: > > app -----------------> ~ross/OMPI > \ / > --> library -- > > But what we're getting is: > > app ---> /usr/OMPI > \ > --> library ---> ~ross/OMPI > > > If one of them was first linked against the /usr/OMPI and managed to get an > RPATH then it could override your LD_LIBRARY_PATH. > I think the relevant app here is R. It was built without any awareness of MPI, I'm pretty sure. R loads the library Rmpi.so, which in turn references MPI. The R binary has no runpath or rpath according to chrpath.
ldd Rmpi.so shows my local MPI libraries and not the system ones, though it references plenty of other system libraries. The system MPI libraries are in a standard place, /usr/lib (/usr/lib/openmpi/lib/ more precisely) and so I don't think an rpath is necessary to look for it. Ross > -Dave > > On Mar 12, 2014, at 5:39 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> > wrote: > > > Generally, all you need to ensure that your personal copy of OMPI is used > > is to set the PATH and LD_LIBRARY_PATH to point to your new Open MPI > > installation. I do this all the time on my development cluster (where I > > have something like 6 billion different installations of OMPI available... > > mmm... should probably clean that up...) > > > > export LD_LIBRARY_PATH=path_to_my_ompi/lib:$LD_LIBRARY_PATH > > export PATH=path-to-my-ompi/bin:$PATH > > > > It should be noted that: > > > > 1. you need to *prefix* your PATH and LD_LIBRARY_PATH with these values > > 2. you need to set these values in a way that will be picked up on all > > servers that you use in your job. The safest way to do this is in your > > shell startup files (e.g., $HOME/.bashrc or whatever is relevant for your > > shell). > > > > See http://www.open-mpi.org/faq/?category=running#run-prereqs, > > http://www.open-mpi.org/faq/?category=running#adding-ompi-to-path, and > > http://www.open-mpi.org/faq/?category=running#mpirun-prefix. > > > > Note the --prefix option that is described in the 3rd FAQ item I cited -- > > that can be a bit easier, too. > > > > > > > > On Mar 12, 2014, at 2:51 AM, Ross Boylan <r...@biostat.ucsf.edu> wrote: > > > >> I took the advice here and built a personal copy of the current openmpi, > >> to see if the problems I was having with Rmpi were a result of the old > >> version on the system. > >> > >> When I do ldd on the relevant libraries (Rmpi.so is loaded dynamically > >> by R) everything looks fine; path references that should be local are. > >> But when I run the program and do lsof it shows that both the system and > >> personal versions of key libraries are opened. > >> > >> First, does anyone know which library will actually be used, or how to > >> tell which library is actually used, in this situation. I'm running on > >> linux (Debian squeeze)? > >> > >> Second, it there some way to prevent the wrong/old/sytem libraries from > >> being loaded? > >> > >> FWIW I'm still seeing the old misbehavior when I run this way, but, as I > >> said, I'm really not sure which libraries are being used. Since Rmpi > >> was built against the new/local ones, I think the fact that it doesn't > >> crash means I really am using the new ones. > >> > >> Here are highlights of lsof on the process running R: > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > >> R 17634 ross cwd DIR 254,2 12288 150773764 > >> /home/ross/KHC/sunbelt > >> R 17634 ross rtd DIR 8,1 4096 2 / > >> R 17634 ross txt REG 8,1 5648 3058294 > >> /usr/lib/R/bin/exec/R > >> R 17634 ross DEL REG 8,1 2416718 > >> /tmp/openmpi-sessions-ross@n100_0/60429/1/shared_mem_pool.n100 > >> R 17634 ross mem REG 8,1 335240 3105336 > >> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0 > >> R 17634 ross mem REG 8,1 304576 3105337 > >> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0 > >> R 17634 ross mem REG 8,1 679992 3105332 > >> /usr/lib/openmpi/lib/libmpi.so.0.0.2 > >> R 17634 ross mem REG 8,1 93936 2967826 > >> /usr/lib/libz.so.1.2.3.4 > >> R 17634 ross mem REG 8,1 10648 3187256 > >> /lib/libutil-2.11.3.so > >> R 17634 ross mem REG 8,1 32320 2359631 > >> /usr/lib/libpciaccess.so.0.10.8 > >> R 17634 ross mem REG 8,1 33368 2359338 > >> /usr/lib/libnuma.so.1 > >> R 17634 ross mem REG 254,2 979113 152045740 > >> /home/ross/install/lib/libopen-pal.so.6.1.0 > >> R 17634 ross mem REG 8,1 183456 2359592 > >> /usr/lib/libtorque.so.2.0.0 > >> R 17634 ross mem REG 254,2 1058125 152045781 > >> /home/ross/install/lib/libopen-rte.so.7.0.0 > >> R 17634 ross mem REG 8,1 49936 2359341 > >> /usr/lib/libibverbs.so.1.0.0 > >> R 17634 ross mem REG 254,2 2802579 152045867 > >> /home/ross/install/lib/libmpi.so.1.3.0 > >> R 17634 ross mem REG 254,2 106626 152046481 > >> /home/ross/Rlib-3.0.1/Rmpi/libs/Rmpi.so > >> > >> So libmpi, libopen-pal, and libopen-rte all are opened in two versions and > >> two locations. > >> > >> Thanks. > >> Ross Boylan > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
