Tim, Have you tried using LD_LIBRARY_PATH? I guess "hardcoding the full path" means "link with -rpath", and IIRC, LD_LIBRARY_PATH overrides this setting.
If this does not work, here something you can try (disclaimer: I did not) export LD_LIBRARY_PATH=/same/install/prefix/ucx/1.9.0/lib configure ... --with-ucx CPPFLAGS=-I/same/install/prefix/ucx/1.9.0/include LDFLAGS=-L/same/install/prefix/ucx/1.9.0/lib I expect the UCX components use libuct.so instead of /same/install/prefix/ucx/1.9.0/lib/libuct.so. If your users want the debug version, then you can simply change your LD_LIBRARY_PATH (module swap ucx should do the trick) Three caveats you should keep in mind: - it is your responsibility to ensure the debug and prod versions of UCX are ABI compatible - it will be mandatory to load a ucx module (otherwise Open MPI won't find UCX libraries) - this is a guess and I did not test this. An other option (I did not try) would be to install UCX on your build machine in /usr (since I expect /usr/lib/libuct.so is not hardcoded) and then use LD_LIBRARY_PATH (I assume your ucx module set it) to point to the UCX flavor of your choice). Cheers, Gilles On Mon, Jan 25, 2021 at 7:43 AM Tim Mattox via devel <devel@lists.open-mpi.org> wrote: > > I'm specifically wanting my users to be able to load a "debug" vs. > "tuned" UCX module, without me having to make two different Open MPI > installs... the combinatorics get bad after a few versions.... (I'm > already having multiple versions of Open MPI to handle the differences > in Fortran mpi mod files for various compilers.) > Here are the differences in the configure options between the two UCX modules: > debug version: --enable-logging --enable-debug --enable-assertions > --enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug > tuned version: --disable-logging --disable-debug --disable-assertions > --disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned > > We noticed that the --enable-debug option for UCX has a pretty > dramatic performance hit for one application (so far). > I've already tested that everything works fine if I replace UCX's .so > files manually in the filesystem, and the "new/changed" ones get > loaded, but a user can't make that kind of swap. > My hope is a user could type "module swap ucx/1.9.0/tuned > ucx/1.9.0/debug" when they want to enable debugging at the UCX layer. > > On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin <yos...@nvidia.com> wrote: > > > > Hi, > > > > One option is to use LD_PRELOAD to load all ucx libraries from a specific > > location > > For example: mpirun -x > > LD_PRELOAD=<path-to-libucp.so>:<path-to-libuct.so>:<path-to-libucs.so>:<path-to-libucm.so> > > ... <exe> <args> > > > > BTW, what is different about the other UCX configuration? Maybe this is > > something which can be resolved another way. > > > > --Yossi > > > > -----Original Message----- > > From: devel <devel-boun...@lists.open-mpi.org> On Behalf Of Tim Mattox via > > devel > > Sent: Sunday, 24 January 2021 23:18 > > To: devel@lists.open-mpi.org > > Cc: Tim Mattox <tmat...@gmail.com> > > Subject: [OMPI devel] How to build Open MPI so the UCX used can be changed > > at runtime? > > > > Hello, > > I've run into an application that has its performance dramatically affected > > by some configuration options to the underlying UCX library. > > Is there a way to configure/build Open MPI so that which UCX library is > > used is determined at runtime (e.g. by an environment module), rather than > > having to configure/build different instances of Open MPI? > > > > When I configure Open MPI 4.1.0 with "--with-ucx" it is hardcoding the full > > path to the UCX .so library files to the UCX version it found at configure > > time. > > -- > > Tim Mattox, Ph.D. - tmat...@gmail.com > > > > -- > Tim Mattox, Ph.D. - tmat...@gmail.com
