[OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-24 Thread Tim Mattox via devel
Hello, I've run into an application that has its performance dramatically affected by some configuration options to the underlying UCX library. Is there a way to configure/build Open MPI so that which UCX library is used is determined at runtime (e.g. by an environment module), rather than having t

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-24 Thread Tim Mattox via devel
ng which can be resolved another way. > > --Yossi > > -Original Message- > From: devel On Behalf Of Tim Mattox via > devel > Sent: Sunday, 24 January 2021 23:18 > To: devel@lists.open-mpi.org > Cc: Tim Mattox > Subject: [OMPI devel] How to build Open MPI so the

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-24 Thread Gilles Gouaillardet via devel
ific > > location > > For example: mpirun -x > > LD_PRELOAD=::: > > ... > > > > BTW, what is different about the other UCX configuration? Maybe this is > > something which can be resolved another way. > > > > --Yossi > > > > ---

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-26 Thread Tim Mattox via devel
PM Yossi Itigin wrote: > > > > > > Hi, > > > > > > One option is to use LD_PRELOAD to load all ucx libraries from a specific > > > location > > > For example: mpirun -x > > > LD_PRELOAD=::: > > > ... > > > > >

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-26 Thread Tim Mattox via devel
gt; > dramatic performance hit for one application (so far). > > > I've already tested that everything works fine if I replace UCX's .so > > > files manually in the filesystem, and the "new/changed" ones get > > > loaded, but a user can't make t

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-26 Thread Tim Mattox via devel
; > --enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug > > > > tuned version: --disable-logging --disable-debug --disable-assertions > > > > --disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned > > > > > > > &g

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-26 Thread Gilles Gouaillardet via devel
swap ucx/1.9.0/tuned ucx/1.9.0/debug" when they want to enable debugging at the UCX layer. On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin wrote: Hi, One option is to use LD_PRELOAD to load all ucx libraries from a specific location For example: mpirun -x LD_PRELOAD=::: ... BTW, what is differ

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-27 Thread Jeff Squyres (jsquyres) via devel
On Jan 27, 2021, at 2:00 AM, Gilles Gouaillardet via devel mailto:devel@lists.open-mpi.org>> wrote: Tim, a simple option is to configure ... LDFLAGS="-Wl,--enable-new-dtags" If Open MPI is built with this option, then LD_LIBRARY_PATH takes precedence over rpath (the default is the opposite

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-27 Thread Tim Mattox via devel
Thank you for the suggestion of 'configure ... LDFLAGS="-Wl,--enable-new-dtags"'. I'm still reading up on its meaning, but wouldn't that change the behavior across all dependencies that are dynamically linked when I build Open MPI? I was specifically wanting *just* these UCX .so files to be dynami

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-01-27 Thread Gilles Gouaillardet via devel
Jeff, What I meant is the default Linux behavior is to first lookup dependencies in the rpath, and then fallback to LD_LIBRARY_PATH *unless* -Wl,--enable-new-dtags was used at link time. In the case of Open MPI, -Wl,--enable-new-dtags is added to the MPI wrappers, but Open MPI is *not* b

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-02-01 Thread Jeff Squyres (jsquyres) via devel
On Jan 27, 2021, at 7:19 PM, Gilles Gouaillardet wrote: > > What I meant is the default Linux behavior is to first lookup dependencies in > the rpath, and then fallback to LD_LIBRARY_PATH > *unless* -Wl,--enable-new-dtags was used at link time. > > In the case of Open MPI, -Wl,--enable-new-dtag

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-02-01 Thread Peter Kjellström via devel
On Mon, 1 Feb 2021 14:46:22 + "Jeff Squyres \(jsquyres\) via devel" wrote: > On Jan 27, 2021, at 7:19 PM, Gilles Gouaillardet > wrote: > > > > What I meant is the default Linux behavior is to first lookup > > dependencies in the rpath, and then fallback to LD_LIBRARY_PATH > > *unless* -Wl,-

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

2021-02-01 Thread Tim Mattox via devel
FYI - I wasn’t bothered by the default behavior... I was just looking for a sanctioned way for an installer (e.g. a sysadmin) to make the UCX be loaded based on LD_LIBRARY_PATH so that there was an ability for the user to swap in a debug build of UCX at runtime. On Mon, Feb 1, 2021 at 10:14 AM Pe