To be clear, I'm actually a bit torn both ways on this.
On the one hand, including the rpath makes it easy for those who don't
know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass
their paths to their external dependencies at configure time and then
run their binaries without further effort.
On the other hand, maybe they should be cognizant of the fact that
something is going on under the hood to actually allow their binaries
to link properly (i.e. I can imagine a situation where someone builds,
runs, and tests everything locally, and then is confused as to why it
nothing works once deployed).
In my previous email, I argue that we should include the rpath by
default, and can strip it later if we don't want it for some reason
(i.e. when bundling into debs/rpms). Conversely, we could leave it
out by default and only set it as a post-processing step in situations
where we actually care about it.
I'm curious what other people's thoughts are.
On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <klue...@gmail.com> wrote:
> Hi all,
>
> A discussion came up recently around including rpaths in our mesos
> binaries to help resolve any shared library dependencies that don't
> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.).
>
> By default, there are no shared library dependencies that exist in
> non-standard paths, because we bundle all of these dependencies into
> the mesos source and statically link them into our executables (e.g.
> glog, zookeeper, etc.)
>
> However, if you configure mesos with e.g.
>
> ../configure ----disable-bundled
>
> or the more selective
>
> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ...
>
> then mesos will be built with an external shared library dependency
> (e.g. glog and zookeeper in this case).
>
> The build system is smart enough to set up LDFLAGS so we can link
> against whatever external libraries are passed in via the --with-*
> flags.
>
> However, when we go to run the binaries that are produced (e.g.
> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them
> with an LD_LIBRARY_PATH pointing to the location of the shared
> libraires from these external dependencies, e.g.
>
> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master
>
> It would be nice if we didn't have to explicitly set the
> LD_LIBRARY_PATH to launch these binaries when linking against any
> external shared library dependencies.
>
> One way around this would be to make sure that all external library
> dependencies were stored in standard search paths for the dynamic
> linker. This is typically what happens if you install these
> dependencies via a standard package manager (e.g. apt-get, yum, etc.).
> Sometimes this is undesirable (or impossible) though, especially if
> the external dependencies do not exist as packages or follow a
> non-standard directory hierarchy in terms of where it places its
> include files, libraries, etc.
>
> Another option is to install the paths to these external libraries
> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the
> dynamic linker will search them at runtime. This is also unfeasible
> at times and has the added disadvantage that these library paths will
> now be searched for *all* binaries that get executed (not just the
> ones we currently care about).
>
> The final option (and the one I'm proposing here) is to set the
> 'rpath' of the binary to point to the location of the external shared
> library on the build machine. The rpath is embedded into the binary
> at link time and is used to give the linker an extra set of paths to
> search for shared libraries at runtime. This obvious advantage here
> is that setting rpath allows us to run our binaries without requiring
> LD_LIBRARY_PATH or any of the other methods mentioned above to tell
> the linker where to find our shared libraries. However, it has the
> disadvantage of baking a path into the binary that may only exist on
> the specific machine the binary was built on.
>
> That said, the standard search order used by the dynamic linker to
> find shared libraries is:
>
> 1) LD_LIBRARY_PATH
> 2) rpath
> 4) the ldcache (/etc/ld.so.conf on linux)
> 3) default paths (e.g. /lib, /usr/local/lib)
>
> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH
> if we wanted to. Moreover, we could even change the rpath at the time
> of deployment (e.g. via chrpath on linux). This may be desirable if
> the shared libraries are installed at different locations on the
> deployment machine.
>
> If there are no objections, I therefore propose we modify the
> following files to add rpaths to all external dependencies set via
> --with-* flags:
>
> ./configure.ac
> ./3rdparty/libprocess/3rdparty/stout/configure.ac
> ./3rdparty/libprocess/configure.ac
>
> The pattern would change from:
>
> CPPFLAGS="-I${with_thing}/include $CPPFLAGS"
> LDFLAGS="-L${with_thing}/lib $LDFLAGS"
>
> to include an additional line with:
>
> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS"
>
> I know there has some hesitation with this in the past (especially
> when it comes to producing rpms or debs, where baking in an rpath
> seems really strange), but I'm hoping people will agree that it's
> useful enough that it makes sense to include the rpaths as the default
> case. We can always run a post-processing step to strip them in cases
> where they are undesirable.
>
> Thanks!
>
> --
> ~Kevin
--
~Kevin
