Re: [CMake] Changing installation prefix triggers re-linking of all libraries
On 05/10/2011 11:24 AM, Pere Mato Vila wrote:
>>
>> To my regret, I don't see any easy solution for your concern, but if
>> the unnecessary rebuilds due to the the RPATH placeholder mechanism
>> are a serious issue in your project, the above-noted approach can
>> possibly be adapted to your needs.
>>
>> 'hope that helps.
> Dear Michael,
>
> Thank-you very much for your interest and the time you have spend developing
> this solution. But I am afraid that the added complexity in the coding and
> procedure does not balance with benefits of having an installation RPATH. We
> will continue with what we have been doing and set the LD_LIBRARY_PATH when
> using the installed project. Nevertheless I must say that this solution has
> been very educative for me, since I have learnt a few things (or tricks) that
> I was not aware you could do with CMake. Thanks again.
>
> Pere
Hi Pere,
sorry for re-entering this thread so late, but recently, I stumbled over
the CMAKE_NO_BUILTIN_CHRPATH variable which possibly provides an easy
solution for your issue. Look at the following exemplary project:
# CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(RPATH C)
SET(CMAKE_VERBOSE_MAKEFILE ON)
SET(CMAKE_NO_BUILTIN_CHRPATH ON)
ADD_EXECUTABLE(gen gen.c)
ADD_CUSTOM_COMMAND(OUTPUT main.c COMMAND gen > main.c DEPENDS gen)
ADD_EXECUTABLE(main main.c)
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
ADD_LIBRARY(f SHARED f.c)
ADD_LIBRARY(g SHARED g.c)
INSTALL(TARGETS f g LIBRARY DESTINATION lib)
TARGET_LINK_LIBRARIES(gen f g)
/* f.c: */
void f(void){}
/* g.c: */
void g(void){}
/* gen.c: */
#include
int main(void)
{
printf("int main(void){return 0;}\n"); return 0;
}
Essentially, it's the same as in my previous posting, but without the
somewhat weird gen-on-f/g dependency. Instead, it is the usual simple
setup of a code generator which depends on two libraries. Without the
CMAKE_NO_BUILTIN_CHRPATH set to ON, a change in CMAKE_INSTALL_PREFIX
would result in relinking the f/g libraries, and subsequently in re-
building gen and main due to their dependency relations. This is what
you complained about, IIRC. Now, with CMAKE_NO_BUILTIN_CHRPATH set to
ON, the concerned binaries are actually relinked at installation time
to get the RPATHs right, so there's no need anymore to provide enough
space for the final RPATHs at build time. As a consequence, the f/g
libraries aren't relinked when CMAKE_INSTALL_PREFIX changes, and gen
along with main doesn't become out-of-date. However, lib{f,g}.so are
relinked at installation time, but this should be by far a cheaper
alternative to the nearly complete project's recompilation you've
reported.
'hope that helps.
Regards,
Michael
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
> > To my regret, I don't see any easy solution for your concern, but if > the unnecessary rebuilds due to the the RPATH placeholder mechanism > are a serious issue in your project, the above-noted approach can > possibly be adapted to your needs. > > 'hope that helps. Dear Michael, Thank-you very much for your interest and the time you have spend developing this solution. But I am afraid that the added complexity in the coding and procedure does not balance with benefits of having an installation RPATH. We will continue with what we have been doing and set the LD_LIBRARY_PATH when using the installed project. Nevertheless I must say that this solution has been very educative for me, since I have learnt a few things (or tricks) that I was not aware you could do with CMake. Thanks again. Pere - Pere Mato CERN, PH Department, CH 1211 Geneva 23, Switzerland e-mail: pere.m...@cern.ch tel: +41 22 76 78696 fax: +41 22 76 68792gsm: +41 76 48 70855 ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
On 05/05/2011 05:14 PM, Pere Mato Vila wrote:
>
>
>> If I understand correctly, it's this second point which causes your
>> astonishment, but it already happens during the build phase in the
>> build tree, not just when installing with "make install", right?
>
> Yes, this is indeed my astonishment. I understand that for 'normal' projects
> this re-link in the build tree will be almost unnoticeable. Unfortunately in
> our project we build at least one executable using these libraries that is a
> code generator and therefore all generated code for many other libraries gets
> regenerated, re-compiled and re-linked. It is a major re-build just for
> changing the length of the installation directory. So, I see that the only
> solution I have is to avoid using CMAKE_INSTALL_RPATH. Thanks very much for
> your explanations.
> Regards,
>
> Pere
Uuhhh, a code generator with its far-reaching dependency implications
is quite unpleasant in this regard, indeed. Perhaps, there's a chance
to accomplish nearly what you have in mind. If I've got it right, the
problem is that one can not distinguish the "good" relink operations,
i.e. due to changes in source code or prerequisite targets, from the
"bad" ones, i.e. due to the placeholder mechanism when the RPATH has
changed. So, a possible approach to prevent the code generator from
becoming out-of-date due to a mere RPATH change in its prerequisite
libraries is to dissolve the generator's dependency on the affected
libraries, take it into account by other means and ignore it when
solely a new RPATH is written to the binaries. See the following
CMakeLists.txt file et al. for an example how to achieve this:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(RPATH C)
SET(CMAKE_VERBOSE_MAKEFILE ON)
ADD_EXECUTABLE(gen gen.c)
ADD_CUSTOM_COMMAND(OUTPUT main.c COMMAND gen > main.c DEPENDS gen)
ADD_EXECUTABLE(main main.c)
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
ADD_LIBRARY(f SHARED f.c)
IF(NOT RPATH)
ADD_CUSTOM_COMMAND(TARGET f POST_BUILD COMMAND ${CMAKE_COMMAND} -E
copy $ ${CMAKE_BINARY_DIR}/libf_gen.so)
ENDIF()
ADD_LIBRARY(g SHARED g.c)
IF(NOT RPATH)
ADD_CUSTOM_COMMAND(TARGET g POST_BUILD COMMAND ${CMAKE_COMMAND} -E
copy $ ${CMAKE_BINARY_DIR}/libg_gen.so)
ENDIF()
INSTALL(TARGETS f g LIBRARY DESTINATION lib)
ADD_CUSTOM_TARGET(gen_libs
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target f
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target g)
ADD_DEPENDENCIES(gen gen_libs)
ADD_LIBRARY(f_gen SHARED IMPORTED)
SET_TARGET_PROPERTIES(f_gen PROPERTIES
IMPORTED_LOCATION ${CMAKE_BINARY_DIR}/libf_gen.so)
ADD_LIBRARY(g_gen SHARED IMPORTED)
SET_TARGET_PROPERTIES(g_gen PROPERTIES
IMPORTED_LOCATION ${CMAKE_BINARY_DIR}/libg_gen.so)
TARGET_LINK_LIBRARIES(gen f_gen g_gen)
/* f.c: */
void f(void){}
/* g.c: */
void g(void){}
/* gen.c: */
#include
int main(void)
{
printf("int main(void){return 0;}\n"); return 0;
}
The executable gen is a code generator that produces the main target's
source code and is linked against the libraries f and g which in turn
are affected by the RPATH placeholder mechanism. If I'm not mistaken,
this models the setup of your project as you have outlined it in your
previous posting. Now, if there's the usual dependency of gen on f/g,
i.e. TARGET_LINK_LIBRARIES(gen f g), the generator would become out-
of-date when f or g is touched, including simple RPATH changes, and
this would result in the generator and the entire depending stuff
being rebuilt. So, the job is to address gen's dependency on f/g.
The dependency of gen on f/g is realised in an unusual manner: The
libraries have a POST_BUILD custom command that copies the library
files libf.so and libg.so to libf_gen.so and libg_gen.so, and gen is
linked against these copies. Additionally, gen depends on the custom
target gen_lib which (re)builds the libraries including their copies
via "cmake --build". In this way, whenever gen is to be rebuilt, the
libraries are rebuilt before - if necessary - and gen links against
the updated libf_gen.so and libg_gen.so. Such a dependency is *not*
tracked by CMake, so it might be dissolved without invalidating the
generator target. This is done by reconfiguring the project while not
taking the libraries' POST_BUILD custom commands into account anymore,
i.e. a reconfiguration with -DRPATH=ON, cf. the IF(NOT RPATH)-ENDIF()
constructs around the custom commands. Thereafter, lib{f,g}_gen.so are
not updated along with lib{f,g}.so and accordingly, the generator with
its file-level dependencies on solely the formers is not relinked, i.e.
gen's dependency on f and g is removed without invalidating gen at the
same time. Afterwards, you can change CMAKE_INSTALL_PREFIX and rebuild
and you'll see that just lib{f,g}.so are relinked with the new RPATH's
placeholders, but gen and especially main are left alone. A subsequent
"make install" installs the project with lib{f,g}.so equipped with the
final RPATH
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
> If I understand correctly, it's this second point which causes your > astonishment, but it already happens during the build phase in the > build tree, not just when installing with "make install", right? Yes, this is indeed my astonishment. I understand that for 'normal' projects this re-link in the build tree will be almost unnoticeable. Unfortunately in our project we build at least one executable using these libraries that is a code generator and therefore all generated code for many other libraries gets regenerated, re-compiled and re-linked. It is a major re-build just for changing the length of the installation directory. So, I see that the only solution I have is to avoid using CMAKE_INSTALL_RPATH. Thanks very much for your explanations. Regards, Pere - Pere Mato CERN, PH Department, CH 1211 Geneva 23, Switzerland e-mail: pere.m...@cern.ch tel: +41 22 76 78696 fax: +41 22 76 68792gsm: +41 76 48 70855 ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
On 05/03/2011 11:17 AM, Pere Mato Vila wrote:
>
>> CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
>> PROJECT(PREFIX C)
>> SET(CMAKE_VERBOSE_MAKEFILE ON)
>> ADD_LIBRARY(f SHARED f.c)
>> INSTALL(TARGETS f
>>RUNTIME DESTINATION bin
>>LIBRARY DESTINATION lib
>>ARCHIVE DESTINATION lib)
>>
>> with ${CMAKE_SOURCE_DIR}/f.c containing just "void f(void){}". After
>> configuring/building/installing, a reconfiguration with a modified
>> CMAKE_INSTALL_PREFIX does not result in recompiling or relinking.
>
>
> Hi Michael,
>
> In you simple example adding SET(CMAKE_INSTALL_RPATH
> "${CMAKE_INSTALL_PREFIX}/lib") before ADD_LIBRARY(...) does trigger a
> re-build of the library when changing the length of CMAKE_INSTALL_PREFIX. The
> reason for this behavior was given by Alex Neundorf:
>
>> Then you set the install RPATH once to "/build/mato/ROOT/install/lib" and
>> then
>> to "/build/mato/ROOT/install2/lib", which is one byte longer. So the binary
>> has to be created again with one byte more room for the install RPATH.
Yes, that's true, indeed, but one must distinguish two factors:
1. Basically, it's the incorporation of the CMAKE_INSTALL_PREFIX in the
library via CMAKE_INSTALL_RPATH that causes the library to be relinked
when CMAKE_INSTALL_PREFIX changes. This is why I have said that solely
changing the latter doesn't cause any recompilation or relinking; one
rather needs further interconnections for this to happen.
2. The, say, placeholder mechanism described by Alex is the reason why
the library is relinked in the *build tree* when CMAKE_INSTALL_RPATH's
length changes although the latter isn't incorporated in the library's
build-tree-instance due to CMAKE_BUILD_WITH_INSTALL_RPATH being FALSE.
If I understand correctly, it's this second point which causes your
astonishment, but it already happens during the build phase in the
build tree, not just when installing with "make install", right?
> The fact that the build is re-done because I change the length of the
> installation destination I would call this a bug and not really a feature.
IMO, it's not a bug, but even quite clever. The placeholder mechanism
allows to avoid a complete relink operation at installation time when
the final RPATH must be written to the installed binaries. Look at the
cmake_install.cmake script for the above-noted project with the line
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib") you mentioned.
You will find a FILE(INSTALL ...) followed by a FILE(RPATH_CHANGE ...)
which operates on the already installed library. The latter command is
implemented by cmSystemTools::ChangeRPath() and probably much cheaper
than a full-featured invocation of the linker involving fork()/exec(),
the compiler's front-end, object files, libraries, LTO, recompilation
of automatically generated source code etc. In other words: Linking
might be expensive, but patching binaries is not.
Now, with the placeholder mechanism, you currently have two (re)link
operations: One for each CMAKE_INSTALL_PREFIX in the build tree, but
none during the two installations. Without the placeholder mechanism,
you'd need three operations: One in the build tree without the final
RPATHs or placeholders and one during each of the two installations.
Therefore, the placeholder mechanism is highly useful - despite the
potentially surprising relink operations within the build tree.
Regards,
Michael
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
> Are you sure you actually need this RPATH in this library ? In the link > command you posted there are no libraries linked... You are right, this particular library is the lowest level library and does no link with any-other, but there are another 85 libraries in the project that would require the RPATH. ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
> CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
> PROJECT(PREFIX C)
> SET(CMAKE_VERBOSE_MAKEFILE ON)
> ADD_LIBRARY(f SHARED f.c)
> INSTALL(TARGETS f
>RUNTIME DESTINATION bin
>LIBRARY DESTINATION lib
>ARCHIVE DESTINATION lib)
>
> with ${CMAKE_SOURCE_DIR}/f.c containing just "void f(void){}". After
> configuring/building/installing, a reconfiguration with a modified
> CMAKE_INSTALL_PREFIX does not result in recompiling or relinking.
Hi Michael,
In you simple example adding SET(CMAKE_INSTALL_RPATH
"${CMAKE_INSTALL_PREFIX}/lib") before ADD_LIBRARY(...) does trigger a re-build
of the library when changing the length of CMAKE_INSTALL_PREFIX. The reason for
this behavior was given by Alex Neundorf:
> Then you set the install RPATH once to "/build/mato/ROOT/install/lib" and
> then
> to "/build/mato/ROOT/install2/lib", which is one byte longer. So the binary
> has to be created again with one byte more room for the install RPATH.
The fact that the build is re-done because I change the length of the
installation destination I would call this a bug and not really a feature.
-
Pere Mato CERN, PH Department, CH 1211 Geneva 23, Switzerland
e-mail: pere.m...@cern.ch tel: +41 22 76 78696
fax: +41 22 76 68792gsm: +41 76 48 70855
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
On 04/29/2011 03:39 PM, Pere Mato Vila wrote:
> Perhaps somebody can give me some hints on this problem I have. I build a
> project and install it in /install with "make install". Then I do
> change the installation prefix with
> cmake -DCMAKE_INSTALL_PREFIX=/install2 ../root_cmake/
> and if I execute again "make install" it rebuilds basically every library of
> the project. I have traced the dependency to the generated link.txt file,
> which contains the link command. In one case the contents of the file is:
>
> /usr/bin/c++ -fPIC -pipe -m64 -Wall -W -Woverloaded-virtual -fPIC -O2 -g
> -shared -Wl,-soname,libCint.so -o ../../lib/libCint.so
> CMakeFiles/Cint.dir/src/loadfile.o ../../lib/libCint_static.a -lpthread
> -Wl,-rpath,
>
> and in the other is:
>
> /usr/bin/c++ -fPIC -pipe -m64 -Wall -W -Woverloaded-virtual -fPIC -O2 -g
> -shared -Wl,-soname,libCint.so -o ../../lib/libCint.so
> CMakeFiles/Cint.dir/src/loadfile.o ../../lib/libCint_static.a -lpthread
> -Wl,-rpath,:
>
> Notice the different number of colons in the -rpath settings. The first
> question I have is what is the origin of these many colons in the rpath
> settings? The second question, why is different when I choose another
> installation directory. My CMake variables that control RPATH are the
> following:
>
> first
>
> CMAKE_INSTALL_RPATH -->/build/mato/ROOT/install/lib
> CMAKE_INSTALL_RPATH_USE_LINK_PATH -->TRUE
> CMAKE_SKIP_BUILD_RPATH -->FALSE
> CMAKE_BUILD_WITH_INSTALL_RPATH -->FALSE
>
> and later
>
> CMAKE_INSTALL_RPATH -->/build/mato/ROOT/install2/lib
> CMAKE_INSTALL_RPATH_USE_LINK_PATH -->TRUE
> CMAKE_SKIP_BUILD_RPATH -->FALSE
> CMAKE_BUILD_WITH_INSTALL_RPATH -->FALSE
AFAIK, changing CMAKE_INSTALL_PREFIX - or any other variable in CMake's
cache - doesn't trigger a recompilation or relinking by itself. Look at
the following CMakeLists.txt
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(PREFIX C)
SET(CMAKE_VERBOSE_MAKEFILE ON)
ADD_LIBRARY(f SHARED f.c)
INSTALL(TARGETS f
RUNTIME DESTINATION bin
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib)
with ${CMAKE_SOURCE_DIR}/f.c containing just "void f(void){}". After
configuring/building/installing, a reconfiguration with a modified
CMAKE_INSTALL_PREFIX does not result in recompiling or relinking.
Nevertheless, what may be very well is that there are files which get
generated during a (re)configuration, typically configured headers, so
their regeneration will invalidate other files that depend on them and,
thus, trigger recompilations and the relinking of the affected targets.
There's a further mechanism beyond the regeneration of headers or
source files that triggers recompilations in this regard: Add
SET_SOURCE_FILES_PROPERTIES(
${CMAKE_SOURCE_DIR}/f.c
PROPERTIES
COMPILE_DEFINITIONS "PREFIX=${CMAKE_INSTALL_PREFIX}")
to the above-noted CMakeLists.txt and you'll see that a modified
CMAKE_INSTALL_PREFIX does result in a rebuild now. Refer to the
${CMAKE_BINARY_DIR}/CMakeFiles/f.dir/flags.make file for the
reason.
Regards,
Michael
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Changing installation prefix triggers re-linking of all libraries
On Friday 29 April 2011, Pere Mato Vila wrote: > Perhaps somebody can give me some hints on this problem I have. I build a > project and install it in /install with "make install". Then I do > change the installation prefix with cmake > -DCMAKE_INSTALL_PREFIX=/install2 ../root_cmake/ > and if I execute again "make install" it rebuilds basically every library > of the project. I have traced the dependency to the generated link.txt > file, which contains the link command. In one case the contents of the file > is: > > /usr/bin/c++ -fPIC -pipe -m64 -Wall -W -Woverloaded-virtual -fPIC -O2 -g > -shared -Wl,-soname,libCint.so -o ../../lib/libCint.so > CMakeFiles/Cint.dir/src/loadfile.o ../../lib/libCint_static.a -lpthread > -Wl,-rpath, > > and in the other is: > > /usr/bin/c++ -fPIC -pipe -m64 -Wall -W -Woverloaded-virtual -fPIC -O2 -g > -shared -Wl,-soname,libCint.so -o ../../lib/libCint.so > CMakeFiles/Cint.dir/src/loadfile.o ../../lib/libCint_static.a -lpthread > -Wl,-rpath,: > > Notice the different number of colons in the -rpath settings. The first > question I have is what is the origin of these many colons in the rpath > settings? When cmake builds a binary, it builds it by default with the build RPATH. This means in the ELF header the entry for the RPATH is created as long as necessary. When installing, cmake then patches the build RPATH out of the binaries and replaces it with the install RPATH. Now if the install RPATH is longer than the build RPATH, the build RPATH is filled up with colons until it is long enough so that at make install cmake can successfully write the install RPATH in this location. In your case, the build RPATH is empty. This seems to be the case because your libCint.so doesn't link against any other shared libraries. Then you set the install RPATH once to "/build/mato/ROOT/install/lib" and then to "/build/mato/ROOT/install2/lib", which is one byte longer. So the binary has to be created again with one byte more room for the install RPATH. Are you sure you actually need this RPATH in this library ? In the link command you posted there are no libraries linked... Alex ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
