[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 20 Sep 2019 01:07:37 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Reviewed-on: http://gerrit.cloudera.org:8080/6521 Reviewed-by: Joe McDonnell Tested-by: Tim Armstrong --- M functions.sh M init-compiler.sh M source/cmake/build.sh M source/flatbuffers/build.sh M source/kudu/build.sh M source/llvm/build.sh M source/protobuf/build.sh M source/thrift/build.sh 8 files changed, 136 insertions(+), 11 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 13: Code-Review+2 Looks good -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 20 Sep 2019 00:07:11 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#13). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh M source/flatbuffers/build.sh M source/kudu/build.sh M source/llvm/build.sh M source/protobuf/build.sh M source/thrift/build.sh 8 files changed, 136 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/13 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#12). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh M source/flatbuffers/build.sh M source/kudu/build.sh M source/llvm/build.sh M source/protobuf/build.sh M source/thrift/build.sh 8 files changed, 138 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/12 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Sep 2019 21:59:46 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 11: Sorry for the churn, I finally got to a version of this that worked OK on all OSes and in my non-hermetic dev environment. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Sep 2019 21:59:20 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#11). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh M source/flatbuffers/build.sh M source/kudu/build.sh M source/llvm/build.sh M source/protobuf/build.sh M source/thrift/build.sh 8 files changed, 139 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/11 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 10: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/6521/10/source/kudu/build.sh File source/kudu/build.sh: http://gerrit.cloudera.org:8080/#/c/6521/10/source/kudu/build.sh@98 PS10, Line 98: LD_LIBRARY_PATH="" wrap ./build-if-necessary.sh Found a minor issue building on my system where ninja was installed. The docker build was not affected. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Sep 2019 22:30:51 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 10: I'm going to do a full docker build with the final version of the code change, then verify, then submit. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Sep 2019 22:31:12 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/6521 )
Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
..
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6521/9/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/6521/9/functions.sh@615
PS9, Line 615: print os.path.relpath('${src}', '${dst}')
> Something I forgot before: We could future-proof this for Python3 by using
Done
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh
File init-compiler.sh:
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh@62
PS9, Line 62: linker
> Nit: I think it would be clearer if we used "dynamic linker"
Done
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh@74
PS9, Line 74: We need to make sure they link against the correct libraries
because
: # the libraries are not symlinked until after the component
build completes.
> Let me check my understanding:
Done
--
To view, visit http://gerrit.cloudera.org:8080/6521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67
Gerrit-Change-Number: 6521
Gerrit-PatchSet: 9
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 18 Sep 2019 22:30:14 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#10). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh M source/kudu/build.sh 4 files changed, 117 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/10 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/6521 )
Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
..
Patch Set 9: Code-Review+2
(3 comments)
Couple nits, but otherwise, I think this looks good.
http://gerrit.cloudera.org:8080/#/c/6521/9/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/6521/9/functions.sh@615
PS9, Line 615: print os.path.relpath('${src}', '${dst}')
Something I forgot before: We could future-proof this for Python3 by using
print with parens: print(os.path.relpath('${src}', '${dst}'))
That syntax is backwards compatible on python 2.
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh
File init-compiler.sh:
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh@62
PS9, Line 62: linker
Nit: I think it would be clearer if we used "dynamic linker"
http://gerrit.cloudera.org:8080/#/c/6521/9/init-compiler.sh@74
PS9, Line 74: We need to make sure they link against the correct libraries
because
: # the libraries are not symlinked until after the component
build completes.
Let me check my understanding:
The static linker that produced the executable knows about the gcc libs because
we pass in the gcc libraries via FULL_LPATH's -L flag. It doesn't care about
LD_LIBRARY_PATH.
The dynamic linker needs to be able to find the library at runtime. The RPATH
doesn't have it yet, because the symlink happens later. So, we need
LD_LIBRARY_PATH.
Nit: I think it would be nice for the comment to clarify that this is the
dynamic linker at runtime. Something like the dynamic linker needs to be able
to find the correct libraries at runtime etc etc.
--
To view, visit http://gerrit.cloudera.org:8080/6521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67
Gerrit-Change-Number: 6521
Gerrit-PatchSet: 9
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 18 Sep 2019 18:46:23 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/6521 )
Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
..
Patch Set 8:
(2 comments)
I had to make a couple of other changes:
* I had some issues with ccache and cmake locally, so disabled the redundant
ccache enablement for the cmake build
* I moved the LD_LIBRARY_PATH setup to a shared location instead of doing it
per-component, since I ran into the same problem for cmake, flatbuffers and
kudu.
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh@612
PS7, Line 612: nction calc_relpath() {
> This is stale. and must be relative paths from the curre
Done
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh@626
PS7, Line 626: # The lib/ subfolder is on the rpath of all binaries. Add a
symlink there.
: if [[ "$(basename "$binary_dir")" == "lib" ]]; then
: # Don't insert extraneous ../lib
: local dst_lib_dir="$binary_dir"
: else
: local dst_lib_dir="$binary_dir/../lib"
: fi
: loca
> I'm thinking we might want to use python for this relative path calculation
For some reason I didn't think we could use python when I wrote this, but it is
clearly a dependency based on the dockerfiles. This is nicer.
--
To view, visit http://gerrit.cloudera.org:8080/6521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67
Gerrit-Change-Number: 6521
Gerrit-PatchSet: 8
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 18 Sep 2019 17:17:21 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#9). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh 3 files changed, 111 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/9 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#8). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Also remove redundant ccache enablement in the cmake build scripts. We already use ccache globally via a wrapper script. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/cmake/build.sh 3 files changed, 112 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/8 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/6521 )
Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
..
Patch Set 7:
(2 comments)
Overall, this makes sense to me.
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh@612
PS7, Line 612: and must be relative paths from the current
directory.
This is stale. and must be relative paths from the current
directory.
http://gerrit.cloudera.org:8080/#/c/6521/7/functions.sh@626
PS7, Line 626: # We need to figure out how many "../" are in the relative
path from the dst dir to
: # to the src dir, since the dst may be nested arbitrarily
deeply (e.g. debug/bin/).
: local rel_path=""
: local curr_dir="$binary_dir"
: while [[ "$curr_dir" != "." ]]; do
: curr_dir="$(dirname "$curr_dir")"
: rel_path="../$rel_path"
: done
I'm thinking we might want to use python for this relative path calculation.
os.path.relpath handles everything nicely:
https://docs.python.org/2/library/os.path.html#os.path.relpath
It was added in python 2.6, which should be ok.
I think this would do it:
python -c "import os.path; print os.path.relpath('${src_lib}',
'${dst_lib_dir}')"
(Or you could do it after pushd into ${dst_lib_dir} and then you don't need the
second argument.)
This would handle things even if the inputs are absolute paths.
--
To view, visit http://gerrit.cloudera.org:8080/6521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67
Gerrit-Change-Number: 6521
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 17 Sep 2019 23:38:21 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#7). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/flatbuffers/build.sh M source/kudu/build.sh 4 files changed, 121 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/7 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 6: This is still relevant if someone has patience to read through all the bash. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 16:44:56 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#6). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/flatbuffers/build.sh M source/kudu/build.sh 4 files changed, 122 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/6 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 5: Verified+1 Fixed the issue, confirmed that it builds on all platforms. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Jun 2018 22:51:35 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#5). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/flatbuffers/build.sh M source/kudu/build.sh 4 files changed, 122 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/5 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
