Hi Wes, Here is the shell transcript [1], I ran it with your patch included, and it's still not resolved.
[1] https://gist.github.com/keithgchapman/b3e9ee2df2d2a3e6d6efde1f2b6c6d89 Regards, Keith. http://keith-chapman.com On Wed, Feb 22, 2017 at 1:14 PM, Wes McKinney <wesmck...@gmail.com> wrote: > Thanks, this is helpful. The shared library 3rd-party dependencies are > not being properly linked in the executables. It seems (?) it could > depend on the CMake version. For example, on my system (Ubuntu 14.04) > with > > $ cmake --version > cmake version 2.8.12.2 > > with $CC and $CXX set and > > cmake .. -DPARQUET_ARROW=on -DCMAKE_BUILD_TYPE=release > -DPARQUET_BUILD_TESTS=off > > I have the linker command for parquet-scan: > > Linking CXX executable ../release/parquet-scan > cd /home/wesm/code/parquet-cpp/test-build/tools && /usr/bin/cmake -E > cmake_link_script CMakeFiles/parquet-scan.dir/link.txt --verbose=1 > ccache /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/g++ > -O3 -g -std=c++11 -Wall -fno-strict-aliasing -fvisibility=hidden > -fvisibility-inlines-hidden -O3 -DNDEBUG > CMakeFiles/parquet-scan.dir/parquet-scan.cc.o -o > ../release/parquet-scan -rdynamic /usr/lib/libsnappy.a > /home/wesm/code/native-toolchain/build/thrift-0.9.0-p2/lib/libthrift.a > ../release/libparquet.a /usr/lib/libsnappy.a > /home/wesm/code/native-toolchain/build/thrift-0.9.0-p2/lib/libthrift.a > /home/wesm/local/lib/libarrow.so /home/wesm/local/lib/libarrow_io.so > /home/wesm/code/native-toolchain/build/boost-1.57.0/lib/libboost_regex.so > /home/wesm/local/lib/libbrotlidec.a > /home/wesm/local/lib/libbrotlienc.a > /home/wesm/local/lib/libbrotlicommon.a > ../zlib_ep/src/zlib_ep-install/lib/libz.a > -Wl,-rpath,/home/wesm/local/lib:/home/wesm/code/native- > toolchain/build/boost-1.57.0/lib > > Here you can see all the link requirements for using libparquet.a have > been included > > To unblock yourself, I suggest passing > > -DPARQUET_BUILD_EXECUTABLES=off > > I'll open a blocker JIRA for 1.0.0 > (https://issues.apache.org/jira/browse/PARQUET-892) until we figure > out what's going on. Since I can't reproduce this can you run another > build with > > make VERBOSE=1 > > so I can see all the linker commands? Please also build with only one > thread. Thanks > > I'm going to put up a patch quickly for you to try. Will ping you on GitHub > > - Wes > > On Wed, Feb 22, 2017 at 3:24 PM, Keith Chapman <keithgchap...@gmail.com> > wrote: > > Hi Wes, > > > > You can find the shell transcript here [1]. > > > > [1] https://gist.github.com/keithgchapman/ee2ce6f27868f532d7c0dfceaae874 > ed > > > > Regards, > > Keith. > > > > http://keith-chapman.com > > > > On Wed, Feb 22, 2017 at 12:14 PM, Keith Chapman <keithgchap...@gmail.com > > > > wrote: > >> > >> I actually had 2 versions of boost, I built 1.54 once for a different > >> project (which I do not need anymore), I got rid of it and now it picks > up > >> 1.58 but still gives the same error. Will upload a complete shell > >> transcript. > >> > >> Regards, > >> Keith. > >> > >> http://keith-chapman.com > >> > >> On Wed, Feb 22, 2017 at 12:04 PM, Wes McKinney <wesmck...@gmail.com> > >> wrote: > >>> > >>> How did you build Boost? If you could upload a complete shell > >>> transcript starting from a clean git clone (run "git clean -fdx ." in > >>> your clone) to GitHub gist or someplace else I can take a closer look. > >>> Ubuntu 16.04 claims to be shipping Boost 1.58, so you may need to set > >>> a different $BOOST_ROOT or update your Boost. > >>> > >>> https://launchpad.net/ubuntu/xenial/+package/libboost-dev > >>> > >>> On Wed, Feb 22, 2017 at 3:02 PM, Keith Chapman < > keithgchap...@gmail.com> > >>> wrote: > >>> > Interestingly if I build with the following (instead of having > >>> > -DCMAKE_CXX_COMPILER=g++-4.9 -DCMAKE_C_COMPILER=gcc-4.9) > >>> > > >>> > export CC="gcc-4.9" > >>> > export CXX="g++-4.9" > >>> > cmake .. -DCMAKE_BUILD_TYPE=release -DPARQUET_BUILD_TESTS=Off > >>> > > >>> > I get rid of the arrow related linkage errors and end up with only > >>> > boost > >>> > related ones. Cmake shows that I have boost 1.54 and I'm building on > >>> > Ubuntu > >>> > 16.04 > >>> > > >>> > [ 94%] Linking CXX executable ../release/parquet-scan > >>> > ../release/libparquet.a(metadata.cc.o): In function > >>> > `boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > >, boost::regex_traits<char, > >>> > boost::cpp_regex_traits<char> > > >>> >>::match_match()': > >>> > /usr/local/include/boost/regex/v4/perl_matcher_non_ > recursive.hpp:973: > >>> > undefined reference to > >>> > `boost::match_results<__gnu_cxx::__normal_iterator<char const*, > >>> > std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > > > >>> >>::maybe_assign(boost::match_results<__gnu_cxx::__normal_ > iterator<char > >>> > const*, std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > > > const&)' > >>> > ../release/libparquet.a(metadata.cc.o): In function > >>> > `boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > >, boost::regex_traits<char, > >>> > boost::cpp_regex_traits<char> > > >>> >>::match_prefix()': > >>> > /usr/local/include/boost/regex/v4/perl_matcher_common.hpp:333: > >>> > undefined > >>> > reference to `boost::match_results<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > > > >>> >>::maybe_assign(boost::match_results<__gnu_cxx::__normal_ > iterator<char > >>> > const*, std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > > > const&)' > >>> > ../release/libparquet.a(metadata.cc.o): In function `perl_matcher': > >>> > /usr/local/include/boost/regex/v4/perl_matcher.hpp:374: undefined > >>> > reference > >>> > to `boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string>, > >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char > >>> > const*, > >>> > std::string> > >, boost::regex_traits<char, > >>> > boost::cpp_regex_traits<char> > > >>> >>::construct_init(boost::basic_regex<char, boost::regex_traits<char, > >>> > boost::cpp_regex_traits<char> > > const&, > >>> > boost::regex_constants::_match_flags)' > >>> > collect2: error: ld returned 1 exit status > >>> > tools/CMakeFiles/parquet-dump-schema.dir/build.make:106: recipe for > >>> > target > >>> > 'release/parquet-dump-schema' failed > >>> > make[2]: *** [release/parquet-dump-schema] Error 1 > >>> > CMakeFiles/Makefile2:716: recipe for target > >>> > 'tools/CMakeFiles/parquet-dump-schema.dir/all' failed > >>> > make[1]: *** [tools/CMakeFiles/parquet-dump-schema.dir/all] Error 2 > >>> > make[1]: *** Waiting for unfinished jobs.... > >>> > > >>> > Regards, > >>> > Keith. > >>> > > >>> > http://keith-chapman.com > >>> > > >>> > On Wed, Feb 22, 2017 at 11:51 AM, Keith Chapman > >>> > <keithgchap...@gmail.com> > >>> > wrote: > >>> >> > >>> >> Yes, I get it on a clean build. > >>> >> > >>> >> Regards, > >>> >> Keith. > >>> >> > >>> >> http://keith-chapman.com > >>> >> > >>> >> On Wed, Feb 22, 2017 at 11:42 AM, Wes McKinney <wesmck...@gmail.com > > > >>> >> wrote: > >>> >>> > >>> >>> Do you get these errors from a clean build (i.e. you've removed all > >>> >>> the artifacts and CMake files from your prior build)? We are > building > >>> >>> in Travis CI with gcc/g++ 4.9 and have not been having any issues: > >>> >>> > >>> >>> > >>> >>> > >>> >>> https://github.com/apache/parquet-cpp/blob/master/ci/ > before_script_travis.sh#L20 > >>> >>> > >>> >>> On Wed, Feb 22, 2017 at 2:28 PM, Keith Chapman > >>> >>> <keithgchap...@gmail.com> > >>> >>> wrote: > >>> >>> > Hi Wes, > >>> >>> > > >>> >>> > Tried the patch and it does the trick, thank you for the prompt > >>> >>> > response. > >>> >>> > Also I noticed another issue, not sure if you'd regard it as a > bug > >>> >>> > or > >>> >>> > not > >>> >>> > but I'll throw it out for your consideration. If I build the > >>> >>> > parquet > >>> >>> > library with > >>> >>> > > >>> >>> > "cmake . -DCMAKE_BUILD_TYPE=release -DPARQUET_BUILD_TESTS=Off" > >>> >>> > everything > >>> >>> > builds fine as expected, but if I build it with "cmake . > >>> >>> > -DCMAKE_BUILD_TYPE=release -DCMAKE_CXX_COMPILER=g++-4.9 > >>> >>> > -DCMAKE_C_COMPILER=gcc-4.9 -DPARQUET_BUILD_TESTS=Off" I end up > with > >>> >>> > a > >>> >>> > bunch > >>> >>> > of linkage errors as follows, > >>> >>> > > >>> >>> > Linking CXX executable ../release/parquet_reader > >>> >>> > ../release/libparquet.a(reader.cc.o): In function > >>> >>> > > >>> >>> > `parquet::DictionaryDecoder<parquet::DataType<(parquet:: > Type::type)6> > >>> >>> > >>> >>> >> >>::SetDict(parquet::Decoder<parquet::DataType<(parquet:: > Type::type)6> > >>> >>> >> >*)': > >>> >>> > > >>> >>> > > >>> >>> > /home/keith/projects/bs/axstreamBigD/external_ > packages/parquetcpp/src/parquet/encoding-internal.h:382: > >>> >>> > undefined reference to `arrow::Status::ToString() const' > >>> >>> > ../release/libparquet.a(reader.cc.o): In function > >>> >>> > > >>> >>> > `parquet::DictionaryDecoder<parquet::DataType<(parquet:: > Type::type)7> > >>> >>> > >>> >>> >> >>::SetDict(parquet::Decoder<parquet::DataType<(parquet:: > Type::type)7> > >>> >>> >> >*)': > >>> >>> > > >>> >>> > > >>> >>> > /home/keith/projects/bs/axstreamBigD/external_ > packages/parquetcpp/src/parquet/encoding-internal.h:402: > >>> >>> > undefined reference to `arrow::Status::ToString() const' > >>> >>> > ../release/libparquet.a(metadata.cc.o): In function > >>> >>> > `boost::re_detail::perl_matcher<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > >, boost::regex_traits<char, > >>> >>> > boost::cpp_regex_traits<char> > > >>> >>> >>::match_match()': > >>> >>> > > >>> >>> > /usr/local/include/boost/regex/v4/perl_matcher_non_ > recursive.hpp:973: > >>> >>> > undefined reference to > >>> >>> > `boost::match_results<__gnu_cxx::__normal_iterator<char const*, > >>> >>> > std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > > > >>> >>> > >>> >>> >> >>::maybe_assign(boost::match_results<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > > > const&)' > >>> >>> > ../release/libparquet.a(metadata.cc.o): In function > >>> >>> > `boost::re_detail::perl_matcher<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > >, boost::regex_traits<char, > >>> >>> > boost::cpp_regex_traits<char> > > >>> >>> >>::match_prefix()': > >>> >>> > /usr/local/include/boost/regex/v4/perl_matcher_common.hpp:333: > >>> >>> > undefined > >>> >>> > reference to > >>> >>> > `boost::match_results<__gnu_cxx::__normal_iterator<char > >>> >>> > const*, std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > > > >>> >>> > >>> >>> >> >>::maybe_assign(boost::match_results<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, std::string>, > >>> >>> > std::allocator<boost::sub_match<__gnu_cxx::__normal_ > iterator<char > >>> >>> > const*, > >>> >>> > std::string> > > > const&)' > >>> >>> > ../release/libparquet.a(metadata.cc.o): In function > `perl_matcher': > >>> >>> > ..... > >>> >>> > And a lot more > >>> >>> > > >>> >>> > Regards, > >>> >>> > Keith. > >>> >>> > > >>> >>> > http://keith-chapman.com > >>> >>> > > >>> >>> > On Wed, Feb 22, 2017 at 10:57 AM, Wes McKinney > >>> >>> > <wesmck...@gmail.com> > >>> >>> > wrote: > >>> >>> > > >>> >>> >> You can try out this patch if you like: > >>> >>> >> > >>> >>> >> https://github.com/apache/parquet-cpp/pull/258 > >>> >>> >> > >>> >>> >> This should be included in the next 1.0.0 RC > >>> >>> >> > >>> >>> >> On Wed, Feb 22, 2017 at 1:32 PM, Keith Chapman > >>> >>> >> <keithgchap...@gmail.com> > >>> >>> >> wrote: > >>> >>> >> > Thanks Wes. > >>> >>> >> > > >>> >>> >> > Regards, > >>> >>> >> > Keith. > >>> >>> >> > > >>> >>> >> > http://keith-chapman.com > >>> >>> >> > > >>> >>> >> > On Wed, Feb 22, 2017 at 10:30 AM, Wes McKinney > >>> >>> >> > <wesmck...@gmail.com> > >>> >>> >> wrote: > >>> >>> >> >> > >>> >>> >> >> I'm able to reproduce the issue on Ubuntu 14.04 > >>> >>> >> >> > >>> >>> >> >> Linking CXX shared library debug/libparquet.so > >>> >>> >> >> /usr/bin/ld: /usr/lib/libsnappy.a(snappy.o): relocation > >>> >>> >> >> R_X86_64_32S > >>> >>> >> >> against `.rodata' can not be used when making a shared > object; > >>> >>> >> >> recompile with -fPIC > >>> >>> >> >> /usr/lib/libsnappy.a: error adding symbols: Bad value > >>> >>> >> >> collect2: error: ld returned 1 exit status > >>> >>> >> >> > >>> >>> >> >> Patch forthcoming > >>> >>> >> >> > >>> >>> >> >> On Wed, Feb 22, 2017 at 1:29 PM, Keith Chapman > >>> >>> >> >> <keithgchap...@gmail.com > >>> >>> >> > > >>> >>> >> >> wrote: > >>> >>> >> >> > Hi Wes, > >>> >>> >> >> > > >>> >>> >> >> > No I don't have SNAPPY_HOME set. Yes this seems similar to > >>> >>> >> >> > 885 > >>> >>> >> >> > > >>> >>> >> >> > On Feb 22, 2017 10:25 AM, "Wes McKinney" > >>> >>> >> >> > <wesmck...@gmail.com> > >>> >>> >> >> > wrote: > >>> >>> >> >> >> > >>> >>> >> >> >> hi Keith, > >>> >>> >> >> >> > >>> >>> >> >> >> It's not caused by PARQUET-885, but it is the same type of > >>> >>> >> >> >> problem. > >>> >>> >> Do > >>> >>> >> >> >> you have $SNAPPY_HOME set? If you compile your own > >>> >>> >> >> >> thirdparty > >>> >>> >> >> >> libraries, they need to be built with -fPIC to support > >>> >>> >> >> >> static > >>> >>> >> >> >> linking > >>> >>> >> >> >> into a shared library. Otherwise it's easier to let > >>> >>> >> >> >> parquet-cpp > >>> >>> >> >> >> build > >>> >>> >> >> >> its own thirdparty toolchain. Because some Linux > >>> >>> >> >> >> distributions > >>> >>> >> >> >> don't > >>> >>> >> >> >> set -fPIC in their packages, we should make a similar > change > >>> >>> >> >> >> as > >>> >>> >> >> >> PARQUET-885 for Snappy. Looking into it now... > >>> >>> >> >> >> > >>> >>> >> >> >> I'm also going to add a note about this to the README. > >>> >>> >> >> >> > >>> >>> >> >> >> - Wes > >>> >>> >> >> >> > >>> >>> >> >> >> On Wed, Feb 22, 2017 at 1:05 PM, Keith Chapman > >>> >>> >> >> >> <keithgchap...@gmail.com> > >>> >>> >> >> >> wrote: > >>> >>> >> >> >> > Hi, > >>> >>> >> >> >> > > >>> >>> >> >> >> > I'm trying to build master and I get the following > error, > >>> >>> >> >> >> > I > >>> >>> >> >> >> > have > >>> >>> >> the > >>> >>> >> >> >> > .so > >>> >>> >> >> >> > versions of the files as seen below too, Is this > somewhat > >>> >>> >> >> >> > relaped > >>> >>> >> to > >>> >>> >> >> >> > Parquet-885? > >>> >>> >> >> >> > > >>> >>> >> >> >> > ls /usr/lib/x86_64-linux-gnu/libsnappy. > >>> >>> >> >> >> > libsnappy.a libsnappy.so libsnappy.so.1 > >>> >>> >> >> >> > libsnappy.so.1.3.0 > >>> >>> >> >> >> > > >>> >>> >> >> >> > Error seen: > >>> >>> >> >> >> > > >>> >>> >> >> >> > /usr/bin/ld: > >>> >>> >> >> >> > /usr/lib/x86_64-linux-gnu/libsnappy.a(snappy.o): > >>> >>> >> >> >> > relocation > >>> >>> >> >> >> > R_X86_64_32S against `.rodata' can not be used when > making > >>> >>> >> >> >> > a > >>> >>> >> >> >> > shared > >>> >>> >> >> >> > object; > >>> >>> >> >> >> > recompile with -fPIC > >>> >>> >> >> >> > /usr/lib/x86_64-linux-gnu/libsnappy.a: error adding > >>> >>> >> >> >> > symbols: > >>> >>> >> >> >> > Bad > >>> >>> >> >> >> > value > >>> >>> >> >> >> > collect2: error: ld returned 1 exit status > >>> >>> >> >> >> > CMakeFiles/parquet_shared.dir/build.make:115: recipe > for > >>> >>> >> >> >> > target > >>> >>> >> >> >> > 'release/libparquet.so' failed > >>> >>> >> >> >> > make[2]: *** [release/libparquet.so] Error 1 > >>> >>> >> >> >> > CMakeFiles/Makefile2:218: recipe for target > >>> >>> >> >> >> > 'CMakeFiles/parquet_shared.dir/all' failed > >>> >>> >> >> >> > make[1]: *** [CMakeFiles/parquet_shared.dir/all] Error > 2 > >>> >>> >> >> >> > Makefile:138: recipe for target 'all' failed > >>> >>> >> >> >> > make: *** [all] Error 2 > >>> >>> >> >> >> > > >>> >>> >> >> >> > Regards, > >>> >>> >> >> >> > Keith. > >>> >>> >> >> >> > > >>> >>> >> >> >> > http://keith-chapman.com > >>> >>> >> > > >>> >>> >> > > >>> >>> >> > >>> >> > >>> >> > >>> > > >> > >> > > >
