I'm a bit confused. The text of the error was "You need to implement atomic operations for this architecture", right? Maybe could you be a bit more descriptive about what your understanding of that is and why it is a surprising error?
On Tue, Aug 27, 2019 at 2:51 AM Koltsov Dmitriy <[email protected]> wrote: > Ok, Tim. > > Now it's clear - github.com/apache/impala and > github.com/cloudera/native-toolchain should be used for build in Ubuntu > 16.04 ppc64le. > > But then still there is the question - could you, please, give some > comments about possible reason of the error shown below when I tried this > combination of impala and native-toolchain ? ... and about possible ways to > solve it ? May be there is some another commit of impala which is more > "friendly" with ppc64le and native-toolchain for now ? > > >>And I have a lot of error messages while compiling process, e.g.: > >> > >> > >>... > >>make[2]: *** Deleting file > 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp' > >>In file included from > /home/build/impala/be/src/gutil/spinlock_internal.h:40:0, > >> from > /home/build/impala/be/src/gutil/spinlock_internal.cc:44: > >>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You need > to implement atomic operations for this architecture > >> #error You need to implement atomic operations for this architecture > >> ^ > >>/home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â > does not name a type > >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr, > >> > >>... > >> > >> Impala build process failed. > > > > > On 08/27/19 01:51, Tim Armstrong wrote: > > Hi Dmitry, > > > But only by that moment I've noticed that there is a significant > difference between github.com/cloudera/Impala and github.com/apache/impala > - while build. > > And all previous steps I tried with github apache impala and cloudera > native-toolchain. > That is the right thing to do. Apache Impala master is the latest and > greatest. github.com/cloudera/impala mainly hosts snapshots of the source > from different Cloudera releases. It looks like the default branch was > cdh5-trunk, which is a fairly old version of Impala at this point. > > > > On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy <[email protected]> > wrote: > >> Hello Tim. >> >> After some tricks I have success in building Kudu stub. Thanks for advice. >> >> But only by that moment I've noticed that there is a significant >> difference between github.com/cloudera/Impala and >> github.com/apache/impala - while build. >> And all previous steps I tried with github apache impala and cloudera >> native-toolchain. >> >> So, for now, after I successfully got Kudu stub, I've tried to build >> final Impala build with github apache impala and cloudera native-toolchain >> in Ubuntu 16.04 ppc64le. And I have a lot of error messages while compiling >> process, e.g.: >> >> ... >> make[2]: *** Deleting file >> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp' >> In file included from >> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0, >> from >> /home/build/impala/be/src/gutil/spinlock_internal.cc:44: >> /home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You need >> to implement atomic operations for this architecture >> #error You need to implement atomic operations for this architecture >> ^ >> /home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â does >> not name a type >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr, >> >> ... >> >> Impala build process failed. >> >> >> >> So, hence now I'm trying only to use cloudera github impala and cloudera >> native-toolchain. I use previously build cloudera native-toolchain (git >> cloned from github), set IMPALA_TOOLCHAIN var to the appropriate location; >> current commit of cloudera impala is 7dd2c81. But when ./buildall.sh is >> executed I have the following error: >> >> -- Added static library dependency pprof: >> /home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a >> -- ----------> Adding thirdparty library gtest. <---------- >> -- Header files: /home/build/impala/toolchain/gtest-1.6.0/include >> -- Added static library dependency gtest: >> /home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a >> CMake Error at cmake_modules/FindLlvmBinaries.cmake:33 (message): >> Could not find llvm-config >> Call Stack (most recent call first): >> CMakeLists.txt:208 (find_package) >> >> >> -- Configuring incomplete, errors occurred! >> See also "/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log". >> See also "/home/build/impala_cloudera/CMakeFiles/CMakeError.log". >> Error in /home/build/impala_cloudera/bin/make_impala.sh at line 161: >> cmake . ${CMAKE_ARGS[@]} >> >> Though IMPALA_TOOLCHAIN path points to directory which contains LLVM >> build: "llvm-5.0.1-p1" dir is in the path, with all appropriate files. >> >> >> >> >> >> *Question:* Could you, please, give a comment about possible reason of >> the error and about how it could be resolved ? >> >> >> >> >> >> btw: there is some inconsistency about llvm versions between one in >> ./bin/impala-config.sh (export IMPALA_LLVM_VERSION=3.9.1) and default one >> of llvm build in cloudera native-toolchain (5.0.1-p1). >> >> >> >> >> >> >> On 08/24/19 03:49, Tim Armstrong wrote: >> >> Ah I see. I think if you run bin/bootstrap_toolchain.py manually it will >> build the stub for you. >> >> I wouldn't be surprised if this requires some tweaks to the scripts to >> make it work since they weren't written for this exact case in mind (the >> original use case for the kudu stubs was for OSes that Kudu didn't support >> but Impala did, like CentOS 5). >> >> On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy <[email protected]> >> wrote: >> >>> Hello, Tim. >>> >>> Thanks for reply. >>> >>> When I'm trying to build Impala I am of course using already built >>> clouder native-toolchain (cloned from github repo). This native-toolchain >>> has no >>> >>> kudu in final build directory. So, when I'm trying to build Impala >>> pointing this build through IMPALA_TOOLCHAIN variable with >>> SKIP_TOOLCHAIN_BOOSTRAP=true then I have an error mentioned by me above >>> (see the fragment below). Could you, please, inform if there is a >>> possibility to instruct Impala building process not to use Kudu in Impala >>> build on Ubuntu 16.04 ppc64le ? >>> >>> > > -- ----------> Adding thirdparty library libunwind. <----------> >>> >>> > > -- Header files: >>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include> >>> >>> > > -- Added shared library dependency libunwind:> >>> >>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/> >>> >>> > > CMake Error at CMakeLists.txt:409 (find_package):> >>> >>> > > Could not find a package configuration file provided by "kuduClient" >>> with> >>> >>> > > any of the following names:> >>> >>> > >> >>> >>> > > kuduClientConfig.cmake> >>> >>> > > kuduclient-config.cmake> >>> >>> > >> >>> >>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or >>> set> >>> >>> > > "kuduClient_DIR" to a directory containing one of the above files. >>> If> >>> >>> > > "kuduClient" provides a separate development package or SDK, be sure >>> it> >>> >>> > > has> >>> >>> > > been installed.> >>> >>> > >> >>> >>> > >> >>> >>> > > -- Configuring incomplete, errors occurred!> >>> >>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".> >>> >>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".> >>> >>> > > /home/build/impala#> >>> >>> > > /home/build/impala#> >>> >>> On 2019/08/21 16:47:48, Tim Armstrong <[email protected]> >>> <[email protected]> wrote: >>> >>> > Hi Dmitry,> >>> >>> > >>> >>> > The toolchain bootstrap process downloads pre-built binaries, and we >>> don't> >>> >>> > have any for PPC.> >>> >>> > >>> >>> > You would need to build the toolchain locally for PPC, then point the> >>> >>> > Impala build at it. These instructions might be helpful for you -> >>> >>> > >>> https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala> >>> >>> >>> > .> >>> >>> > >>> >>> > >>> >>> > >>> >>> > On Tue, Aug 20, 2019 at 3:53 PM Koltsov Dmitriy <[email protected]> >>> <[email protected]> wrote:> >>> >>> > >>> >>> > > I've looked at the contents of impala-config.sh. As I understand - >>> I> >>> >>> > > should manually set KUDU_IS_SUPPORTED=false in order to make general >>> impala> >>> >>> > > build process to download kudu and build stub. But I use >>> native_toolchain> >>> >>> > > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true. So it seems that there is >>> mutual> >>> >>> > > exclusion: if I want to build impala with github cloudera >>> native_toolchain> >>> >>> > > then code branch for kudu stub building is disabled> >>> >>> > > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set> >>> >>> > > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable general build process to >>> download> >>> >>> > > and build kudu stub then there will be no other components available >>> from> >>> >>> > > cloudera native_toolchain.> >>> >>> > >> >>> >>> > > For example, when I use first case with >>> SKIP_TOOLCHAIN_BOOTSTRAP=true and> >>> >>> > > manually KUDU_IS_SUPPORTED=false set in impala-config.sh, then I >>> have the> >>> >>> > > following error when trying ./buildall.sh for cloudera impala:> >>> >>> > >> >>> >>> > > ...> >>> >>> > > -- ----------> Adding thirdparty library krb5. <----------> >>> >>> > > -- Header files: /usr/include> >>> >>> > > -- Added shared library dependency krb5:> >>> >>> > > /usr/lib/powerpc64le-linux-gnu/libkrb5.so> >>> >>> > > -- Found Kerberos: /usr/sbin/kadmin.local> >>> >>> > > -- Added shared library dependency rt:> >>> >>> > > /usr/lib/powerpc64le-linux-gnu/librt.so> >>> >>> > > -- Added shared library dependency dl:> >>> >>> > > /usr/lib/powerpc64le-linux-gnu/libdl.so> >>> >>> > > -- Found LIBUNWIND:> >>> >>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/> >>> >>> > > -- ----------> Adding thirdparty library libunwind. <----------> >>> >>> > > -- Header files: >>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include> >>> >>> > > -- Added shared library dependency libunwind:> >>> >>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/> >>> >>> > > CMake Error at CMakeLists.txt:409 (find_package):> >>> >>> > > Could not find a package configuration file provided by "kuduClient" >>> with> >>> >>> > > any of the following names:> >>> >>> > >> >>> >>> > > kuduClientConfig.cmake> >>> >>> > > kuduclient-config.cmake> >>> >>> > >> >>> >>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or >>> set> >>> >>> > > "kuduClient_DIR" to a directory containing one of the above files. >>> If> >>> >>> > > "kuduClient" provides a separate development package or SDK, be sure >>> it> >>> >>> > > has> >>> >>> > > been installed.> >>> >>> > >> >>> >>> > >> >>> >>> > > -- Configuring incomplete, errors occurred!> >>> >>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".> >>> >>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".> >>> >>> > > /home/build/impala#> >>> >>> > > /home/build/impala#> >>> >>> > >> >>> >>> > >> >>> >>> > > Could you please describe how can one exactly build cloudera impala >>> with> >>> >>> > > its native_toolchain and kudu stub simultaneously on Ubuntu 16.04 >>> ppc64le ?> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > > Regards,> >>> >>> > > Koltsov Dmitriy> >>> >>> > > System Engineer, YADRO> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > > On 08/16/19 18:15, Lars Volker wrote:> >>> >>> > >> >>> >>> > > For the Impala toolchain you can start with this list of commits> >>> >>> > > <https://gerrit.cloudera.org/#/q/ppc64le> >>> <https://gerrit.cloudera.org/#/q/ppc64le>, in particular this review> >>> >>> > > <https://gerrit.cloudera.org/#/c/9077/> >>> <https://gerrit.cloudera.org/#/c/9077/>.> >>> >>> > >> >>> >>> > > I don't know about kudu, you might want to ask on their mailing list >>> or> >>> >>> > > their slack channel. If you look in impala-config.sh for> >>> >>> > > the KUDU_IS_SUPPORTED flag, you should find a way to build Impala >>> without> >>> >>> > > Kudu.> >>> >>> > >> >>> >>> > > Cheers, Lars> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > > On Fri, Aug 16, 2019 at 5:32 AM Koltsov Dmitriy <[email protected]> >>> <[email protected]>> >>> >>> > > wrote:> >>> >>> > >> >>> >>> > >> Hello, Lars!> >>> >>> > >>> >>> >>> > >> Thanks for operative answer.> >>> >>> > >>> >>> >>> > >> And could please inform about actual commit number of the> >>> >>> > >> native-toolchain on github which can be build successfully at that >>> moment ?> >>> >>> > >>> >>> >>> > >> Btw: I ask about it because build process for current commit of> >>> >>> > >> native-toolchain finishes with error. And error is because of kudu >>> build> >>> >>> > >> fail (other components of toolchain have been built fine). For now >>> build.sh> >>> >>> > >> file in this commit of kudu src have explicit switch to disable >>> ppc64le.> >>> >>> > >> And if I manually enable ppc64le case then code will try to build >>> kudu from> >>> >>> > >> github; but github version kudu tries to build own toolchain while> >>> >>> > >> thirdparty building and it seems that for know current commit of >>> github> >>> >>> > >> kudu has an error on the way to build kudu on Ubuntu 16.04 >>> ppc64le.> >>> >>> > >>> >>> >>> > >> So another question is - Can Kudu be build on Ubuntu 16.04 ppc64le >>> ? If> >>> >>> > >> yes - could you please inform me about appropriate commit of github >>> kudu or> >>> >>> > >> maybe about another way to build it ?> >>> >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > >> Regards,> >>> >>> > >> Koltsov Dmitriy> >>> >>> > >> System Engineer, YADRO> >>> >>> > >>> >>> >>> > >>> >>> >>> > >> On 16.08.2019 00:39, Lars Volker wrote:> >>> >>> > >>> >>> >>> > >> In short the answer is "likely not". We don't have any tests for >>> ppc64le.> >>> >>> > >> About two years ago we had some commits to enable the toolchain on >>> power,> >>> >>> > >> but I can't recall or find any changes to Impala itself related to >>> that> >>> >>> > >> effort.> >>> >>> > >>> >>> >>> > >> Cheers, Lars> >>> >>> > >>> >>> >>> > >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov Dmitriy <[email protected]> >>> <[email protected]>> >>> >>> > >> wrote:> >>> >>> > >>> >>> >>> > >>> Hello!> >>> >>> > >>>> >>> >>> > >>> Could you please inform if Impala can be build and run on Ubuntu >>> 16.04> >>> >>> > >>> ppc64le ?> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>> --> >>> >>> > >>> Regards,> >>> >>> > >>> Koltsov Dmitriy> >>> >>> > >>> System Engineer, YADRO.> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>> >>> >>> > >> >>> >>> > >>> >> >> >
