Hi Dmitry,
> But only by that moment I've noticed that there is a
significant difference between
github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and
github.com/apache/impala
<http://github.com/apache/impala> - while build.
> And all previous steps I tried with github apache
impala and cloudera native-toolchain.
That is the right thing to do. Apache Impala master is
the latest and greatest. github.com/cloudera/impala
<http://github.com/cloudera/impala> mainly hosts
snapshots of the source from different Cloudera
releases. It looks like the default branch was
cdh5-trunk, which is a fairly old version of Impala at
this point.
On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy
<[email protected] <mailto:[email protected]>> wrote:
Hello Tim.
After some tricks I have success in building Kudu
stub. Thanks for advice.
But only by that moment I've noticed that there is a
significant difference between
github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and
github.com/apache/impala
<http://github.com/apache/impala> - while build.
And all previous steps I tried with github apache
impala and cloudera native-toolchain.
So, for now, after I successfully got Kudu stub,
I've tried to build final Impala build with github
apache impala and cloudera native-toolchain in
Ubuntu 16.04 ppc64le. And I have a lot of error
messages while compiling process, e.g.:
...
make[2]: *** Deleting file
'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
In file included from
/home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
from
/home/build/impala/be/src/gutil/spinlock_internal.cc:44:
/home/build/impala/be/src/gutil/atomicops.h:88:2:
error: #error You need to implement atomic
operations for this architecture
#error You need to implement atomic operations
for this architecture
^
/home/build/impala/be/src/gutil/atomicops.h:321:8:
error: âAtomic32â does not name a type
inline Atomic32 Acquire_CompareAndSwap(volatile
Atomic32* ptr,
...
Impala build process failed.
So, hence now I'm trying only to use cloudera github
impala and cloudera native-toolchain. I use
previously build cloudera native-toolchain (git
cloned from github), set IMPALA_TOOLCHAIN var to the
appropriate location; current commit of cloudera
impala is 7dd2c81. But when ./buildall.sh is
executed I have the following error:
-- Added static library dependency pprof:
/home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
-- ----------> Adding thirdparty library gtest.
<----------
-- Header files:
/home/build/impala/toolchain/gtest-1.6.0/include
-- Added static library dependency gtest:
/home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
CMake Error at
cmake_modules/FindLlvmBinaries.cmake:33 (message):
Could not find llvm-config
Call Stack (most recent call first):
CMakeLists.txt:208 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
See also
"/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
Error in
/home/build/impala_cloudera/bin/make_impala.sh at
line 161: cmake . ${CMAKE_ARGS[@]}
Though IMPALA_TOOLCHAIN path points to directory
which contains LLVM build: "llvm-5.0.1-p1" dir is in
the path, with all appropriate files.
_
__*Question:*_ Could you, please, give a comment
about possible reason of the error and about how it
could be resolved ?
btw: there is some inconsistency about llvm versions
between one in ./bin/impala-config.sh (export
IMPALA_LLVM_VERSION=3.9.1) and default one of llvm
build in cloudera native-toolchain (5.0.1-p1).
On 08/24/19 03:49, Tim Armstrong wrote:
Ah I see. I think if you run
bin/bootstrap_toolchain.py manually it will build
the stub for you.
I wouldn't be surprised if this requires some
tweaks to the scripts to make it work since they
weren't written for this exact case in mind (the
original use case for the kudu stubs was for OSes
that Kudu didn't support but Impala did, like
CentOS 5).
On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy
<[email protected] <mailto:[email protected]>>
wrote:
Hello, Tim.
Thanks for reply.
When I'm trying to build Impala I am of course
using already built clouder native-toolchain
(cloned from github repo). This
native-toolchain has no
kudu in final build directory. So, when I'm
trying to build Impala pointing this build
through IMPALA_TOOLCHAIN variable with
SKIP_TOOLCHAIN_BOOSTRAP=true then I have an
error mentioned by me above (see the fragment
below). Could you, please, inform if there is a
possibility to instruct Impala building process
not to use Kudu in Impala build on Ubuntu 16.04
ppc64le ?
> > -- ----------> Adding thirdparty library
libunwind. <---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409
(find_package):>
> > Could not find a package configuration file
provided by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of "kuduClient"
to CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory containing
one of the above files. If>
> > "kuduClient" provides a separate
development package or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
On 2019/08/21 16:47:48, Tim Armstrong
<[email protected]> <mailto:[email protected]>
wrote:
> Hi Dmitry,>
>
> The toolchain bootstrap process downloads
pre-built binaries, and we don't>
> have any for PPC.>
>
> You would need to build the toolchain locally
for PPC, then point the>
> Impala build at it. These instructions might
be helpful for you ->
>
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
> .>
>
>
>
> On Tue, Aug 20, 2019 at 3:53 PM Koltsov
Dmitriy <[email protected]>
<mailto:[email protected]> wrote:>
>
> > I've looked at the contents of
impala-config.sh. As I understand - I>
> > should manually set KUDU_IS_SUPPORTED=false
in order to make general impala>
> > build process to download kudu and build
stub. But I use native_toolchain>
> > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true.
So it seems that there is mutual>
> > exclusion: if I want to build impala with
github cloudera native_toolchain>
> > then code branch for kudu stub building is
disabled>
> > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set>
> > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable
general build process to download>
> > and build kudu stub then there will be no
other components available from>
> > cloudera native_toolchain.>
> >>
> > For example, when I use first case with
SKIP_TOOLCHAIN_BOOTSTRAP=true and>
> > manually KUDU_IS_SUPPORTED=false set in
impala-config.sh, then I have the>
> > following error when trying ./buildall.sh
for cloudera impala:>
> >>
> > ...>
> > -- ----------> Adding thirdparty library
krb5. <---------->
> > -- Header files: /usr/include>
> > -- Added shared library dependency krb5:>
> > /usr/lib/powerpc64le-linux-gnu/libkrb5.so>
> > -- Found Kerberos: /usr/sbin/kadmin.local>
> > -- Added shared library dependency rt:>
> > /usr/lib/powerpc64le-linux-gnu/librt.so>
> > -- Added shared library dependency dl:>
> > /usr/lib/powerpc64le-linux-gnu/libdl.so>
> > -- Found LIBUNWIND:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > -- ----------> Adding thirdparty library
libunwind. <---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409
(find_package):>
> > Could not find a package configuration file
provided by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of "kuduClient"
to CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory containing
one of the above files. If>
> > "kuduClient" provides a separate
development package or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
> >>
> >>
> > Could you please describe how can one
exactly build cloudera impala with>
> > its native_toolchain and kudu stub
simultaneously on Ubuntu 16.04 ppc64le ?>
> >>
> >>
> >>
> >>
> >>
> > Regards,>
> > Koltsov Dmitriy>
> > System Engineer, YADRO>
> >>
> >>
> >>
> > On 08/16/19 18:15, Lars Volker wrote:>
> >>
> > For the Impala toolchain you can start with
this list of commits>
> > <https://gerrit.cloudera.org/#/q/ppc64le>
<https://gerrit.cloudera.org/#/q/ppc64le>, in
particular this review>
> > <https://gerrit.cloudera.org/#/c/9077/>
<https://gerrit.cloudera.org/#/c/9077/>.>
> >>
> > I don't know about kudu, you might want to
ask on their mailing list or>
> > their slack channel. If you look in
impala-config.sh for>
> > the KUDU_IS_SUPPORTED flag, you should find
a way to build Impala without>
> > Kudu.>
> >>
> > Cheers, Lars>
> >>
> >>
> >>
> > On Fri, Aug 16, 2019 at 5:32 AM Koltsov
Dmitriy <[email protected]>
<mailto:[email protected]>>
> > wrote:>
> >>
> >> Hello, Lars!>
> >>>
> >> Thanks for operative answer.>
> >>>
> >> And could please inform about actual
commit number of the>
> >> native-toolchain on github which can be
build successfully at that moment ?>
> >>>
> >> Btw: I ask about it because build process
for current commit of>
> >> native-toolchain finishes with error. And
error is because of kudu build>
> >> fail (other components of toolchain have
been built fine). For now build.sh>
> >> file in this commit of kudu src have
explicit switch to disable ppc64le.>
> >> And if I manually enable ppc64le case then
code will try to build kudu from>
> >> github; but github version kudu tries to
build own toolchain while>
> >> thirdparty building and it seems that for
know current commit of github>
> >> kudu has an error on the way to build kudu
on Ubuntu 16.04 ppc64le.>
> >>>
> >> So another question is - Can Kudu be build
on Ubuntu 16.04 ppc64le ? If>
> >> yes - could you please inform me about
appropriate commit of github kudu or>
> >> maybe about another way to build it ?>
> >>>
> >>>
> >>>
> >> Regards,>
> >> Koltsov Dmitriy>
> >> System Engineer, YADRO>
> >>>
> >>>
> >> On 16.08.2019 00:39, Lars Volker wrote:>
> >>>
> >> In short the answer is "likely not". We
don't have any tests for ppc64le.>
> >> About two years ago we had some commits to
enable the toolchain on power,>
> >> but I can't recall or find any changes to
Impala itself related to that>
> >> effort.>
> >>>
> >> Cheers, Lars>
> >>>
> >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov
Dmitriy <[email protected]>
<mailto:[email protected]>>
> >> wrote:>
> >>>
> >>> Hello!>
> >>>>
> >>> Could you please inform if Impala can be
build and run on Ubuntu 16.04>
> >>> ppc64le ?>
> >>>>
> >>>>
> >>>>
> >>>>
> >>> -->
> >>> Regards,>
> >>> Koltsov Dmitriy>
> >>> System Engineer, YADRO.>
> >>>>
> >>>>
> >>>
> >>
>
