Hi Dmitry,
> But only by that moment I've noticed that there
is a significant difference between
github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and
github.com/apache/impala
<http://github.com/apache/impala> - while build.
> And all previous steps I tried with github apache
impala and cloudera native-toolchain.
That is the right thing to do. Apache Impala master
is the latest and greatest.
github.com/cloudera/impala
<http://github.com/cloudera/impala> mainly hosts
snapshots of the source from different Cloudera
releases. It looks like the default branch was
cdh5-trunk, which is a fairly old version of Impala
at this point.
On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy
<[email protected] <mailto:[email protected]>>
wrote:
Hello Tim.
After some tricks I have success in building
Kudu stub. Thanks for advice.
But only by that moment I've noticed that there
is a significant difference between
github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and
github.com/apache/impala
<http://github.com/apache/impala> - while build.
And all previous steps I tried with github
apache impala and cloudera native-toolchain.
So, for now, after I successfully got Kudu
stub, I've tried to build final Impala build
with github apache impala and cloudera
native-toolchain in Ubuntu 16.04 ppc64le. And I
have a lot of error messages while compiling
process, e.g.:
...
make[2]: *** Deleting file
'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
In file included from
/home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
from
/home/build/impala/be/src/gutil/spinlock_internal.cc:44:
/home/build/impala/be/src/gutil/atomicops.h:88:2:
error: #error You need to implement atomic
operations for this architecture
#error You need to implement atomic
operations for this architecture
^
/home/build/impala/be/src/gutil/atomicops.h:321:8:
error: âAtomic32â does not name a type
inline Atomic32
Acquire_CompareAndSwap(volatile Atomic32* ptr,
...
Impala build process failed.
So, hence now I'm trying only to use cloudera
github impala and cloudera native-toolchain. I
use previously build cloudera native-toolchain
(git cloned from github), set IMPALA_TOOLCHAIN
var to the appropriate location; current commit
of cloudera impala is 7dd2c81. But when
./buildall.sh is executed I have the following
error:
-- Added static library dependency pprof:
/home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
-- ----------> Adding thirdparty library gtest.
<----------
-- Header files:
/home/build/impala/toolchain/gtest-1.6.0/include
-- Added static library dependency gtest:
/home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
CMake Error at
cmake_modules/FindLlvmBinaries.cmake:33 (message):
Could not find llvm-config
Call Stack (most recent call first):
CMakeLists.txt:208 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
See also
"/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
Error in
/home/build/impala_cloudera/bin/make_impala.sh
at line 161: cmake . ${CMAKE_ARGS[@]}
Though IMPALA_TOOLCHAIN path points to
directory which contains LLVM build:
"llvm-5.0.1-p1" dir is in the path, with all
appropriate files.
_
__*Question:*_ Could you, please, give a
comment about possible reason of the error and
about how it could be resolved ?
btw: there is some inconsistency about llvm
versions between one in ./bin/impala-config.sh
(export IMPALA_LLVM_VERSION=3.9.1) and default
one of llvm build in cloudera native-toolchain
(5.0.1-p1).
On 08/24/19 03:49, Tim Armstrong wrote:
Ah I see. I think if you run
bin/bootstrap_toolchain.py manually it will
build the stub for you.
I wouldn't be surprised if this requires some
tweaks to the scripts to make it work since
they weren't written for this exact case in
mind (the original use case for the kudu stubs
was for OSes that Kudu didn't support but
Impala did, like CentOS 5).
On Wed, Aug 21, 2019 at 11:12 AM Koltsov
Dmitriy <[email protected]
<mailto:[email protected]>> wrote:
Hello, Tim.
Thanks for reply.
When I'm trying to build Impala I am of
course using already built clouder
native-toolchain (cloned from github
repo). This native-toolchain has no
kudu in final build directory. So, when
I'm trying to build Impala pointing this
build through IMPALA_TOOLCHAIN variable
with SKIP_TOOLCHAIN_BOOSTRAP=true then I
have an error mentioned by me above (see
the fragment below). Could you, please,
inform if there is a possibility to
instruct Impala building process not to
use Kudu in Impala build on Ubuntu 16.04
ppc64le ?
> > -- ----------> Adding thirdparty
library libunwind. <---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency
libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409
(find_package):>
> > Could not find a package configuration
file provided by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of
"kuduClient" to CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory
containing one of the above files. If>
> > "kuduClient" provides a separate
development package or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors
occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
On 2019/08/21 16:47:48, Tim Armstrong
<[email protected]>
<mailto:[email protected]> wrote:
> Hi Dmitry,>
>
> The toolchain bootstrap process
downloads pre-built binaries, and we don't>
> have any for PPC.>
>
> You would need to build the toolchain
locally for PPC, then point the>
> Impala build at it. These instructions
might be helpful for you ->
>
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
> .>
>
>
>
> On Tue, Aug 20, 2019 at 3:53 PM Koltsov
Dmitriy <[email protected]>
<mailto:[email protected]> wrote:>
>
> > I've looked at the contents of
impala-config.sh. As I understand - I>
> > should manually set
KUDU_IS_SUPPORTED=false in order to make
general impala>
> > build process to download kudu and
build stub. But I use native_toolchain>
> > and so var
SKIP_TOOLCHAIN_BOOTSTRAP=true. So it seems
that there is mutual>
> > exclusion: if I want to build impala
with github cloudera native_toolchain>
> > then code branch for kudu stub
building is disabled>
> > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and
if I set>
> > SKIP_TOOLCHAIN_BOOTSTRAP=false to
enable general build process to download>
> > and build kudu stub then there will be
no other components available from>
> > cloudera native_toolchain.>
> >>
> > For example, when I use first case
with SKIP_TOOLCHAIN_BOOTSTRAP=true and>
> > manually KUDU_IS_SUPPORTED=false set
in impala-config.sh, then I have the>
> > following error when trying
./buildall.sh for cloudera impala:>
> >>
> > ...>
> > -- ----------> Adding thirdparty
library krb5. <---------->
> > -- Header files: /usr/include>
> > -- Added shared library dependency krb5:>
> >
/usr/lib/powerpc64le-linux-gnu/libkrb5.so>
> > -- Found Kerberos:
/usr/sbin/kadmin.local>
> > -- Added shared library dependency rt:>
> > /usr/lib/powerpc64le-linux-gnu/librt.so>
> > -- Added shared library dependency dl:>
> > /usr/lib/powerpc64le-linux-gnu/libdl.so>
> > -- Found LIBUNWIND:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > -- ----------> Adding thirdparty
library libunwind. <---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency
libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409
(find_package):>
> > Could not find a package configuration
file provided by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of
"kuduClient" to CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory
containing one of the above files. If>
> > "kuduClient" provides a separate
development package or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors
occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
> >>
> >>
> > Could you please describe how can one
exactly build cloudera impala with>
> > its native_toolchain and kudu stub
simultaneously on Ubuntu 16.04 ppc64le ?>
> >>
> >>
> >>
> >>
> >>
> > Regards,>
> > Koltsov Dmitriy>
> > System Engineer, YADRO>
> >>
> >>
> >>
> > On 08/16/19 18:15, Lars Volker wrote:>
> >>
> > For the Impala toolchain you can start
with this list of commits>
> >
<https://gerrit.cloudera.org/#/q/ppc64le>
<https://gerrit.cloudera.org/#/q/ppc64le>,
in particular this review>
> >
<https://gerrit.cloudera.org/#/c/9077/>
<https://gerrit.cloudera.org/#/c/9077/>.>
> >>
> > I don't know about kudu, you might
want to ask on their mailing list or>
> > their slack channel. If you look in
impala-config.sh for>
> > the KUDU_IS_SUPPORTED flag, you should
find a way to build Impala without>
> > Kudu.>
> >>
> > Cheers, Lars>
> >>
> >>
> >>
> > On Fri, Aug 16, 2019 at 5:32 AM
Koltsov Dmitriy <[email protected]>
<mailto:[email protected]>>
> > wrote:>
> >>
> >> Hello, Lars!>
> >>>
> >> Thanks for operative answer.>
> >>>
> >> And could please inform about actual
commit number of the>
> >> native-toolchain on github which can
be build successfully at that moment ?>
> >>>
> >> Btw: I ask about it because build
process for current commit of>
> >> native-toolchain finishes with error.
And error is because of kudu build>
> >> fail (other components of toolchain
have been built fine). For now build.sh>
> >> file in this commit of kudu src have
explicit switch to disable ppc64le.>
> >> And if I manually enable ppc64le case
then code will try to build kudu from>
> >> github; but github version kudu tries
to build own toolchain while>
> >> thirdparty building and it seems that
for know current commit of github>
> >> kudu has an error on the way to build
kudu on Ubuntu 16.04 ppc64le.>
> >>>
> >> So another question is - Can Kudu be
build on Ubuntu 16.04 ppc64le ? If>
> >> yes - could you please inform me
about appropriate commit of github kudu or>
> >> maybe about another way to build it ?>
> >>>
> >>>
> >>>
> >> Regards,>
> >> Koltsov Dmitriy>
> >> System Engineer, YADRO>
> >>>
> >>>
> >> On 16.08.2019 00:39, Lars Volker wrote:>
> >>>
> >> In short the answer is "likely not".
We don't have any tests for ppc64le.>
> >> About two years ago we had some
commits to enable the toolchain on power,>
> >> but I can't recall or find any
changes to Impala itself related to that>
> >> effort.>
> >>>
> >> Cheers, Lars>
> >>>
> >> On Thu, Aug 15, 2019 at 11:53 AM
Koltsov Dmitriy <[email protected]>
<mailto:[email protected]>>
> >> wrote:>
> >>>
> >>> Hello!>
> >>>>
> >>> Could you please inform if Impala
can be build and run on Ubuntu 16.04>
> >>> ppc64le ?>
> >>>>
> >>>>
> >>>>
> >>>>
> >>> -->
> >>> Regards,>
> >>> Koltsov Dmitriy>
> >>> System Engineer, YADRO.>
> >>>>
> >>>>
> >>>
> >>
>
