[
https://issues.apache.org/jira/browse/IMPALA-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106777#comment-18106777
]
Joe McDonnell commented on IMPALA-12975:
----------------------------------------
For the purposes of building Impala, this works fine. However, HDFS, Hive, and
the frontend tests also need to be able to find the native library. Those work
easily when the binaries are in the usual location in HADOOP_HOME, but it gets
tedious to hook them up to look at a different location.
I'm going to redefine the purpose of this JIRA. The real problem with the
existing buildall.sh logic is that running it can cause minicluster components
to die. This is because the current logic overwrites the files unconditionally.
So, when Hive is trying to use the native library, there is a period of time
when the library has been overwritten and isn't complete. The purpose of this
Jira is to eliminate that issue. Someone with a running minicluster should be
able to run buildall.sh and still have a working minicluster after. This is
important for the perf-AB-test job, which currently doesn't work on ARM.
> Rework organization of Hadoop dependency on ARM builds
> ------------------------------------------------------
>
> Key: IMPALA-12975
> URL: https://issues.apache.org/jira/browse/IMPALA-12975
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The hadoop binaries that we download from the CDP build number are built for
> x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP
> hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces
> (including the C++ build) use these environment variables to find the native
> libraries.
> On ARM, we leave those environment variables pointed to that same location.
> We fix things up by downloading a separate hadoop-client built for ARM, then
> copying the contents into the usual location in the CDP hadoop directory,
> overwriting the x86_64 contents. The code to overwrite the libraries runs on
> each invocation of buildall.sh
> On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded
> hadoop-client (which is built for ARM). With a bit of work on the
> hadoop-client, we could get it to also have the header files and also point
> HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during
> buildall.sh. Any build that wants to pass in a custom hadoop can then use
> HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
