[
https://issues.apache.org/jira/browse/IMPALA-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106778#comment-18106778
]
Joe McDonnell commented on IMPALA-12975:
----------------------------------------
Toolchain commit (not strictly needed for the new purpose):
{noformat}
commit 5197f32adfea2be37c6935aebf5491504663af6b
Author: Joe McDonnell <[email protected]>
Date: Fri Jun 19 13:40:43 2026 -0700
IMPALA-12975: Match upstream structure for hadoop-client tarball
The typical Hadoop tarball structure has an include directory
with libhdfs libraries in lib/native. Currently, our hadoop-client
build doesn't not have the include directory and puts libraries
in lib (no /native). To make it easier to use them in a similar
way, this switches the structure to match the typical Hadoop
directory structures.
This uses an empty patch to bump the patch level to avoid version
ambiguity.
Change-Id: Ibc9faf88ec5195a21767234783e78dca289c4234
Reviewed-on: http://gerrit.cloudera.org:8080/24498
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
{noformat}
> Rework organization of Hadoop dependency on ARM builds
> ------------------------------------------------------
>
> Key: IMPALA-12975
> URL: https://issues.apache.org/jira/browse/IMPALA-12975
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The hadoop binaries that we download from the CDP build number are built for
> x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP
> hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces
> (including the C++ build) use these environment variables to find the native
> libraries.
> On ARM, we leave those environment variables pointed to that same location.
> We fix things up by downloading a separate hadoop-client built for ARM, then
> copying the contents into the usual location in the CDP hadoop directory,
> overwriting the x86_64 contents. The code to overwrite the libraries runs on
> each invocation of buildall.sh
> On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded
> hadoop-client (which is built for ARM). With a bit of work on the
> hadoop-client, we could get it to also have the header files and also point
> HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during
> buildall.sh. Any build that wants to pass in a custom hadoop can then use
> HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
