[
https://issues.apache.org/jira/browse/HADOOP-19001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799641#comment-17799641
]
ASF GitHub Bot commented on HADOOP-19001:
-----------------------------------------
zhuzilong2013 commented on PR #6367:
URL: https://github.com/apache/hadoop/pull/6367#issuecomment-1867138139
@iwasakims Thank you for your reply！
Can I consider the 'export LD_LIBRARY_PATH' in function
'hadoop_finalize_libpaths' to be redundant?
> LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
> -------------------------------------------------------
>
> Key: HADOOP-19001
> URL: https://issues.apache.org/jira/browse/HADOOP-19001
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.2.4
> Reporter: Zilong Zhu
> Priority: Major
> Labels: pull-request-available
>
> When we run a spark job, we find that it cannot load the native library
> successfully.
> We found a difference between hadoop2 and hadoop3.
> hadoop2-Spark-System Properties：
> |java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> hadoop3-Spark-System Properties：
> |java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> The key point is：
> hadoop2-hadoop-config.sh：
> HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" <--267
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH <--268
>
> hadoop3-hadoop-functions.sh:
> hadoop_add_param HADOOP_OPTS java.library.path \
> "-Djava.library.path=${JAVA_LIBRARY_PATH}"
> export LD_LIBRARY_PATH <--1484
>
> At the same time, the hadoop3 will clear all non-whitelisted environment
> variables.
> I'm not sure if it was intentional. But it makes our spark job unable to find
> the native library on hadoop3.
> Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to
> the default configuration item yarn.nodemanager.env-whitelist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
