[ https://issues.apache.org/jira/browse/HADOOP-19001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797917#comment-17797917 ]
ASF GitHub Bot commented on HADOOP-19001: ----------------------------------------- iwasakims commented on PR #6367: URL: https://github.com/apache/hadoop/pull/6367#issuecomment-1859160072 @zhuzilong2013 I think you can use [hadoop_add_ldlibpath](https://github.com/apache/hadoop/blob/rel/release-3.2.4/hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example.sh#L81-L82) in your [shell profile](https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/UnixShellGuide.html#Shell_Profiles). > LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR > ------------------------------------------------------- > > Key: HADOOP-19001 > URL: https://issues.apache.org/jira/browse/HADOOP-19001 > Project: Hadoop Common > Issue Type: Bug > Affects Versions: 3.2.4 > Reporter: Zilong Zhu > Priority: Major > > When we run a spark job, we find that it cannot load the native library > successfully. > We found a difference between hadoop2 and hadoop3. > hadoop2-Spark-System Properties: > |java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib| > hadoop3-Spark-System Properties: > |java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib| > The key point is: > hadoop2-hadoop-config.sh: > HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" <--267 > export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH <--268 > > hadoop3-hadoop-functions.sh: > hadoop_add_param HADOOP_OPTS java.library.path \ > "-Djava.library.path=${JAVA_LIBRARY_PATH}" > export LD_LIBRARY_PATH <--1484 > > At the same time, the hadoop3 will clear all non-whitelisted environment > variables. > I'm not sure if it was intentional. But it makes our spark job unable to find > the native library on hadoop3. > Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to > the default configuration item yarn.nodemanager.env-whitelist. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org