RE: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Thanks for the response. I will try with log4j. That said, I am running in windows using winutil.exe and still getting the warning. Thanks, Assaf. From: Steve Loughran [mailto:ste...@hortonworks.com] Sent: Tuesday, May 16, 2017 6:55 PM To: Mendelson, Assaf Cc: user@spark.apache.org Subject: Re: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable On 10 May 2017, at 13:40, Mendelson, Assaf <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote: Hi all, When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch “WARN” in the log keep flagging this. I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine. Specifically, I can’t find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing). Log4J is your friend. I usually have (at least) log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN if you are working on Windows though, you do actually need the native libraries an winutils.exe on your path, or things won't work Any help would be appreciated. Thanks, Assaf.
Re: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
On 10 May 2017, at 13:40, Mendelson, Assaf <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote: Hi all, When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch “WARN” in the log keep flagging this. I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine. Specifically, I can’t find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing). Log4J is your friend. I usually have (at least) log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN if you are working on Windows though, you do actually need the native libraries an winutils.exe on your path, or things won't work Any help would be appreciated. Thanks, Assaf.
[WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Hi all, When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch "WARN" in the log keep flagging this. I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning, http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine. Specifically, I can't find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing). Any help would be appreciated. Thanks, Assaf.