On 10 May 2017, at 13:40, Mendelson, Assaf <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote:
Hi all, When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch “WARN” in the log keep flagging this. I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine. Specifically, I can’t find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing). Log4J is your friend. I usually have (at least) log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN if you are working on Windows though, you do actually need the native libraries an winutils.exe on your path, or things won't work Any help would be appreciated. Thanks, Assaf.
