Hi
I want to make sure we use short-circuit local reads for performance. I
have set the LD_LIBRARY_PATH correctly, confirmed that the native libraries
match our platform (i.e. are 64 bit and are loaded successfully). When I
start spark, i get the following message after increasing the logging level
for the relevant classes.
*15/11/18 17:47:23 DEBUG NativeCodeLoader: Failed to load native-hadoop
with error: java.lang.UnsatisfiedLinkError: Native Library
/usr/hdp/2.3.2.0-2950/hadoop/lib/native/libhadoop.so.1.0.0 already loaded
in another classloader*
Any idea what might be causing it and how to resolve this.
Regards
Deenar
[spark@edgenode1 spark-1.5.2-bin-hadoop2.6]$ bin/spark-shell
15/11/18 17:46:56 DEBUG NativeCodeLoader: Trying to load the custom-built
native-hadoop library...
15/11/18 17:46:56 DEBUG NativeCodeLoader: Loaded the native-hadoop library
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.2
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
15/11/18 17:47:00 WARN Utils: Service 'SparkUI' could not bind on port
4040. Attempting port 4041.
15/11/18 17:47:00 WARN Utils: Service 'SparkUI' could not bind on port
4041. Attempting port 4042.
15/11/18 17:47:00 WARN MetricsSystem: Using default name DAGScheduler for
source because spark.app.id is not set.
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
Spark context available as sc.
15/11/18 17:47:23 DEBUG NativeCodeLoader: Trying to load the custom-built
native-hadoop library...
*15/11/18 17:47:23 DEBUG NativeCodeLoader: Failed to load native-hadoop
with error: java.lang.UnsatisfiedLinkError: Native Library
/usr/hdp/2.3.2.0-2950/hadoop/lib/native/libhadoop.so.1.0.0 already loaded
in another classloader*
15/11/18 17:47:23 DEBUG NativeCodeLoader:
java.library.path=/usr/hdp/2.3.2.0-2950/hadoop/lib/native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
15/11/18 17:47:23 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/11/18 17:47:24 WARN DomainSocketFactory: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
SQL context available as sqlContext.
*Think Reactive Ltd*
deenar.toras...@thinkreactive.co.uk
07714140812
