Could you try "jar tf" on the assembly jar and grep "netlib-native_system-linux-x86_64.so"? -Xiangrui
On Tue, Nov 11, 2014 at 7:11 PM, jpl <jlefe...@soe.ucsc.edu> wrote: > Hi, > I am having trouble using the BLAS libs with the MLLib functions. I am > using org.apache.spark.mllib.clustering.KMeans (on a single machine) and > running the Spark-shell with the kmeans example code (from > https://spark.apache.org/docs/latest/mllib-clustering.html) which runs > successfully but I get the following warning in the log: > > WARN netlib.BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeSystemBLAS > WARN netlib.BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeRefBLAS > > I compiled spark 1.1.0 with mvn -Phadoop-2.4 -Dhadoop.version=2.4.0 > -Pnetlib-lgpl -DskipTests clean package > > If anyone could please clarify the steps to get the dependencies correctly > installed and visible to spark (from > https://spark.apache.org/docs/latest/mllib-guide.html), that would be > greatly appreciated. Using yum, I installed blas.x86_64, lapack.x86_64, > gcc-gfortran.x86_64, libgfortran.x86_64 and then downloaded Breeze and built > that successfully with Maven. I verified that I do have > /usr/lib/libblas.so.3 and /usr/lib/liblapack.so.3 present on the machine and > ldconf -p shows these listed. > > I also tried adding /usr/lib/ to spark.executor.extraLibraryPath and I > verified it is present in the Spark webUI environment tab. I downloaded > and compiled jblas with mvn clean install, which creates > jblas-1.2.4-SNAPSHOT.jar, and then also tried adding that to > spark.executor.extraClassPath but I still get the same WARN message. Maybe > there are a few simple steps that I am missing? Thanks a lot. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/MLLIB-usage-BLAS-dependency-warning-tp18660.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
