Hi Sean, Thanks for the reply! I did double-check that the jar is one I think I am running:
[image: Inline image 2] jar tf /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep netlib | grep Native com/github/fommil/netlib/NativeRefARPACK.class com/github/fommil/netlib/NativeRefBLAS.class com/github/fommil/netlib/NativeRefLAPACK.class com/github/fommil/netlib/NativeSystemARPACK.class com/github/fommil/netlib/NativeSystemBLAS.class com/github/fommil/netlib/NativeSystemLAPACK.class Also, I checked the gfortran version on the cluster nodes and it is available and is 5.1 $ gfortran --version GNU Fortran (GCC) 5.1.0 Copyright (C) 2015 Free Software Foundation, Inc. and still see: 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK Does anything need to be adjusted in my application POM? Thanks, Arun On Thu, Jul 16, 2015 at 5:26 PM, Sean Owen <so...@cloudera.com> wrote: > Yes, that's most of the work, just getting the native libs into the > assembly. netlib can find them from there even if you don't have BLAS > libs on your OS, since it includes a reference implementation as a > fallback. > > One common reason it won't load is not having libgfortran installed on > your OSes though. It has to be 4.6+ too. That can't be shipped even in > netlib and has to exist on your hosts. > > The other thing I'd double-check is whether you are really using this > assembly you built for your job -- like, it's the actually the > assembly the executors are using. > > > On Tue, Jul 7, 2015 at 8:47 PM, Arun Ahuja <aahuj...@gmail.com> wrote: > > Is there more documentation on what is needed to setup BLAS/LAPACK native > > suport with Spark. > > > > I’ve built spark with the -Pnetlib-lgpl flag and see that the netlib > classes > > are in the assembly jar. > > > > jar tvf spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep netlib | > grep > > Native > > 6625 Tue Jul 07 15:22:08 EDT 2015 > > com/github/fommil/netlib/NativeRefARPACK.class > > 21123 Tue Jul 07 15:22:08 EDT 2015 > > com/github/fommil/netlib/NativeRefBLAS.class > > 178334 Tue Jul 07 15:22:08 EDT 2015 > > com/github/fommil/netlib/NativeRefLAPACK.class > > 6640 Tue Jul 07 15:22:10 EDT 2015 > > com/github/fommil/netlib/NativeSystemARPACK.class > > 21138 Tue Jul 07 15:22:10 EDT 2015 > > com/github/fommil/netlib/NativeSystemBLAS.class > > 178349 Tue Jul 07 15:22:10 EDT 2015 > > com/github/fommil/netlib/NativeSystemLAPACK.class > > > > Also I see the following in /usr/lib64 > > > >> ls /usr/lib64/libblas. > > libblas.a libblas.so libblas.so.3 libblas.so.3.2 > > libblas.so.3.2.1 > > > >> ls /usr/lib64/liblapack > > liblapack.a liblapack_pic.a liblapack.so > liblapack.so.3 > > liblapack.so.3.2 liblapack.so.3.2.1 > > > > But I stil see the following in the Spark logs: > > > > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: > > com.github.fommil.netlib.NativeSystemBLAS > > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: > > com.github.fommil.netlib.NativeRefBLAS > > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: > > com.github.fommil.netlib.NativeSystemLAPACK > > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: > > com.github.fommil.netlib.NativeRefLAPACK > > > > Anything in this process I missed? > > > > Thanks, > > Arun >
