Can you try setting the spark.yarn.jar property to make sure it points to the jar you're thinking of?
-Sandy On Fri, Jul 17, 2015 at 11:32 AM, Arun Ahuja <aahuj...@gmail.com> wrote: > Yes, it's a YARN cluster and using spark-submit to run. I have SPARK_HOME > set to the directory above and using the spark-submit script from there. > > bin/spark-submit --master yarn-client --executor-memory 10g --driver-memory > 8g --num-executors 400 --executor-cores 1 --class > org.hammerlab.guacamole.Guacamole --conf spark.default.parallelism=4000 > --conf spark.storage.memoryFraction=0.15 > > ​ > > libgfortran.so.3 is also there > > ls /usr/lib64/libgfortran.so.3 > /usr/lib64/libgfortran.so.3 > > These are jniloader files in the jar > > jar tf > /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar > | grep jniloader > META-INF/maven/com.github.fommil/jniloader/ > META-INF/maven/com.github.fommil/jniloader/pom.xml > META-INF/maven/com.github.fommil/jniloader/pom.properties > > ​ > > Thanks, > Arun > > On Fri, Jul 17, 2015 at 1:30 PM, Sean Owen <so...@cloudera.com> wrote: > >> Make sure /usr/lib64 contains libgfortran.so.3; that's really the issue. >> >> I'm pretty sure the answer is 'yes', but, make sure the assembly has >> jniloader too. I don't see why it wouldn't, but, that's needed. >> >> What is your env like -- local, standalone, YARN? how are you running? >> Just want to make sure you are using this assembly across your cluster. >> >> On Fri, Jul 17, 2015 at 6:26 PM, Arun Ahuja <aahuj...@gmail.com> wrote: >> >>> Hi Sean, >>> >>> Thanks for the reply! I did double-check that the jar is one I think I >>> am running: >>> >>> [image: Inline image 2] >>> >>> jar tf >>> /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>> | grep netlib | grep Native >>> com/github/fommil/netlib/NativeRefARPACK.class >>> com/github/fommil/netlib/NativeRefBLAS.class >>> com/github/fommil/netlib/NativeRefLAPACK.class >>> com/github/fommil/netlib/NativeSystemARPACK.class >>> com/github/fommil/netlib/NativeSystemBLAS.class >>> com/github/fommil/netlib/NativeSystemLAPACK.class >>> >>> Also, I checked the gfortran version on the cluster nodes and it is >>> available and is 5.1 >>> >>> $ gfortran --version >>> GNU Fortran (GCC) 5.1.0 >>> Copyright (C) 2015 Free Software Foundation, Inc. >>> >>> and still see: >>> >>> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemBLAS >>> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefBLAS >>> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemLAPACK >>> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefLAPACK >>> >>> ​ >>> >>> Does anything need to be adjusted in my application POM? >>> >>> Thanks, >>> Arun >>> >>> On Thu, Jul 16, 2015 at 5:26 PM, Sean Owen <so...@cloudera.com> wrote: >>> >>>> Yes, that's most of the work, just getting the native libs into the >>>> assembly. netlib can find them from there even if you don't have BLAS >>>> libs on your OS, since it includes a reference implementation as a >>>> fallback. >>>> >>>> One common reason it won't load is not having libgfortran installed on >>>> your OSes though. It has to be 4.6+ too. That can't be shipped even in >>>> netlib and has to exist on your hosts. >>>> >>>> The other thing I'd double-check is whether you are really using this >>>> assembly you built for your job -- like, it's the actually the >>>> assembly the executors are using. >>>> >>>> >>>> On Tue, Jul 7, 2015 at 8:47 PM, Arun Ahuja <aahuj...@gmail.com> wrote: >>>> > Is there more documentation on what is needed to setup BLAS/LAPACK >>>> native >>>> > suport with Spark. >>>> > >>>> > I’ve built spark with the -Pnetlib-lgpl flag and see that the netlib >>>> classes >>>> > are in the assembly jar. >>>> > >>>> > jar tvf spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep netlib >>>> | grep >>>> > Native >>>> > 6625 Tue Jul 07 15:22:08 EDT 2015 >>>> > com/github/fommil/netlib/NativeRefARPACK.class >>>> > 21123 Tue Jul 07 15:22:08 EDT 2015 >>>> > com/github/fommil/netlib/NativeRefBLAS.class >>>> > 178334 Tue Jul 07 15:22:08 EDT 2015 >>>> > com/github/fommil/netlib/NativeRefLAPACK.class >>>> > 6640 Tue Jul 07 15:22:10 EDT 2015 >>>> > com/github/fommil/netlib/NativeSystemARPACK.class >>>> > 21138 Tue Jul 07 15:22:10 EDT 2015 >>>> > com/github/fommil/netlib/NativeSystemBLAS.class >>>> > 178349 Tue Jul 07 15:22:10 EDT 2015 >>>> > com/github/fommil/netlib/NativeSystemLAPACK.class >>>> > >>>> > Also I see the following in /usr/lib64 >>>> > >>>> >> ls /usr/lib64/libblas. >>>> > libblas.a libblas.so libblas.so.3 libblas.so.3.2 >>>> > libblas.so.3.2.1 >>>> > >>>> >> ls /usr/lib64/liblapack >>>> > liblapack.a liblapack_pic.a liblapack.so >>>> liblapack.so.3 >>>> > liblapack.so.3.2 liblapack.so.3.2.1 >>>> > >>>> > But I stil see the following in the Spark logs: >>>> > >>>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: >>>> > com.github.fommil.netlib.NativeSystemBLAS >>>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: >>>> > com.github.fommil.netlib.NativeRefBLAS >>>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: >>>> > com.github.fommil.netlib.NativeSystemLAPACK >>>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: >>>> > com.github.fommil.netlib.NativeRefLAPACK >>>> > >>>> > Anything in this process I missed? >>>> > >>>> > Thanks, >>>> > Arun >>>> >>> >>> >> >
