Make sure /usr/lib64 contains libgfortran.so.3; that's really the issue.

I'm pretty sure the answer is 'yes', but, make sure the assembly has
jniloader too. I don't see why it wouldn't, but, that's needed.

What is your env like -- local, standalone, YARN? how are you running?
Just want to make sure you are using this assembly across your cluster.

On Fri, Jul 17, 2015 at 6:26 PM, Arun Ahuja <aahuj...@gmail.com> wrote:

> Hi Sean,
>
> Thanks for the reply! I did double-check that the jar is one I think I am
> running:
>
> [image: Inline image 2]
>
> jar tf 
> /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar
>  | grep netlib | grep Native
> com/github/fommil/netlib/NativeRefARPACK.class
> com/github/fommil/netlib/NativeRefBLAS.class
> com/github/fommil/netlib/NativeRefLAPACK.class
> com/github/fommil/netlib/NativeSystemARPACK.class
> com/github/fommil/netlib/NativeSystemBLAS.class
> com/github/fommil/netlib/NativeSystemLAPACK.class
>
> Also, I checked the gfortran version on the cluster nodes and it is
> available and is 5.1
>
> $ gfortran --version
> GNU Fortran (GCC) 5.1.0
> Copyright (C) 2015 Free Software Foundation, Inc.
>
> and still see:
>
> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: 
> com.github.fommil.netlib.NativeSystemBLAS
> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: 
> com.github.fommil.netlib.NativeRefBLAS
> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: 
> com.github.fommil.netlib.NativeSystemLAPACK
> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: 
> com.github.fommil.netlib.NativeRefLAPACK
>
> ​
>
> Does anything need to be adjusted in my application POM?
>
> Thanks,
> Arun
>
> On Thu, Jul 16, 2015 at 5:26 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Yes, that's most of the work, just getting the native libs into the
>> assembly. netlib can find them from there even if you don't have BLAS
>> libs on your OS, since it includes a reference implementation as a
>> fallback.
>>
>> One common reason it won't load is not having libgfortran installed on
>> your OSes though. It has to be 4.6+ too. That can't be shipped even in
>> netlib and has to exist on your hosts.
>>
>> The other thing I'd double-check is whether you are really using this
>> assembly you built for your job -- like, it's the actually the
>> assembly the executors are using.
>>
>>
>> On Tue, Jul 7, 2015 at 8:47 PM, Arun Ahuja <aahuj...@gmail.com> wrote:
>> > Is there more documentation on what is needed to setup BLAS/LAPACK
>> native
>> > suport with Spark.
>> >
>> > I’ve built spark with the -Pnetlib-lgpl flag and see that the netlib
>> classes
>> > are in the assembly jar.
>> >
>> > jar tvf spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar  | grep netlib |
>> grep
>> > Native
>> >   6625 Tue Jul 07 15:22:08 EDT 2015
>> > com/github/fommil/netlib/NativeRefARPACK.class
>> >  21123 Tue Jul 07 15:22:08 EDT 2015
>> > com/github/fommil/netlib/NativeRefBLAS.class
>> > 178334 Tue Jul 07 15:22:08 EDT 2015
>> > com/github/fommil/netlib/NativeRefLAPACK.class
>> >   6640 Tue Jul 07 15:22:10 EDT 2015
>> > com/github/fommil/netlib/NativeSystemARPACK.class
>> >  21138 Tue Jul 07 15:22:10 EDT 2015
>> > com/github/fommil/netlib/NativeSystemBLAS.class
>> > 178349 Tue Jul 07 15:22:10 EDT 2015
>> > com/github/fommil/netlib/NativeSystemLAPACK.class
>> >
>> > Also I see the following in /usr/lib64
>> >
>> >> ls /usr/lib64/libblas.
>> > libblas.a         libblas.so        libblas.so.3      libblas.so.3.2
>> > libblas.so.3.2.1
>> >
>> >> ls /usr/lib64/liblapack
>> > liblapack.a         liblapack_pic.a     liblapack.so
>> liblapack.so.3
>> > liblapack.so.3.2    liblapack.so.3.2.1
>> >
>> > But I stil see the following in the Spark logs:
>> >
>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from:
>> > com.github.fommil.netlib.NativeSystemBLAS
>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from:
>> > com.github.fommil.netlib.NativeRefBLAS
>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from:
>> > com.github.fommil.netlib.NativeSystemLAPACK
>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from:
>> > com.github.fommil.netlib.NativeRefLAPACK
>> >
>> > Anything in this process I missed?
>> >
>> > Thanks,
>> > Arun
>>
>
>

Reply via email to