Re: Mllib native netlib-java/OpenBLAS

[xhudik]

I'm half-way there
follow
1. compiled and installed open blas library
2. ln -s libopenblas_sandybridgep-r0.2.13.so /usr/lib/libblas.so.3
3. compiled and built spark:
mvn -Pnetlib-lgpl -DskipTests clean compile package

So far so fine. Then I run into problems by testing the solution:
bin/run-example mllib.LinearRegression data/mllib/sample_libsvm_data.txt

/14/12/30 18:39:57 INFO BlockManagerMaster: Registered BlockManager
14/12/30 18:39:58 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/12/30 18:39:58 WARN LoadSnappy: Snappy native library not loaded
Training: 80, test: 20.
*/usr/local/lib/jdk1.8.0//bin/java: symbol lookup error:
/tmp/jniloader1826801168744171087netlib-native_system-linux-x86_64.so:
undefined symbol: cblas_dscal*/


I created a issue report:
https://issues.apache.org/jira/browse/SPARK-5010

any help is deeply appreciated, Tomas




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p20912.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Mllib native netlib-java/OpenBLAS

[Guillaume Pitel]

Hi,

I had the same problem, and tried to compile with mvn -Pnetlib-lgpl

$ mvn -Pnetlib-lgpl -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean 
package


Unfortunately, the resulting assembly jar still lacked the netlib-system class. 
This command :


$ jar tvf assembly/target/scala-2.10/spark-assembly-1.1.1-hadoop2.3.0.jar |grep 
netlib | grep Native


returns nothing...

(and for some reason, including the netlib-all in my shipped jar did not solve 
the problem either, apparently the classloader does not find the class)


In Spark, the profile is defined in mllib submodule, but the -Pnetlib-lgpl seems 
not to be transmitted to the child from the parent pom.xml


I don't know how to fix that cleanly (I just added 
true in mllib's pom.xml), maybe it's just a 
problem with my maven version (3.0.5)


Guillaume


I tried building Spark from the source, by downloading it and running:

mvn -Pnetlib-lgpl -DskipTests clean package



--
eXenSa


*Guillaume PITEL, Président*
+33(0)626 222 431

eXenSa S.A.S. 
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)184 163 677 / Fax +33(0)972 283 705

Re: Mllib native netlib-java/OpenBLAS

[Jaonary Rabarisoa]

+1 with 1.3-SNAPSHOT.

On Mon, Dec 1, 2014 at 5:49 PM, agg212 
wrote:

> Thanks for your reply, but I'm still running into issues
> installing/configuring the native libraries for MLlib.  Here are the steps
> I've taken, please let me know if anything is incorrect.
>
> - Download Spark source
> - unzip and compile using `mvn -Pnetlib-lgpl -DskipTests clean package `
> - Run `sbt/sbt publish-local`
>
> The last step fails with the following error (full stack trace is attached
> here:  error.txt
>  >
> ):
> [error] (sql/compile:compile) java.lang.AssertionError: assertion failed:
> List(object package$DebugNode, object package$DebugNode)
>
> Do I still have to install OPENBLAS/anything else if I build Spark from the
> source using the -Pnetlib-lgpl flag?  Also, do I change the Spark version
> (from 1.1.0 to 1.2.0-SNAPSHOT) in the .sbt file for my app?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p20110.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Mllib native netlib-java/OpenBLAS

[agg212]

Thanks for your reply, but I'm still running into issues
installing/configuring the native libraries for MLlib.  Here are the steps
I've taken, please let me know if anything is incorrect.

- Download Spark source
- unzip and compile using `mvn -Pnetlib-lgpl -DskipTests clean package `
- Run `sbt/sbt publish-local`

The last step fails with the following error (full stack trace is attached
here:  error.txt
 
):
[error] (sql/compile:compile) java.lang.AssertionError: assertion failed:
List(object package$DebugNode, object package$DebugNode)

Do I still have to install OPENBLAS/anything else if I build Spark from the
source using the -Pnetlib-lgpl flag?  Also, do I change the Spark version
(from 1.1.0 to 1.2.0-SNAPSHOT) in the .sbt file for my app?

Thanks!



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p20110.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Mllib native netlib-java/OpenBLAS

[Evan R. Sparks]

You can try recompiling spark with that option, and doing an sbt/sbt
publish-local, then change your spark version from 1.1.0 to 1.2.0-SNAPSHOT
(assuming you're building from the 1.1 branch) - sbt or maven (whichever
you're compiling your app with) will pick up the version of spark that you
just built.

On Mon, Nov 24, 2014 at 6:31 PM, agg212 
wrote:

> I am running it in local.  How can I use the built version (in local mode)
> so
> that I can use the native libraries?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p19705.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Mllib native netlib-java/OpenBLAS

[agg212]

I am running it in local.  How can I use the built version (in local mode) so
that I can use the native libraries?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p19705.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Mllib native netlib-java/OpenBLAS

[Shivaram Venkataraman]

Can you clarify what is the Spark master URL you are using ? Is it 'local'
or is it a cluster ? If it is 'local' then rebuilding Spark wouldn't help
as Spark is getting pulled in from Maven and that'll just pick up the
released artifacts.

Shivaram

On Mon, Nov 24, 2014 at 1:08 PM, agg212 
wrote:

> I tried building Spark from the source, by downloading it and running:
>
> mvn -Pnetlib-lgpl -DskipTests clean package
>
> I then installed OpenBLAS by doing the following:
>
> - Download and unpack .tar from http://www.openblas.net/
> - Run `make`
>
> I then linked /usr/lib/libblas.so.3 to /usr/lib/libopenblas.so (which links
> to /usr/lib/libopenblas_sandybridgep-r0.2.12.so)
>
> I am still getting the following error when running a job after installing
> spark from the source with the -Pnetlib-lgpl flag:
>
> WARN BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemBLAS
> WARN BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeRefBLAS
>
> Any thoughts on what else I need to do to get the native libraries
> recognized by Spark?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p19681.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Mllib native netlib-java/OpenBLAS

[agg212]

I tried building Spark from the source, by downloading it and running:

mvn -Pnetlib-lgpl -DskipTests clean package

I then installed OpenBLAS by doing the following:

- Download and unpack .tar from http://www.openblas.net/
- Run `make`

I then linked /usr/lib/libblas.so.3 to /usr/lib/libopenblas.so (which links
to /usr/lib/libopenblas_sandybridgep-r0.2.12.so)

I am still getting the following error when running a job after installing
spark from the source with the -Pnetlib-lgpl flag:

WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS

Any thoughts on what else I need to do to get the native libraries
recognized by Spark?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p19681.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Mllib native netlib-java/OpenBLAS

[Evan R. Sparks]

Additionally - I strongly recommend using OpenBLAS over the Atlas build
from the default Ubuntu repositories. Alternatively, you can build ATLAS on
the hardware you're actually going to be running the matrix ops on (the
master/workers), but we've seen modest performance gains doing this vs.
OpenBLAS, at least on the bigger EC2 machines (e.g. cc2.8xlarge,
c3.8xlarge).

On Mon, Nov 24, 2014 at 11:26 AM, Xiangrui Meng  wrote:

> Try building Spark with -Pnetlib-lgpl, which includes the JNI library
> in the Spark assembly jar. This is the simplest approach. If you want
> to include it as part of your project, make sure the library is inside
> the assembly jar or you specify it via `--jars` with spark-submit.
> -Xiangrui
>
> On Mon, Nov 24, 2014 at 8:51 AM, agg212 
> wrote:
> > Hi, i'm trying to improve performance for Spark's Mllib, and I am having
> > trouble getting native netlib-java libraries installed/recognized by
> Spark.
> > I am running on a single machine, Ubuntu 14.04 and here is what I've
> tried:
> >
> > sudo apt-get install libgfortran3
> > sudo apt-get install libatlas3-base libopenblas-base (this is how
> > netlib-java's website says to install it)
> >
> > I also double checked and it looks like the libraries are linked
> correctly
> > in /usr/lib (see below):
> > /usr/lib/libblas.so.3 -> /etc/alternatives/libblas.so.3
> > /usr/lib/liblapack.so.3 -> /etc/alternatives/liblapack.so.3
> >
> >
> > The "Dependencies" section on Spark's Mllib website also says to include
> > "com.github.fommil.netlib:all:1.1.2" as a dependency.  I therefore tried
> > adding this to my sbt file like so:
> >
> > libraryDependencies += "com.github.fommil.netlib" % "all" % "1.1.2"
> >
> > After all this, i'm still seeing the following error message.  Does
> anyone
> > have more detailed installation instructions?
> >
> > 14/11/24 16:49:29 WARN BLAS: Failed to load implementation from:
> > com.github.fommil.netlib.NativeSystemBLAS
> > 14/11/24 16:49:29 WARN BLAS: Failed to load implementation from:
> > com.github.fommil.netlib.NativeRefBLAS
> >
> > Thanks!
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> > For additional commands, e-mail: user-h...@spark.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Mllib native netlib-java/OpenBLAS

[Xiangrui Meng]

Try building Spark with -Pnetlib-lgpl, which includes the JNI library
in the Spark assembly jar. This is the simplest approach. If you want
to include it as part of your project, make sure the library is inside
the assembly jar or you specify it via `--jars` with spark-submit.
-Xiangrui

On Mon, Nov 24, 2014 at 8:51 AM, agg212  wrote:
> Hi, i'm trying to improve performance for Spark's Mllib, and I am having
> trouble getting native netlib-java libraries installed/recognized by Spark.
> I am running on a single machine, Ubuntu 14.04 and here is what I've tried:
>
> sudo apt-get install libgfortran3
> sudo apt-get install libatlas3-base libopenblas-base (this is how
> netlib-java's website says to install it)
>
> I also double checked and it looks like the libraries are linked correctly
> in /usr/lib (see below):
> /usr/lib/libblas.so.3 -> /etc/alternatives/libblas.so.3
> /usr/lib/liblapack.so.3 -> /etc/alternatives/liblapack.so.3
>
>
> The "Dependencies" section on Spark's Mllib website also says to include
> "com.github.fommil.netlib:all:1.1.2" as a dependency.  I therefore tried
> adding this to my sbt file like so:
>
> libraryDependencies += "com.github.fommil.netlib" % "all" % "1.1.2"
>
> After all this, i'm still seeing the following error message.  Does anyone
> have more detailed installation instructions?
>
> 14/11/24 16:49:29 WARN BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemBLAS
> 14/11/24 16:49:29 WARN BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeRefBLAS
>
> Thanks!
>
>
>
>
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org