This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 35db3482af6 [SPARK-40251][MLLIB][DOCS][FOLLOWUP] Update doc for mllib linear algebra acceleration 35db3482af6 is described below commit 35db3482af6a27b303af2b214dd0162b7272bc9d Author: panbingkun <pbk1...@gmail.com> AuthorDate: Sun Sep 4 08:30:20 2022 -0500 [SPARK-40251][MLLIB][DOCS][FOLLOWUP] Update doc for mllib linear algebra acceleration ### What changes were proposed in this pull request? The pr aims to update doc for mllib linear algebra acceleration after https://github.com/apache/spark/pull/37002 & https://github.com/apache/spark/pull/37700 ### Why are the changes needed? > A.Dependence on netlib-java has been completely eliminated after https://github.com/apache/spark/pull/37002 & https://github.com/apache/spark/pull/37700. > B.Some interfaces of dev.ludovic.netlib.blas have changed. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes #37791 from panbingkun/update_doc_for_mllib. Authored-by: panbingkun <pbk1...@gmail.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- docs/ml-guide.md | 4 ++-- docs/ml-linalg-guide.md | 10 ++++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/ml-guide.md b/docs/ml-guide.md index 3202647240b..572f61ef973 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -62,11 +62,11 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin # Dependencies -MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/), [dev.ludovic.netlib](https://github.com/luhenry/netlib), and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. +MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [dev.ludovic.netlib](https://github.com/luhenry/netlib) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. However, native acceleration libraries can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.html) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message like below and a pure JVM implementation will be used instead: ``` -WARN BLAS: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS +WARNING: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS ``` To use MLlib in Python, you will need [NumPy](http://www.numpy.org) version 1.4 or newer. diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index 1e3d9ccbc82..74dcabf987e 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -58,15 +58,17 @@ sudo yum install openblas To verify native libraries are properly loaded, start `spark-shell` and run the following code: ``` -scala> import dev.ludovic.netlib.NativeBLAS +scala> import dev.ludovic.netlib.blas.NativeBLAS scala> NativeBLAS.getInstance() ``` -If they are correctly loaded, it should print `dev.ludovic.netlib.NativeBLAS = dev.ludovic.netlib.blas.JNIBLAS@...`. Otherwise the warnings should be printed: +If they are correctly loaded, it should print `dev.ludovic.netlib.blas.NativeBLAS = dev.ludovic.netlib.blas.JNIBLAS@...`. Otherwise the warnings should be printed: ``` -WARN NativeBLAS: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS +WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS +... java.lang.RuntimeException: Unable to load native implementation - at dev.ludovic.netlib.NativeBLAS.getInstance(NativeBLAS.java:44) + at dev.ludovic.netlib.blas.InstanceBuilder.nativeBlas(InstanceBuilder.java:59) + at dev.ludovic.netlib.blas.NativeBLAS.getInstance(NativeBLAS.java:31) ... ``` --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org