OK, after various testing, I found the native library can be loaded if
running in yarn-cluster mode. But I still cannot find out why it won't load
when running in yarn-client mode...
Thanks,
David
On Thu, Mar 26, 2015 at 4:21 PM Xi Shen davidshe...@gmail.com wrote:
Not of course...all
Not of course...all machines in HDInsight are Windows 64bit server. And I
have made sure all my DLLs are for 64bit machines. I have managed to get
those DLLs loade on my local machine which is also Windows 64bit.
[image: --]
Xi Shen
[image: http://]about.me/davidshen
You probably need to add the dll directory to the path (not classpath!)
environment variable on all nodes.
Le 26 mars 2015 06:23, Xi Shen davidshe...@gmail.com a écrit :
Not of course...all machines in HDInsight are Windows 64bit server. And I
have made sure all my DLLs are for 64bit machines.
Are you deploying the windows dll to linux machine?
Sincerely,
DB Tsai
---
Blog: https://www.dbtsai.com
On Wed, Mar 25, 2015 at 3:57 AM, Xi Shen davidshe...@gmail.com wrote:
I think you meant to use the --files to deploy the DLLs. I gave a
I think you meant to use the --files to deploy the DLLs. I gave a try,
but it did not work.
From the Spark UI, Environment tab, I can see
spark.yarn.dist.files
Hi,
I am doing ML using Spark mllib. However, I do not have full control to the
cluster. I am using Microsoft Azure HDInsight
I want to deploy the BLAS or whatever required dependencies to accelerate
the computation. But I don't know how to deploy those DLLs when I submit my
JAR to the cluster.
Both spark-submit and spark-shell have a --jars option for passing
additional jars to the cluster. They will be added to the appropriate
classpaths.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
http://shop.oreilly.com/product/0636920033073.do (O'Reilly)
Typesafe http://typesafe.com
I would recommend to upload those jars to HDFS, and use add jars
option in spark-submit with URI from HDFS instead of URI from local
filesystem. Thus, it can avoid the problem of fetching jars from
driver which can be a bottleneck.
Sincerely,
DB Tsai
