Re: how to run spark job on yarn with jni lib?
We currently choose not to run jobs on yarn, so I stop trying this. Anyway thanks for you guys' suggestions. At least, your solutions may help people who must run their jobs on yarn : ) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-run-spark-job-on-yarn-with-jni-lib-tp15146p15395.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
how to run spark job on yarn with jni lib?
Hi all, I tried to run my spark job on yarn. In my application, I need to call third-parity jni libraries in a spark job. However, I can't find a way to make spark job load my native libraries. Is there anyone who knows how to solve this problem? Thanks. Ziv Huang -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-run-spark-job-on-yarn-with-jni-lib-tp15146.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: how to run spark job on yarn with jni lib?
You're right, I'm suffering from SPARK-1719. I've tried to add their location to /etc/ld.so.conf and I've submitted my job as a yarn-client, but the problem is the same: my native libraries are not loaded. Does this method work in your case? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-run-spark-job-on-yarn-with-jni-lib-tp15146p15195.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
