If it is working for standalone program I would think you can apply the same settings across all the spark worker and client machines and give that a try. Lets start with that.
On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <start.vin...@gmail.com> wrote: > Just subscribed to Spark User. So, forwarding message again. > > On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com> > wrote: > >> Thanks Kant. Can you give me a sample program which allows me to call jni >> from executor task ? I have jni working in standalone program in >> scala/java. >> >> Regards, >> Vineet >> >> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> wrote: >> >>> Yes this is a Java JNI question. Nothing to do with Spark really. >>> >>> java.lang.UnsatisfiedLinkError typically would mean the way you setup >>> LD_LIBRARY_PATH >>> is wrong unless you tell us that it is working for other cases but not this >>> one. >>> >>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com> >>> wrote: >>> >>>> That's just standard JNI and has nothing to do with Spark, does it? >>>> >>>> >>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <start.vin...@gmail.com >>>> > wrote: >>>> >>>>> Thanks Reynold for quick reply. >>>>> >>>>> I have tried following: >>>>> >>>>> class MySimpleApp { >>>>> // ---Native methods >>>>> @native def fooMethod (foo: String): String >>>>> } >>>>> >>>>> object MySimpleApp { >>>>> val flag = false >>>>> def loadResources() { >>>>> System.loadLibrary("foo-C-library") >>>>> val flag = true >>>>> } >>>>> def main() { >>>>> sc.parallelize(1 to 10).mapPartitions ( iter => { >>>>> if(flag == false){ >>>>> MySimpleApp.loadResources() >>>>> val SimpleInstance = new MySimpleApp >>>>> } >>>>> SimpleInstance.fooMethod ("fooString") >>>>> iter >>>>> }) >>>>> } >>>>> } >>>>> >>>>> I don't see way to invoke fooMethod which is implemented in >>>>> foo-C-library. Is I am missing something ? If possible, can you point me >>>>> to >>>>> existing implementation which i can refer to. >>>>> >>>>> Thanks again. >>>>> >>>>> ~ >>>>> >>>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com> >>>>> wrote: >>>>> >>>>>> bcc dev@ and add user@ >>>>>> >>>>>> >>>>>> This is more a user@ list question rather than a dev@ list question. >>>>>> You can do something like this: >>>>>> >>>>>> object MySimpleApp { >>>>>> def loadResources(): Unit = // define some idempotent way to load >>>>>> resources, e.g. with a flag or lazy val >>>>>> >>>>>> def main() = { >>>>>> ... >>>>>> >>>>>> sc.parallelize(1 to 10).mapPartitions { iter => >>>>>> MySimpleApp.loadResources() >>>>>> >>>>>> // do whatever you want with the iterator >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha < >>>>>> start.vin...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am trying to invoke C library from the Spark Stack using JNI >>>>>>> interface (here is sample application code) >>>>>>> >>>>>>> >>>>>>> class SimpleApp { >>>>>>> // ---Native methods >>>>>>> @native def foo (Top: String): String >>>>>>> } >>>>>>> >>>>>>> object SimpleApp { >>>>>>> def main(args: Array[String]) { >>>>>>> >>>>>>> val conf = new SparkConf().setAppName("Simple >>>>>>> Application").set("SPARK_LIBRARY_PATH", "lib") >>>>>>> val sc = new SparkContext(conf) >>>>>>> System.loadLibrary("foolib") >>>>>>> //instantiate the class >>>>>>> val SimpleAppInstance = new SimpleApp >>>>>>> //String passing - Working >>>>>>> val ret = SimpleAppInstance.foo("fooString") >>>>>>> } >>>>>>> >>>>>>> Above code work fines. >>>>>>> >>>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath, >>>>>>> spark.executor.extraLibraryPath at worker node >>>>>>> >>>>>>> How can i invoke JNI library from worker node ? Where should i load >>>>>>> it in executor ? >>>>>>> Calling System.loadLibrary("foolib") inside the work node gives me >>>>>>> following error : >>>>>>> >>>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError: >>>>>>> >>>>>>> Any help would be really appreciated. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >