Just subscribed to Spark User. So, forwarding message again. On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com> wrote:
> Thanks Kant. Can you give me a sample program which allows me to call jni > from executor task ? I have jni working in standalone program in > scala/java. > > Regards, > Vineet > > On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> wrote: > >> Yes this is a Java JNI question. Nothing to do with Spark really. >> >> java.lang.UnsatisfiedLinkError typically would mean the way you setup >> LD_LIBRARY_PATH >> is wrong unless you tell us that it is working for other cases but not this >> one. >> >> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com> >> wrote: >> >>> That's just standard JNI and has nothing to do with Spark, does it? >>> >>> >>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <start.vin...@gmail.com> >>> wrote: >>> >>>> Thanks Reynold for quick reply. >>>> >>>> I have tried following: >>>> >>>> class MySimpleApp { >>>> // ---Native methods >>>> @native def fooMethod (foo: String): String >>>> } >>>> >>>> object MySimpleApp { >>>> val flag = false >>>> def loadResources() { >>>> System.loadLibrary("foo-C-library") >>>> val flag = true >>>> } >>>> def main() { >>>> sc.parallelize(1 to 10).mapPartitions ( iter => { >>>> if(flag == false){ >>>> MySimpleApp.loadResources() >>>> val SimpleInstance = new MySimpleApp >>>> } >>>> SimpleInstance.fooMethod ("fooString") >>>> iter >>>> }) >>>> } >>>> } >>>> >>>> I don't see way to invoke fooMethod which is implemented in >>>> foo-C-library. Is I am missing something ? If possible, can you point me to >>>> existing implementation which i can refer to. >>>> >>>> Thanks again. >>>> >>>> ~ >>>> >>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com> >>>> wrote: >>>> >>>>> bcc dev@ and add user@ >>>>> >>>>> >>>>> This is more a user@ list question rather than a dev@ list question. >>>>> You can do something like this: >>>>> >>>>> object MySimpleApp { >>>>> def loadResources(): Unit = // define some idempotent way to load >>>>> resources, e.g. with a flag or lazy val >>>>> >>>>> def main() = { >>>>> ... >>>>> >>>>> sc.parallelize(1 to 10).mapPartitions { iter => >>>>> MySimpleApp.loadResources() >>>>> >>>>> // do whatever you want with the iterator >>>>> } >>>>> } >>>>> } >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <start.vin...@gmail.com >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to invoke C library from the Spark Stack using JNI >>>>>> interface (here is sample application code) >>>>>> >>>>>> >>>>>> class SimpleApp { >>>>>> // ---Native methods >>>>>> @native def foo (Top: String): String >>>>>> } >>>>>> >>>>>> object SimpleApp { >>>>>> def main(args: Array[String]) { >>>>>> >>>>>> val conf = new SparkConf().setAppName("Simple >>>>>> Application").set("SPARK_LIBRARY_PATH", "lib") >>>>>> val sc = new SparkContext(conf) >>>>>> System.loadLibrary("foolib") >>>>>> //instantiate the class >>>>>> val SimpleAppInstance = new SimpleApp >>>>>> //String passing - Working >>>>>> val ret = SimpleAppInstance.foo("fooString") >>>>>> } >>>>>> >>>>>> Above code work fines. >>>>>> >>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath, >>>>>> spark.executor.extraLibraryPath at worker node >>>>>> >>>>>> How can i invoke JNI library from worker node ? Where should i load >>>>>> it in executor ? >>>>>> Calling System.loadLibrary("foolib") inside the work node gives me >>>>>> following error : >>>>>> >>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError: >>>>>> >>>>>> Any help would be really appreciated. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >