Re: Third party library
Thanks Jakob for sharing the link. Will try it out. Regards, Vineet On Tue, Dec 13, 2016 at 3:00 PM, Jakob Odersky <ja...@odersky.com> wrote: > Hi Vineet, > great to see you solved the problem! Since this just appeared in my > inbox, I wanted to take the opportunity for a shameless plug: > https://github.com/jodersky/sbt-jni. In case you're using sbt and also > developing the native library, this plugin may help with the pains of > building and packaging JNI applications. > > cheers, > --Jakob > > On Tue, Dec 13, 2016 at 11:02 AM, vineet chadha <start.vin...@gmail.com> > wrote: > > Thanks Steve and Kant. Apologies for late reply as I was out for > vacation. > > Got it working. For other users: > > > > def loadResources() { > > > > System.loadLibrary("foolib") > > > > val MyInstance = new MyClass > > > > val retstr = MyInstance.foo("mystring") // method trying to > invoke > > > > } > > > > val conf = new > > SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH", > > "/lib/location") > > > > val sc = new SparkContext(conf) > > > > sc.parallelize(1 to 10, 2).mapPartitions ( iter => { > > > > MySimpleApp.loadResources() > > > > iter > > > > }).count > > > > > > > > Regards, > > Vineet > > > > On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <ste...@hortonworks.com> > > wrote: > >> > >> > >> On 27 Nov 2016, at 02:55, kant kodali <kanth...@gmail.com> wrote: > >> > >> I would say instead of LD_LIBRARY_PATH you might want to use > >> java.library.path > >> > >> in the following way > >> > >> java -Djava.library.path=/path/to/my/library or pass java.library.path > >> along with spark-submit > >> > >> > >> This is only going to set up paths on the submitting system; to load JNI > >> code in the executors, the binary needs to be sent to far end and then > put > >> on the Java load path there. > >> > >> Copy the relevant binary to somewhere on the PATH of the destination > >> machine. Do that and you shouldn't have to worry about other JVM > options, > >> (though it's been a few years since I did any JNI). > >> > >> One trick: write a simple main() object/entry point which calls the JNI > >> method, and doesn't attempt to use any spark libraries; have it log any > >> exception and return an error code if the call failed. This will let > you use > >> it as a link test after deployment: if you can't run that class then > things > >> are broken, before you go near spark > >> > >> > >> On Sat, Nov 26, 2016 at 6:44 PM, Gmail <vgour...@gmail.com> wrote: > >>> > >>> Maybe you've already checked these out. Some basic questions that come > to > >>> my mind are: > >>> 1) is this library "foolib" or "foo-C-library" available on the worker > >>> node? > >>> 2) if yes, is it accessible by the user/program (rwx)? > >>> > >>> Thanks, > >>> Vasu. > >>> > >>> On Nov 26, 2016, at 5:08 PM, kant kodali <kanth...@gmail.com> wrote: > >>> > >>> If it is working for standalone program I would think you can apply the > >>> same settings across all the spark worker and client machines and > give that > >>> a try. Lets start with that. > >>> > >>> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha < > start.vin...@gmail.com> > >>> wrote: > >>>> > >>>> Just subscribed to Spark User. So, forwarding message again. > >>>> > >>>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha < > start.vin...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Thanks Kant. Can you give me a sample program which allows me to call > >>>>> jni from executor task ? I have jni working in standalone program > in > >>>>> scala/java. > >>>>> > >>>>> Regards, > >>>>> Vineet > >>>>> > >>>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> Yes this is a Java JNI question. Nothing to do with Spark rea
Re: Third party library
Thanks Steve and Kant. Apologies for late reply as I was out for vacation. Got it working. For other users: def loadResources() { System.loadLibrary("foolib") val MyInstance = new MyClass val retstr = MyInstance.foo("mystring") // method trying to invoke } val conf = new SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH", "/lib/location") val sc = new SparkContext(conf) sc.parallelize(1 to 10, 2).mapPartitions ( iter => { MySimpleApp.loadResources() iter }).count Regards, Vineet On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 27 Nov 2016, at 02:55, kant kodali <kanth...@gmail.com> wrote: > > I would say instead of LD_LIBRARY_PATH you might want to use java.library. > path > > in the following way > > java -Djava.library.path=/path/to/my/library or pass java.library.path > along with spark-submit > > > This is only going to set up paths on the submitting system; to load JNI > code in the executors, the binary needs to be sent to far end and then put > on the Java load path there. > > Copy the relevant binary to somewhere on the PATH of the destination > machine. Do that and you shouldn't have to worry about other JVM options, > (though it's been a few years since I did any JNI). > > One trick: write a simple main() object/entry point which calls the JNI > method, and doesn't attempt to use any spark libraries; have it log any > exception and return an error code if the call failed. This will let you > use it as a link test after deployment: if you can't run that class then > things are broken, before you go near spark > > > On Sat, Nov 26, 2016 at 6:44 PM, Gmail <vgour...@gmail.com> wrote: > >> Maybe you've already checked these out. Some basic questions that come to >> my mind are: >> 1) is this library "foolib" or "foo-C-library" available on the worker >> node? >> 2) if yes, is it accessible by the user/program (rwx)? >> >> Thanks, >> Vasu. >> >> On Nov 26, 2016, at 5:08 PM, kant kodali <kanth...@gmail.com> wrote: >> >> If it is working for standalone program I would think you can apply the >> same settings across all the spark worker and client machines and give >> that a try. Lets start with that. >> >> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <start.vin...@gmail.com> >> wrote: >> >>> Just subscribed to Spark User. So, forwarding message again. >>> >>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com> >>> wrote: >>> >>>> Thanks Kant. Can you give me a sample program which allows me to call >>>> jni from executor task ? I have jni working in standalone program in >>>> scala/java. >>>> >>>> Regards, >>>> Vineet >>>> >>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >>>>> Yes this is a Java JNI question. Nothing to do with Spark really. >>>>> >>>>> java.lang.UnsatisfiedLinkError typically would mean the way you >>>>> setup LD_LIBRARY_PATH is wrong unless you tell us that it is working >>>>> for other cases but not this one. >>>>> >>>>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com> >>>>> wrote: >>>>> >>>>>> That's just standard JNI and has nothing to do with Spark, does it? >>>>>> >>>>>> >>>>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha < >>>>>> start.vin...@gmail.com> wrote: >>>>>> >>>>>>> Thanks Reynold for quick reply. >>>>>>> >>>>>>> I have tried following: >>>>>>> >>>>>>> class MySimpleApp { >>>>>>> // ---Native methods >>>>>>> @native def fooMethod (foo: String): String >>>>>>> } >>>>>>> >>>>>>> object MySimpleApp { >>>>>>> val flag = false >>>>>>> def loadResources() { >>>>>>> System.loadLibrary("foo-C-library") >>>>>>> val flag = true >>>>>>> } >>>>>>> def main() { >>>>>>> sc.parallelize(1 to 10).mapPartitions ( iter => { >>>>>>
Re: Third party library
Just subscribed to Spark User. So, forwarding message again. On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com> wrote: > Thanks Kant. Can you give me a sample program which allows me to call jni > from executor task ? I have jni working in standalone program in > scala/java. > > Regards, > Vineet > > On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> wrote: > >> Yes this is a Java JNI question. Nothing to do with Spark really. >> >> java.lang.UnsatisfiedLinkError typically would mean the way you setup >> LD_LIBRARY_PATH >> is wrong unless you tell us that it is working for other cases but not this >> one. >> >> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com> >> wrote: >> >>> That's just standard JNI and has nothing to do with Spark, does it? >>> >>> >>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <start.vin...@gmail.com> >>> wrote: >>> >>>> Thanks Reynold for quick reply. >>>> >>>> I have tried following: >>>> >>>> class MySimpleApp { >>>> // ---Native methods >>>> @native def fooMethod (foo: String): String >>>> } >>>> >>>> object MySimpleApp { >>>> val flag = false >>>> def loadResources() { >>>> System.loadLibrary("foo-C-library") >>>> val flag = true >>>> } >>>> def main() { >>>> sc.parallelize(1 to 10).mapPartitions ( iter => { >>>> if(flag == false){ >>>> MySimpleApp.loadResources() >>>> val SimpleInstance = new MySimpleApp >>>> } >>>> SimpleInstance.fooMethod ("fooString") >>>> iter >>>> }) >>>> } >>>> } >>>> >>>> I don't see way to invoke fooMethod which is implemented in >>>> foo-C-library. Is I am missing something ? If possible, can you point me to >>>> existing implementation which i can refer to. >>>> >>>> Thanks again. >>>> >>>> ~ >>>> >>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com> >>>> wrote: >>>> >>>>> bcc dev@ and add user@ >>>>> >>>>> >>>>> This is more a user@ list question rather than a dev@ list question. >>>>> You can do something like this: >>>>> >>>>> object MySimpleApp { >>>>> def loadResources(): Unit = // define some idempotent way to load >>>>> resources, e.g. with a flag or lazy val >>>>> >>>>> def main() = { >>>>> ... >>>>> >>>>> sc.parallelize(1 to 10).mapPartitions { iter => >>>>> MySimpleApp.loadResources() >>>>> >>>>> // do whatever you want with the iterator >>>>> } >>>>> } >>>>> } >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <start.vin...@gmail.com >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to invoke C library from the Spark Stack using JNI >>>>>> interface (here is sample application code) >>>>>> >>>>>> >>>>>> class SimpleApp { >>>>>> // ---Native methods >>>>>> @native def foo (Top: String): String >>>>>> } >>>>>> >>>>>> object SimpleApp { >>>>>>def main(args: Array[String]) { >>>>>> >>>>>> val conf = new SparkConf().setAppName("Simple >>>>>> Application").set("SPARK_LIBRARY_PATH", "lib") >>>>>> val sc = new SparkContext(conf) >>>>>> System.loadLibrary("foolib") >>>>>> //instantiate the class >>>>>> val SimpleAppInstance = new SimpleApp >>>>>> //String passing - Working >>>>>> val ret = SimpleAppInstance.foo("fooString") >>>>>> } >>>>>> >>>>>> Above code work fines. >>>>>> >>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath, >>>>>> spark.executor.extraLibraryPath at worker node >>>>>> >>>>>> How can i invoke JNI library from worker node ? Where should i load >>>>>> it in executor ? >>>>>> Calling System.loadLibrary("foolib") inside the work node gives me >>>>>> following error : >>>>>> >>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError: >>>>>> >>>>>> Any help would be really appreciated. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >