Re: Third party library
Thanks Jakob for sharing the link. Will try it out.
Regards,
Vineet
On Tue, Dec 13, 2016 at 3:00 PM, Jakob Odersky <ja...@odersky.com> wrote:
> Hi Vineet,
> great to see you solved the problem! Since this just appeared in my
> inbox, I wanted to take the opportunity for a shameless plug:
> https://github.com/jodersky/sbt-jni. In case you're using sbt and also
> developing the native library, this plugin may help with the pains of
> building and packaging JNI applications.
>
> cheers,
> --Jakob
>
> On Tue, Dec 13, 2016 at 11:02 AM, vineet chadha <start.vin...@gmail.com>
> wrote:
> > Thanks Steve and Kant. Apologies for late reply as I was out for
> vacation.
> > Got it working. For other users:
> >
> > def loadResources() {
> >
> > System.loadLibrary("foolib")
> >
> > val MyInstance = new MyClass
> >
> > val retstr = MyInstance.foo("mystring") // method trying to
> invoke
> >
> > }
> >
> > val conf = new
> > SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
> > "/lib/location")
> >
> > val sc = new SparkContext(conf)
> >
> > sc.parallelize(1 to 10, 2).mapPartitions ( iter => {
> >
> > MySimpleApp.loadResources()
> >
> > iter
> >
> > }).count
> >
> >
> >
> > Regards,
> > Vineet
> >
> > On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <ste...@hortonworks.com>
> > wrote:
> >>
> >>
> >> On 27 Nov 2016, at 02:55, kant kodali <kanth...@gmail.com> wrote:
> >>
> >> I would say instead of LD_LIBRARY_PATH you might want to use
> >> java.library.path
> >>
> >> in the following way
> >>
> >> java -Djava.library.path=/path/to/my/library or pass java.library.path
> >> along with spark-submit
> >>
> >>
> >> This is only going to set up paths on the submitting system; to load JNI
> >> code in the executors, the binary needs to be sent to far end and then
> put
> >> on the Java load path there.
> >>
> >> Copy the relevant binary to somewhere on the PATH of the destination
> >> machine. Do that and you shouldn't have to worry about other JVM
> options,
> >> (though it's been a few years since I did any JNI).
> >>
> >> One trick: write a simple main() object/entry point which calls the JNI
> >> method, and doesn't attempt to use any spark libraries; have it log any
> >> exception and return an error code if the call failed. This will let
> you use
> >> it as a link test after deployment: if you can't run that class then
> things
> >> are broken, before you go near spark
> >>
> >>
> >> On Sat, Nov 26, 2016 at 6:44 PM, Gmail <vgour...@gmail.com> wrote:
> >>>
> >>> Maybe you've already checked these out. Some basic questions that come
> to
> >>> my mind are:
> >>> 1) is this library "foolib" or "foo-C-library" available on the worker
> >>> node?
> >>> 2) if yes, is it accessible by the user/program (rwx)?
> >>>
> >>> Thanks,
> >>> Vasu.
> >>>
> >>> On Nov 26, 2016, at 5:08 PM, kant kodali <kanth...@gmail.com> wrote:
> >>>
> >>> If it is working for standalone program I would think you can apply the
> >>> same settings across all the spark worker and client machines and
> give that
> >>> a try. Lets start with that.
> >>>
> >>> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <
> start.vin...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Just subscribed to Spark User. So, forwarding message again.
> >>>>
> >>>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <
> start.vin...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Thanks Kant. Can you give me a sample program which allows me to call
> >>>>> jni from executor task ? I have jni working in standalone program
> in
> >>>>> scala/java.
> >>>>>
> >>>>> Regards,
> >>>>> Vineet
> >>>>>
> >>>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Yes this is a Java JNI question. Nothing to do with Spark rea
Re: Third party library
Thanks Steve and Kant. Apologies for late reply as I was out for vacation.
Got it working. For other users:
def loadResources() {
System.loadLibrary("foolib")
val MyInstance = new MyClass
val retstr = MyInstance.foo("mystring") // method trying to invoke
}
val conf = new
SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
"/lib/location")
val sc = new SparkContext(conf)
sc.parallelize(1 to 10, 2).mapPartitions ( iter => {
MySimpleApp.loadResources()
iter
}).count
Regards,
Vineet
On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <ste...@hortonworks.com>
wrote:
>
> On 27 Nov 2016, at 02:55, kant kodali <kanth...@gmail.com> wrote:
>
> I would say instead of LD_LIBRARY_PATH you might want to use java.library.
> path
>
> in the following way
>
> java -Djava.library.path=/path/to/my/library or pass java.library.path
> along with spark-submit
>
>
> This is only going to set up paths on the submitting system; to load JNI
> code in the executors, the binary needs to be sent to far end and then put
> on the Java load path there.
>
> Copy the relevant binary to somewhere on the PATH of the destination
> machine. Do that and you shouldn't have to worry about other JVM options,
> (though it's been a few years since I did any JNI).
>
> One trick: write a simple main() object/entry point which calls the JNI
> method, and doesn't attempt to use any spark libraries; have it log any
> exception and return an error code if the call failed. This will let you
> use it as a link test after deployment: if you can't run that class then
> things are broken, before you go near spark
>
>
> On Sat, Nov 26, 2016 at 6:44 PM, Gmail <vgour...@gmail.com> wrote:
>
>> Maybe you've already checked these out. Some basic questions that come to
>> my mind are:
>> 1) is this library "foolib" or "foo-C-library" available on the worker
>> node?
>> 2) if yes, is it accessible by the user/program (rwx)?
>>
>> Thanks,
>> Vasu.
>>
>> On Nov 26, 2016, at 5:08 PM, kant kodali <kanth...@gmail.com> wrote:
>>
>> If it is working for standalone program I would think you can apply the
>> same settings across all the spark worker and client machines and give
>> that a try. Lets start with that.
>>
>> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <start.vin...@gmail.com>
>> wrote:
>>
>>> Just subscribed to Spark User. So, forwarding message again.
>>>
>>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Kant. Can you give me a sample program which allows me to call
>>>> jni from executor task ? I have jni working in standalone program in
>>>> scala/java.
>>>>
>>>> Regards,
>>>> Vineet
>>>>
>>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes this is a Java JNI question. Nothing to do with Spark really.
>>>>>
>>>>> java.lang.UnsatisfiedLinkError typically would mean the way you
>>>>> setup LD_LIBRARY_PATH is wrong unless you tell us that it is working
>>>>> for other cases but not this one.
>>>>>
>>>>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com>
>>>>> wrote:
>>>>>
>>>>>> That's just standard JNI and has nothing to do with Spark, does it?
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <
>>>>>> start.vin...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Reynold for quick reply.
>>>>>>>
>>>>>>> I have tried following:
>>>>>>>
>>>>>>> class MySimpleApp {
>>>>>>> // ---Native methods
>>>>>>> @native def fooMethod (foo: String): String
>>>>>>> }
>>>>>>>
>>>>>>> object MySimpleApp {
>>>>>>> val flag = false
>>>>>>> def loadResources() {
>>>>>>> System.loadLibrary("foo-C-library")
>>>>>>> val flag = true
>>>>>>> }
>>>>>>> def main() {
>>>>>>> sc.parallelize(1 to 10).mapPartitions ( iter => {
>>>>>>
Re: Third party library
Just subscribed to Spark User. So, forwarding message again.
On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com>
wrote:
> Thanks Kant. Can you give me a sample program which allows me to call jni
> from executor task ? I have jni working in standalone program in
> scala/java.
>
> Regards,
> Vineet
>
> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> wrote:
>
>> Yes this is a Java JNI question. Nothing to do with Spark really.
>>
>> java.lang.UnsatisfiedLinkError typically would mean the way you setup
>> LD_LIBRARY_PATH
>> is wrong unless you tell us that it is working for other cases but not this
>> one.
>>
>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com>
>> wrote:
>>
>>> That's just standard JNI and has nothing to do with Spark, does it?
>>>
>>>
>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <start.vin...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Reynold for quick reply.
>>>>
>>>> I have tried following:
>>>>
>>>> class MySimpleApp {
>>>> // ---Native methods
>>>> @native def fooMethod (foo: String): String
>>>> }
>>>>
>>>> object MySimpleApp {
>>>> val flag = false
>>>> def loadResources() {
>>>> System.loadLibrary("foo-C-library")
>>>> val flag = true
>>>> }
>>>> def main() {
>>>> sc.parallelize(1 to 10).mapPartitions ( iter => {
>>>> if(flag == false){
>>>> MySimpleApp.loadResources()
>>>> val SimpleInstance = new MySimpleApp
>>>> }
>>>> SimpleInstance.fooMethod ("fooString")
>>>> iter
>>>> })
>>>> }
>>>> }
>>>>
>>>> I don't see way to invoke fooMethod which is implemented in
>>>> foo-C-library. Is I am missing something ? If possible, can you point me to
>>>> existing implementation which i can refer to.
>>>>
>>>> Thanks again.
>>>>
>>>> ~
>>>>
>>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com>
>>>> wrote:
>>>>
>>>>> bcc dev@ and add user@
>>>>>
>>>>>
>>>>> This is more a user@ list question rather than a dev@ list question.
>>>>> You can do something like this:
>>>>>
>>>>> object MySimpleApp {
>>>>> def loadResources(): Unit = // define some idempotent way to load
>>>>> resources, e.g. with a flag or lazy val
>>>>>
>>>>> def main() = {
>>>>> ...
>>>>>
>>>>> sc.parallelize(1 to 10).mapPartitions { iter =>
>>>>> MySimpleApp.loadResources()
>>>>>
>>>>> // do whatever you want with the iterator
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <start.vin...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to invoke C library from the Spark Stack using JNI
>>>>>> interface (here is sample application code)
>>>>>>
>>>>>>
>>>>>> class SimpleApp {
>>>>>> // ---Native methods
>>>>>> @native def foo (Top: String): String
>>>>>> }
>>>>>>
>>>>>> object SimpleApp {
>>>>>>def main(args: Array[String]) {
>>>>>>
>>>>>> val conf = new SparkConf().setAppName("Simple
>>>>>> Application").set("SPARK_LIBRARY_PATH", "lib")
>>>>>> val sc = new SparkContext(conf)
>>>>>> System.loadLibrary("foolib")
>>>>>> //instantiate the class
>>>>>> val SimpleAppInstance = new SimpleApp
>>>>>> //String passing - Working
>>>>>> val ret = SimpleAppInstance.foo("fooString")
>>>>>> }
>>>>>>
>>>>>> Above code work fines.
>>>>>>
>>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath,
>>>>>> spark.executor.extraLibraryPath at worker node
>>>>>>
>>>>>> How can i invoke JNI library from worker node ? Where should i load
>>>>>> it in executor ?
>>>>>> Calling System.loadLibrary("foolib") inside the work node gives me
>>>>>> following error :
>>>>>>
>>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>>>>>>
>>>>>> Any help would be really appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
