Just subscribed to Spark User. So, forwarding message again.
On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com>
wrote:
> Thanks Kant. Can you give me a sample program which allows me to call jni
> from executor task ? I have jni working in standalone program in
> scala/java.
>
> Regards,
> Vineet
>
> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com> wrote:
>
>> Yes this is a Java JNI question. Nothing to do with Spark really.
>>
>> java.lang.UnsatisfiedLinkError typically would mean the way you setup
>> LD_LIBRARY_PATH
>> is wrong unless you tell us that it is working for other cases but not this
>> one.
>>
>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com>
>> wrote:
>>
>>> That's just standard JNI and has nothing to do with Spark, does it?
>>>
>>>
>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <start.vin...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Reynold for quick reply.
>>>>
>>>> I have tried following:
>>>>
>>>> class MySimpleApp {
>>>> // ---Native methods
>>>> @native def fooMethod (foo: String): String
>>>> }
>>>>
>>>> object MySimpleApp {
>>>> val flag = false
>>>> def loadResources() {
>>>> System.loadLibrary("foo-C-library")
>>>> val flag = true
>>>> }
>>>> def main() {
>>>> sc.parallelize(1 to 10).mapPartitions ( iter => {
>>>> if(flag == false){
>>>> MySimpleApp.loadResources()
>>>> val SimpleInstance = new MySimpleApp
>>>> }
>>>> SimpleInstance.fooMethod ("fooString")
>>>> iter
>>>> })
>>>> }
>>>> }
>>>>
>>>> I don't see way to invoke fooMethod which is implemented in
>>>> foo-C-library. Is I am missing something ? If possible, can you point me to
>>>> existing implementation which i can refer to.
>>>>
>>>> Thanks again.
>>>>
>>>> ~
>>>>
>>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com>
>>>> wrote:
>>>>
>>>>> bcc dev@ and add user@
>>>>>
>>>>>
>>>>> This is more a user@ list question rather than a dev@ list question.
>>>>> You can do something like this:
>>>>>
>>>>> object MySimpleApp {
>>>>> def loadResources(): Unit = // define some idempotent way to load
>>>>> resources, e.g. with a flag or lazy val
>>>>>
>>>>> def main() = {
>>>>> ...
>>>>>
>>>>> sc.parallelize(1 to 10).mapPartitions { iter =>
>>>>> MySimpleApp.loadResources()
>>>>>
>>>>> // do whatever you want with the iterator
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <start.vin...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to invoke C library from the Spark Stack using JNI
>>>>>> interface (here is sample application code)
>>>>>>
>>>>>>
>>>>>> class SimpleApp {
>>>>>> // ---Native methods
>>>>>> @native def foo (Top: String): String
>>>>>> }
>>>>>>
>>>>>> object SimpleApp {
>>>>>> def main(args: Array[String]) {
>>>>>>
>>>>>> val conf = new SparkConf().setAppName("Simple
>>>>>> Application").set("SPARK_LIBRARY_PATH", "lib")
>>>>>> val sc = new SparkContext(conf)
>>>>>> System.loadLibrary("foolib")
>>>>>> //instantiate the class
>>>>>> val SimpleAppInstance = new SimpleApp
>>>>>> //String passing - Working
>>>>>> val ret = SimpleAppInstance.foo("fooString")
>>>>>> }
>>>>>>
>>>>>> Above code work fines.
>>>>>>
>>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath,
>>>>>> spark.executor.extraLibraryPath at worker node
>>>>>>
>>>>>> How can i invoke JNI library from worker node ? Where should i load
>>>>>> it in executor ?
>>>>>> Calling System.loadLibrary("foolib") inside the work node gives me
>>>>>> following error :
>>>>>>
>>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>>>>>>
>>>>>> Any help would be really appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
