Hi Vineet,
great to see you solved the problem! Since this just appeared in my
inbox, I wanted to take the opportunity for a shameless plug:
https://github.com/jodersky/sbt-jni. In case you're using sbt and also
developing the native library, this plugin may help with the pains of
building and packaging JNI applications.
cheers,
--Jakob
On Tue, Dec 13, 2016 at 11:02 AM, vineet chadha <start.vin...@gmail.com> wrote:
> Thanks Steve and Kant. Apologies for late reply as I was out for vacation.
> Got it working. For other users:
>
> def loadResources() {
>
> System.loadLibrary("foolib")
>
> val MyInstance = new MyClass
>
> val retstr = MyInstance.foo("mystring") // method trying to invoke
>
> }
>
> val conf = new
> SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
> "/lib/location")
>
> val sc = new SparkContext(conf)
>
> sc.parallelize(1 to 10, 2).mapPartitions ( iter => {
>
> MySimpleApp.loadResources()
>
> iter
>
> }).count
>
>
>
> Regards,
> Vineet
>
> On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <ste...@hortonworks.com>
> wrote:
>>
>>
>> On 27 Nov 2016, at 02:55, kant kodali <kanth...@gmail.com> wrote:
>>
>> I would say instead of LD_LIBRARY_PATH you might want to use
>> java.library.path
>>
>> in the following way
>>
>> java -Djava.library.path=/path/to/my/library or pass java.library.path
>> along with spark-submit
>>
>>
>> This is only going to set up paths on the submitting system; to load JNI
>> code in the executors, the binary needs to be sent to far end and then put
>> on the Java load path there.
>>
>> Copy the relevant binary to somewhere on the PATH of the destination
>> machine. Do that and you shouldn't have to worry about other JVM options,
>> (though it's been a few years since I did any JNI).
>>
>> One trick: write a simple main() object/entry point which calls the JNI
>> method, and doesn't attempt to use any spark libraries; have it log any
>> exception and return an error code if the call failed. This will let you use
>> it as a link test after deployment: if you can't run that class then things
>> are broken, before you go near spark
>>
>>
>> On Sat, Nov 26, 2016 at 6:44 PM, Gmail <vgour...@gmail.com> wrote:
>>>
>>> Maybe you've already checked these out. Some basic questions that come to
>>> my mind are:
>>> 1) is this library "foolib" or "foo-C-library" available on the worker
>>> node?
>>> 2) if yes, is it accessible by the user/program (rwx)?
>>>
>>> Thanks,
>>> Vasu.
>>>
>>> On Nov 26, 2016, at 5:08 PM, kant kodali <kanth...@gmail.com> wrote:
>>>
>>> If it is working for standalone program I would think you can apply the
>>> same settings across all the spark worker and client machines and give that
>>> a try. Lets start with that.
>>>
>>> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <start.vin...@gmail.com>
>>> wrote:
>>>>
>>>> Just subscribed to Spark User. So, forwarding message again.
>>>>
>>>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <start.vin...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Thanks Kant. Can you give me a sample program which allows me to call
>>>>> jni from executor task ? I have jni working in standalone program in
>>>>> scala/java.
>>>>>
>>>>> Regards,
>>>>> Vineet
>>>>>
>>>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <kanth...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Yes this is a Java JNI question. Nothing to do with Spark really.
>>>>>>
>>>>>> java.lang.UnsatisfiedLinkError typically would mean the way you setup
>>>>>> LD_LIBRARY_PATH is wrong unless you tell us that it is working for other
>>>>>> cases but not this one.
>>>>>>
>>>>>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <r...@databricks.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> That's just standard JNI and has nothing to do with Spark, does it?
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha
>>>>>>> <start.vin...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Reynold for quick reply.
>>>>>>>>
>>>>>>>> I have tried following:
>>>>>>>>
>>>>>>>> class MySimpleApp {
>>>>>>>> // ---Native methods
>>>>>>>> @native def fooMethod (foo: String): String
>>>>>>>> }
>>>>>>>>
>>>>>>>> object MySimpleApp {
>>>>>>>> val flag = false
>>>>>>>> def loadResources() {
>>>>>>>> System.loadLibrary("foo-C-library")
>>>>>>>> val flag = true
>>>>>>>> }
>>>>>>>> def main() {
>>>>>>>> sc.parallelize(1 to 10).mapPartitions ( iter => {
>>>>>>>> if(flag == false){
>>>>>>>> MySimpleApp.loadResources()
>>>>>>>> val SimpleInstance = new MySimpleApp
>>>>>>>> }
>>>>>>>> SimpleInstance.fooMethod ("fooString")
>>>>>>>> iter
>>>>>>>> })
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> I don't see way to invoke fooMethod which is implemented in
>>>>>>>> foo-C-library. Is I am missing something ? If possible, can you point
>>>>>>>> me to
>>>>>>>> existing implementation which i can refer to.
>>>>>>>>
>>>>>>>> Thanks again.
>>>>>>>>
>>>>>>>> ~
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <r...@databricks.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> bcc dev@ and add user@
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is more a user@ list question rather than a dev@ list
>>>>>>>>> question. You can do something like this:
>>>>>>>>>
>>>>>>>>> object MySimpleApp {
>>>>>>>>> def loadResources(): Unit = // define some idempotent way to load
>>>>>>>>> resources, e.g. with a flag or lazy val
>>>>>>>>>
>>>>>>>>> def main() = {
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> sc.parallelize(1 to 10).mapPartitions { iter =>
>>>>>>>>> MySimpleApp.loadResources()
>>>>>>>>>
>>>>>>>>> // do whatever you want with the iterator
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha
>>>>>>>>> <start.vin...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am trying to invoke C library from the Spark Stack using JNI
>>>>>>>>>> interface (here is sample application code)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> class SimpleApp {
>>>>>>>>>> // ---Native methods
>>>>>>>>>> @native def foo (Top: String): String
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> object SimpleApp {
>>>>>>>>>> def main(args: Array[String]) {
>>>>>>>>>>
>>>>>>>>>> val conf = new
>>>>>>>>>> SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
>>>>>>>>>> "lib")
>>>>>>>>>> val sc = new SparkContext(conf)
>>>>>>>>>> System.loadLibrary("foolib")
>>>>>>>>>> //instantiate the class
>>>>>>>>>> val SimpleAppInstance = new SimpleApp
>>>>>>>>>> //String passing - Working
>>>>>>>>>> val ret = SimpleAppInstance.foo("fooString")
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Above code work fines.
>>>>>>>>>>
>>>>>>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath,
>>>>>>>>>> spark.executor.extraLibraryPath at worker node
>>>>>>>>>>
>>>>>>>>>> How can i invoke JNI library from worker node ? Where should i
>>>>>>>>>> load it in executor ?
>>>>>>>>>> Calling System.loadLibrary("foolib") inside the work node gives
>>>>>>>>>> me following error :
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>>>>>>>>>>
>>>>>>>>>> Any help would be really appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
