Thanks for the hint!
I made some progress.
By adding task.opts to the job configuration file the JVM was able to find
the library cql_engine12d. However, this library depends on some other
libraries e.g. cql_types_12d (located in the same directory as
cql_engine12d) and am hitting the following error:
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)
org.apache.samza.SamzaException: java.lang.UnsatisfiedLinkError:
/home/myuser/path/apps-lib/libcql_engine12d.so: libcql_types_12d.so:
cannot open shared object file: No such file or directory
at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:144)
at
org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:730)
at
org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:12
2)
at
org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:89)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
Also, I don¹t have problems calling the library from another non-samza
application.
Thanks,
Angelica
On 6/8/17, 6:25 PM, "Jagadish Venkatraman" <jagadish1...@gmail.com> wrote:
>Hi Angelica,
>
>Linkage error usually occurs when the JVM runtime is not able to find the
>actual path for the native library.
>
>I assume you encounter this error in your container when you are running
>on
>Yarn. Samza exposes a property called *task.opts* which will allow you to
>pass in command line JVM options to your containers. I'd recommend setting
>task.opts =
>-Djava.library.path="D:/Projects/lib/your_library_dir/sub_dir/
>
> >> I am printing the value for java.library.path and I can see the
>following path to the folder that contains the library cql_engine12d.
>Are you printing it inside your StreamTask implemention?
>
>Also, Just to isolate that it's not a problem with your environment, you
>can try invoking the library call from a non-Samza program?
>
>Let us know how it goes!
>
>Best,
>Jagadish
>
>
>On Thu, Jun 8, 2017 at 12:23 PM, Angelica Garcia-Gutierrez <
>agar...@transcendinsights.com> wrote:
>
>> Hello,
>> I am trying to rung a job that uses additional custom libraries
>> (cql_engine12).
>> During processing I can see the following error in one of the container
>> logs:
>>
>>
>> 2017-06-08 12:09:56 AsyncRunLoop [ERROR] Caught throwable and stopping
>>run
>> loop
>> java.lang.UnsatisfiedLinkError: no cql_engine12d in java.library.path
>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
>> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>> at java.lang.System.loadLibrary(System.java:1122)
>> at com.transcendinsights.cql.CqlEngine.<clinit>(CqlEngine.java:15)
>> at samza.examples.wikipedia.task.MyTask.evaluate(MyTask.java:138)
>> at samza.examples.wikipedia.task.MyTask.process(MyTask.java:118)
>> at org.apache.samza.task.AsyncStreamTaskAdapter.process(
>> AsyncStreamTaskAdapter.java:72)
>> at org.apache.samza.task.AsyncStreamTaskAdapter.processAsync(
>> AsyncStreamTaskAdapter.java:63)
>> at org.apache.samza.container.TaskInstance$$anonfun$process$
>> 1.apply$mcV$sp(TaskInstance.scala:162)
>> at org.apache.samza.container.TaskInstanceExceptionHandler.
>> maybeHandle(TaskInstanceExceptionHandler.scala:54)
>> at org.apache.samza.container.TaskInstance.process(
>> TaskInstance.scala:160)
>> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.
>> process(AsyncRunLoop.java:425)
>> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.
>> run(AsyncRunLoop.java:370)
>> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.
>> access$300(AsyncRunLoop.java:311)
>> at
>>org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:225)
>> at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:154)
>> at org.apache.samza.container.SamzaContainer.run(
>> SamzaContainer.scala:730)
>> at org.apache.samza.container.SamzaContainer$.safeMain(
>> SamzaContainer.scala:122)
>> at org.apache.samza.container.SamzaContainer$.main(
>> SamzaContainer.scala:89)
>> at org.apache.samza.container.SamzaContainer.main(
>> SamzaContainer.scala)
>> 2017-06-08 12:09:56 SamzaContainer [ERROR] Caught exception/error in
>> process loop.
>>
>> I am printing the value for java.library.path and I can see the
>>following
>> path to the folder that contains the library cql_engine12d.
>>
>> Any advice about what other process can be overriding the value for my
>> library path?
>>
>> Thanks,
>> Angelica
>>
>>
>>
>>
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed
>> and may contain CONFIDENTIAL material. If you receive this
>> material/information in error,
>> please contact the sender and delete or destroy the
>>material/information.
>
>
>
>
>--
>Jagadish V,
>Graduate Student,
>Department of Computer Science,
>Stanford University
The information transmitted is intended only for the person or entity to which
it is addressed
and may contain CONFIDENTIAL material. If you receive this
material/information in error,
please contact the sender and delete or destroy the material/information.
