Thanks for the hint! I made some progress. By adding task.opts to the job configuration file the JVM was able to find the library cql_engine12d. However, this library depends on some other libraries e.g. cql_types_12d (located in the same directory as cql_engine12d) and am hitting the following error:
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode) org.apache.samza.SamzaException: java.lang.UnsatisfiedLinkError: /home/myuser/path/apps-lib/libcql_engine12d.so: libcql_types_12d.so: cannot open shared object file: No such file or directory at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:144) at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:730) at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:12 2) at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:89) at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala) Also, I donĀ¹t have problems calling the library from another non-samza application. Thanks, Angelica On 6/8/17, 6:25 PM, "Jagadish Venkatraman" <jagadish1...@gmail.com> wrote: >Hi Angelica, > >Linkage error usually occurs when the JVM runtime is not able to find the >actual path for the native library. > >I assume you encounter this error in your container when you are running >on >Yarn. Samza exposes a property called *task.opts* which will allow you to >pass in command line JVM options to your containers. I'd recommend setting >task.opts = >-Djava.library.path="D:/Projects/lib/your_library_dir/sub_dir/ > > >> I am printing the value for java.library.path and I can see the >following path to the folder that contains the library cql_engine12d. >Are you printing it inside your StreamTask implemention? > >Also, Just to isolate that it's not a problem with your environment, you >can try invoking the library call from a non-Samza program? > >Let us know how it goes! > >Best, >Jagadish > > >On Thu, Jun 8, 2017 at 12:23 PM, Angelica Garcia-Gutierrez < >agar...@transcendinsights.com> wrote: > >> Hello, >> I am trying to rung a job that uses additional custom libraries >> (cql_engine12). >> During processing I can see the following error in one of the container >> logs: >> >> >> 2017-06-08 12:09:56 AsyncRunLoop [ERROR] Caught throwable and stopping >>run >> loop >> java.lang.UnsatisfiedLinkError: no cql_engine12d in java.library.path >> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864) >> at java.lang.Runtime.loadLibrary0(Runtime.java:870) >> at java.lang.System.loadLibrary(System.java:1122) >> at com.transcendinsights.cql.CqlEngine.<clinit>(CqlEngine.java:15) >> at samza.examples.wikipedia.task.MyTask.evaluate(MyTask.java:138) >> at samza.examples.wikipedia.task.MyTask.process(MyTask.java:118) >> at org.apache.samza.task.AsyncStreamTaskAdapter.process( >> AsyncStreamTaskAdapter.java:72) >> at org.apache.samza.task.AsyncStreamTaskAdapter.processAsync( >> AsyncStreamTaskAdapter.java:63) >> at org.apache.samza.container.TaskInstance$$anonfun$process$ >> 1.apply$mcV$sp(TaskInstance.scala:162) >> at org.apache.samza.container.TaskInstanceExceptionHandler. >> maybeHandle(TaskInstanceExceptionHandler.scala:54) >> at org.apache.samza.container.TaskInstance.process( >> TaskInstance.scala:160) >> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker. >> process(AsyncRunLoop.java:425) >> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker. >> run(AsyncRunLoop.java:370) >> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker. >> access$300(AsyncRunLoop.java:311) >> at >>org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:225) >> at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:154) >> at org.apache.samza.container.SamzaContainer.run( >> SamzaContainer.scala:730) >> at org.apache.samza.container.SamzaContainer$.safeMain( >> SamzaContainer.scala:122) >> at org.apache.samza.container.SamzaContainer$.main( >> SamzaContainer.scala:89) >> at org.apache.samza.container.SamzaContainer.main( >> SamzaContainer.scala) >> 2017-06-08 12:09:56 SamzaContainer [ERROR] Caught exception/error in >> process loop. >> >> I am printing the value for java.library.path and I can see the >>following >> path to the folder that contains the library cql_engine12d. >> >> Any advice about what other process can be overriding the value for my >> library path? >> >> Thanks, >> Angelica >> >> >> >> >> >> The information transmitted is intended only for the person or entity to >> which it is addressed >> and may contain CONFIDENTIAL material. If you receive this >> material/information in error, >> please contact the sender and delete or destroy the >>material/information. > > > > >-- >Jagadish V, >Graduate Student, >Department of Computer Science, >Stanford University The information transmitted is intended only for the person or entity to which it is addressed and may contain CONFIDENTIAL material. If you receive this material/information in error, please contact the sender and delete or destroy the material/information.