[ https://issues.apache.org/jira/browse/AMBARI-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weiqing Yang updated AMBARI-18112: ---------------------------------- Description: When spark applications run in yarn cluster mode, there is an error from executors: --------------------------------------------------------------------------------- java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1889) at java.lang.Runtime.loadLibrary0(Runtime.java:849) at java.lang.System.loadLibrary(System.java:1088) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71) at java.lang.Class.forName0(Native Method) ....... org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/08/08 12:59:03 ERROR LzoCodec: Cannot load native-lzo without native-hadoop. In the patch, it will set “spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64” in spark-defaults.conf. was: When spark applications run in yarn cluster mode, there is an exception from executors: --------------------------------------------------------------------------------- java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1889) at java.lang.Runtime.loadLibrary0(Runtime.java:849) at java.lang.System.loadLibrary(System.java:1088) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71) at java.lang.Class.forName0(Native Method) ....... org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/08/08 12:59:03 ERROR LzoCodec: Cannot load native-lzo without native-hadoop. In the patch, it will set “spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64” in spark-defaults.conf. > Fix spark.driver.extraLibraryPath to include native gpl library > --------------------------------------------------------------- > > Key: AMBARI-18112 > URL: https://issues.apache.org/jira/browse/AMBARI-18112 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: trunk, 2.5.0 > Reporter: Weiqing Yang > > When spark applications run in yarn cluster mode, there is an error from > executors: > --------------------------------------------------------------------------------- > java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1889) > at java.lang.Runtime.loadLibrary0(Runtime.java:849) > at java.lang.System.loadLibrary(System.java:1088) > at > com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32) > at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71) > at java.lang.Class.forName0(Native Method) > ....... > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 16/08/08 12:59:03 ERROR LzoCodec: Cannot load native-lzo without > native-hadoop. > In the patch, it will set “spark.executor.extraLibraryPath > /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64” > in spark-defaults.conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)