Its messing up your classpath, there was a discussion happened here
previously
https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/spark-on-yarn-java-lang-UnsatisfiedLinkError-NativeCodeLoader/td-p/22724
Thanks
Best Regards
On Tue, Sep 1, 2015 at 4:58 PM, cgalan <cgalangali...@gmail.com> wrote:
> Hi,
>
> When I am submitting a spark job in the mode "yarn-cluster" with the
> parameter "spark.driver.userClassPathFirst", my job fails; but if I don't
> use this params, my job is concluded with success.. My environment is some
> nodes with CDH5.4 and Spark 1.3.0.
>
> Spark submit with fail:
> spark-submit --class Main --master yarn-cluster --conf
> spark.driver.userClassPathFirst=true --conf
> spark.executor.userClassPathFirst=true Main.jar
>
> Spark-submit with success:
> spark-submit --class Main --master yarn-cluster Main.jar
>
> Error with Snappy:
> java.lang.UnsatisfiedLinkError:
> org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
> at org.xerial.snappy.SnappyNative.maxCompressedLength(Native
> Method)
> at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:316)
> at
> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
> at
>
> org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:157)
> at
>
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
> at
>
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
> at scala.Option.map(Option.scala:145)
> at
>
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:199)
> at
>
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101)
> at
>
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
> at
>
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
>
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
> at
>
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:761)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:589)
>
> My example error code:
> public static void main(String[] args) {
>
> SparkConf conf = new SparkConf().setAppName("Prueba");
> // conf.setMaster("local"); //Comentar para lanzar en el
> cluster
>
> JavaSparkContext sc = new JavaSparkContext(conf);
> JavaRDD asd = sc.textFile("text.txt");
> asd.count();
>
> sc.close();
> }
>
> Does anyone have any suggestions? The problem of why I need to use
> "spark.driver.userClassPathFirst=true" is because I use commons-cli-1.3.1
> in
> my project, and the spark classpath have one previous version, so I create
> a shade-jar, but without "spark.driver.userClassPathFirst=true" I have
> problems of conflicts with the dependencies between my spark.jar and spark
> classpath that one class uses the previous class version of classpath
> instead of the last version.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Error-using-spark-driver-userClassPathFirst-true-tp24536.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
