Hi, When I am submitting a spark job in the mode "yarn-cluster" with the parameter "spark.driver.userClassPathFirst", my job fails; but if I don't use this params, my job is concluded with success.. My environment is some nodes with CDH5.4 and Spark 1.3.0.
Spark submit with fail: spark-submit --class Main --master yarn-cluster --conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true Main.jar Spark-submit with success: spark-submit --class Main --master yarn-cluster Main.jar Error with Snappy: java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I at org.xerial.snappy.SnappyNative.maxCompressedLength(Native Method) at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:316) at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79) at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:157) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199) at scala.Option.map(Option.scala:145) at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:199) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051) at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:761) at org.apache.spark.SparkContext.textFile(SparkContext.scala:589) My example error code: public static void main(String[] args) { SparkConf conf = new SparkConf().setAppName("Prueba"); // conf.setMaster("local"); //Comentar para lanzar en el cluster JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD asd = sc.textFile("text.txt"); asd.count(); sc.close(); } Does anyone have any suggestions? The problem of why I need to use "spark.driver.userClassPathFirst=true" is because I use commons-cli-1.3.1 in my project, and the spark classpath have one previous version, so I create a shade-jar, but without "spark.driver.userClassPathFirst=true" I have problems of conflicts with the dependencies between my spark.jar and spark classpath that one class uses the previous class version of classpath instead of the last version. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-using-spark-driver-userClassPathFirst-true-tp24536.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org