I update the spark version form 1.02 to 1.10 , experienced an snappy version issue with the new Spark-1.1.0. After update the glibc version, occured a another issue. I abstract the log as follows:
14/09/25 11:29:18 WARN [org.apache.hadoop.util.NativeCodeLoader---main]: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/09/25 11:29:19 WARN [org.apache.hadoop.hdfs.DomainSocketFactory---main]: The short-circuit local reads feature is disabled because libhadoop cannot be loaded. WARN [org.apache.spark.scheduler.TaskSetManager---Result resolver thread-0]: Lost task 0.0 in stage 1.0 (TID 1, spark-dev134): org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:236) org.xerial.snappy.Snappy.<clinit>(Snappy.java:48) org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351) WARN [org.apache.spark.scheduler.TaskSetManager---Result resolver thread-3]: Lost task 4.0 in stage 1.0 (TID 4, spark-dev134): java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351) org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:159) org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142) 14/09/25 11:29:24 ERROR [org.apache.spark.network.ConnectionManager---handle-read-write-executor-3]: Corresponding SendingConnection to ConnectionManagerId(spark-dev135,38649) not found 14/09/25 11:29:24 INFO [org.apache.spark.scheduler.DAGScheduler---main]: Failed to run count at SessionSVD2.scala:23 Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 9, spark-dev135): ExecutorLostFailure (executor lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 1)I tried to print JAVA_LIBRARY_PATH ,native-hadoop library is not in the path,and I set the System.setProperty("JAVA_LIBRARY_PATH","hadoop_home/lib/native/"),which only effect in System.getenv(),but not the System.getProperty("JAVA_LIBRARY_PATH") .And "hadoop_home/lib/native/" contain libhadoop libsnappy.so file,whcih I want to include in path. 2)I found in /tmp there are many snappy-uuuid file,each time i submit a job it create a snappy-uuuid file. Before I update the glibc version,my fellow update the snappy version,I think this is the reason why it can find the snappy file but libhadoop. Is there any ideas? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Update-gcc-version-Still-snappy-error-tp15137.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org