Using a single SparkContext should not cause this problem. In the SQL tests we use TestSQLContext and TestHive which are global singletons for all of our unit testing.
On Mon, Dec 15, 2014 at 1:27 PM, Marius Soutier <mps....@gmail.com> wrote: > > Possible, yes, although I’m trying everything I can to prevent it, i.e. > fork in Test := true and isolated. Can you confirm that reusing a single > SparkContext for multiple tests poses a problem as well? > > Other than that, just switching from SQLContext to HiveContext also > provoked the error. > > > On 15.12.2014, at 20:22, Michael Armbrust <mich...@databricks.com> wrote: > > Is it possible that you are starting more than one SparkContext in a > single JVM with out stopping previous ones? I'd try testing with Spark > 1.2, which will throw an exception in this case. > > On Mon, Dec 15, 2014 at 8:48 AM, Marius Soutier <mps....@gmail.com> wrote: >> >> Hi, >> >> I’m seeing strange, random errors when running unit tests for my Spark >> jobs. In this particular case I’m using Spark SQL to read and write Parquet >> files, and one error that I keep running into is this one: >> >> org.apache.spark.SparkException: Job aborted due to stage failure: Task >> 19 in stage 6.0 failed 1 times, most recent failure: Lost task 19.0 in >> stage 6.0 (TID 223, localhost): java.io.IOException: PARSING_ERROR(2) >> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) >> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) >> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) >> >> I can only prevent this from happening by using isolated Specs tests >> thats always create a new SparkContext that is not shared between tests >> (but there can also be only a single SparkContext per test), and also by >> using standard SQLContext instead of HiveContext. It does not seem to have >> anything to do with the actual files that I also create during the test run >> with SQLContext.saveAsParquetFile. >> >> >> Cheers >> - Marius >> >> >> PS The full trace: >> >> org.apache.spark.SparkException: Job aborted due to stage failure: Task >> 19 in stage 6.0 failed 1 times, most recent failure: Lost task 19.0 in >> stage 6.0 (TID 223, localhost): java.io.IOException: PARSING_ERROR(2) >> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) >> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) >> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) >> >> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) >> >> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) >> >> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58) >> >> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) >> >> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) >> >> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readObject$1.apply$mcV$sp(TorrentBroadcast.scala:169) >> org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:927) >> >> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:155) >> sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> java.lang.reflect.Method.invoke(Method.java:606) >> >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >> >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >> >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) >> >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:160) >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> java.lang.Thread.run(Thread.java:745) >> Driver stacktrace: >> at org.apache.spark.scheduler.DAGScheduler.org >> <http://org.apache.spark.scheduler.dagscheduler.org/>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) >> ~[spark-core_2.10-1.1.1.jar:1.1.1] >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) >> ~[spark-core_2.10-1.1.1.jar:1.1.1] >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) >> ~[spark-core_2.10-1.1.1.jar:1.1.1] >> at >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >> ~[scala-library.jar:na] >> at >> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >> ~[scala-library.jar:na] >> at >> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173) >> ~[spark-core_2.10-1.1.1.jar:1.1.1] >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
