No for me as well it is non-deterministic. It happens in a piece of code that does many filter and counts on a small set of records (~1k-10k). The originally set is persisted in memory and we have a Kryo serializer set for it. The task itself takes in just a few filtering parameters. This with the same setting has sometimes completed to sucess and sometimes failed during this step.
Arun On Fri, Sep 26, 2014 at 1:32 PM, Brad Miller <bmill...@eecs.berkeley.edu> wrote: > I've had multiple jobs crash due to "java.io.IOException: unexpected > exception type"; I've been running the 1.1 branch for some time and am now > running the 1.1 release binaries. Note that I only use PySpark. I haven't > kept detailed notes or the tracebacks around since there are other problems > that have caused my greater grief (namely "key not found" errors). > > For me the exception seems to occur non-deterministically, which is a bit > interesting since the error message shows that the same stage has failed > multiple times. Are you able to consistently re-produce the bug across > multiple invocations at the same place? > > On Fri, Sep 26, 2014 at 6:11 AM, Arun Ahuja <aahuj...@gmail.com> wrote: > >> Has anyone else seen this erorr in task deserialization? The task is >> processing a small amount of data and doesn't seem to have much data >> hanging to the closure? I've only seen this with Spark 1.1 >> >> Job aborted due to stage failure: Task 975 in stage 8.0 failed 4 times, most >> recent failure: Lost task 975.3 in stage 8.0 (TID 24777, host.com): >> java.io.IOException: unexpected exception type >> >> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) >> >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025) >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >> >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> java.lang.Thread.run(Thread.java:744) >> >> >
