It is happening on random executors on random nodes. Not on any specific node everytime. Or not happening at all
On Thu, Jan 28, 2016 at 7:42 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Did the UnsupportedOperationException's happen from the executors on all the > nodes or only one node ? > > Thanks > > On Thu, Jan 28, 2016 at 5:13 PM, Khusro Siddiqui <mkhu...@gmail.com> > wrote: > >> Hi Everyone, >> >> Environment used: Datastax Enterprise 4.8.3 which is bundled with Spark >> 1.4.1 and scala 2.10.5. >> >> I am using Dataframes to query Cassandra, do processing and store the >> result back into Cassandra. The job is being submitted using spark-submit >> on a cluster of 3 nodes. While doing so I get three WARN messages: >> >> WARN 2016-01-28 19:08:18 org.apache.spark.scheduler.TaskSetManager: Lost >> task 99.0 in stage 2.0 (TID 107, 10.2.1.82): java.io.InvalidClassException: >> org.apache.spark.sql.types.TimestampType$; unable to create instance >> >> Caused by: java.lang.reflect.InvocationTargetException >> >> Caused by: java.lang.UnsupportedOperationException: tail of empty list >> >> >> For example, if I am running the same job, for the same input set of >> data, say 20 times, >> >> - 11 times it will run successfully without any WARN messages >> >> - 4 times it will run successfully with the above messages >> >> - 6 times it will run successfully by randomly giving one or two of >> the exceptions above >> >> >> In all the 20 runs, the output data is coming as expected and there is no >> error in that. My concern is, why is it not giving these messages every >> time I do a spark-submit but only at times. Also, the stack trace does not >> point to any specific point in my line of code. Full stack trace is as >> follows. Please let me know if you need any other information >> >> >> WARN 2016-01-28 19:08:24 org.apache.spark.scheduler.TaskSetManager: Lost >> task 188.0 in stage 16.0 (TID 637, 10.2.1.82): >> java.io.InvalidClassException: org.apache.spark.sql.types.TimestampType$; >> unable to create instance >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1788) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> >> at >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69) >> >> at >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95) >> >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:70) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.lang.reflect.InvocationTargetException >> >> at >> sun.reflect.GeneratedSerializationConstructorAccessor89.newInstance(Unknown >> Source) >> >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> >> at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:967) >> >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1785) >> >> ... 109 more >> >> Caused by: java.lang.UnsupportedOperationException: tail of empty list >> >> at scala.collection.immutable.Nil$.tail(List.scala:339) >> >> at scala.collection.immutable.Nil$.tail(List.scala:334) >> >> at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172) >> >> at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1491) >> >> at scala.reflect.internal.Types$NoArgsTypeRef.typeParams(Types.scala:2144) >> >> at >> scala.reflect.internal.Types$TypeRef.initializedTypeParams(Types.scala:2408) >> >> at >> scala.reflect.internal.Types$TypeRef.typeParamsMatchArgs(Types.scala:2409) >> >> at >> scala.reflect.internal.Types$AliasTypeRef$class.dealias(Types.scala:2232) >> >> at scala.reflect.internal.Types$TypeRef$$anon$3.dealias(Types.scala:2539) >> >> at >> scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1259) >> >> at >> scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:202) >> >> at >> scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:65) >> >> at org.apache.spark.sql.types.AtomicType.<init>(DataType.scala:96) >> >> at org.apache.spark.sql.types.TimestampType.<init>(TimestampType.scala:37) >> > >
