It's a known issue. See https://issues.apache.org/jira/browse/SPARK-10719
On Thu, Jan 28, 2016 at 5:44 PM, Khusro Siddiqui <mkhu...@gmail.com> wrote: > It is happening on random executors on random nodes. Not on any specific > node everytime. > Or not happening at all > > On Thu, Jan 28, 2016 at 7:42 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Did the UnsupportedOperationException's happen from the executors on all the >> nodes or only one node ? >> >> Thanks >> >> On Thu, Jan 28, 2016 at 5:13 PM, Khusro Siddiqui <mkhu...@gmail.com> >> wrote: >> >>> Hi Everyone, >>> >>> Environment used: Datastax Enterprise 4.8.3 which is bundled with Spark >>> 1.4.1 and scala 2.10.5. >>> >>> I am using Dataframes to query Cassandra, do processing and store the >>> result back into Cassandra. The job is being submitted using spark-submit >>> on a cluster of 3 nodes. While doing so I get three WARN messages: >>> >>> WARN 2016-01-28 19:08:18 org.apache.spark.scheduler.TaskSetManager: >>> Lost task 99.0 in stage 2.0 (TID 107, 10.2.1.82): >>> java.io.InvalidClassException: org.apache.spark.sql.types.TimestampType$; >>> unable to create instance >>> >>> Caused by: java.lang.reflect.InvocationTargetException >>> >>> Caused by: java.lang.UnsupportedOperationException: tail of empty list >>> >>> >>> For example, if I am running the same job, for the same input set of >>> data, say 20 times, >>> >>> - 11 times it will run successfully without any WARN messages >>> >>> - 4 times it will run successfully with the above messages >>> >>> - 6 times it will run successfully by randomly giving one or two of >>> the exceptions above >>> >>> >>> In all the 20 runs, the output data is coming as expected and there is >>> no error in that. My concern is, why is it not giving these messages every >>> time I do a spark-submit but only at times. Also, the stack trace does not >>> point to any specific point in my line of code. Full stack trace is as >>> follows. Please let me know if you need any other information >>> >>> >>> WARN 2016-01-28 19:08:24 org.apache.spark.scheduler.TaskSetManager: >>> Lost task 188.0 in stage 16.0 (TID 637, 10.2.1.82): >>> java.io.InvalidClassException: org.apache.spark.sql.types.TimestampType$; >>> unable to create instance >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1788) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> >>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> >>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> >>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> >>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> >>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) >>> >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>> >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> >>> at >>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69) >>> >>> at >>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95) >>> >>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:70) >>> >>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Caused by: java.lang.reflect.InvocationTargetException >>> >>> at >>> sun.reflect.GeneratedSerializationConstructorAccessor89.newInstance(Unknown >>> Source) >>> >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> >>> at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:967) >>> >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1785) >>> >>> ... 109 more >>> >>> Caused by: java.lang.UnsupportedOperationException: tail of empty list >>> >>> at scala.collection.immutable.Nil$.tail(List.scala:339) >>> >>> at scala.collection.immutable.Nil$.tail(List.scala:334) >>> >>> at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172) >>> >>> at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1491) >>> >>> at >>> scala.reflect.internal.Types$NoArgsTypeRef.typeParams(Types.scala:2144) >>> >>> at >>> scala.reflect.internal.Types$TypeRef.initializedTypeParams(Types.scala:2408) >>> >>> at >>> scala.reflect.internal.Types$TypeRef.typeParamsMatchArgs(Types.scala:2409) >>> >>> at >>> scala.reflect.internal.Types$AliasTypeRef$class.dealias(Types.scala:2232) >>> >>> at scala.reflect.internal.Types$TypeRef$$anon$3.dealias(Types.scala:2539) >>> >>> at >>> scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1259) >>> >>> at >>> scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:202) >>> >>> at >>> scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:65) >>> >>> at org.apache.spark.sql.types.AtomicType.<init>(DataType.scala:96) >>> >>> at >>> org.apache.spark.sql.types.TimestampType.<init>(TimestampType.scala:37) >>> >> >> >
