Did the UnsupportedOperationException's happen from the executors on all the nodes or only one node ?
Thanks On Thu, Jan 28, 2016 at 5:13 PM, Khusro Siddiqui <mkhu...@gmail.com> wrote: > Hi Everyone, > > Environment used: Datastax Enterprise 4.8.3 which is bundled with Spark > 1.4.1 and scala 2.10.5. > > I am using Dataframes to query Cassandra, do processing and store the > result back into Cassandra. The job is being submitted using spark-submit > on a cluster of 3 nodes. While doing so I get three WARN messages: > > WARN 2016-01-28 19:08:18 org.apache.spark.scheduler.TaskSetManager: Lost > task 99.0 in stage 2.0 (TID 107, 10.2.1.82): java.io.InvalidClassException: > org.apache.spark.sql.types.TimestampType$; unable to create instance > > Caused by: java.lang.reflect.InvocationTargetException > > Caused by: java.lang.UnsupportedOperationException: tail of empty list > > > For example, if I am running the same job, for the same input set of data, > say 20 times, > > - 11 times it will run successfully without any WARN messages > > - 4 times it will run successfully with the above messages > > - 6 times it will run successfully by randomly giving one or two of > the exceptions above > > > In all the 20 runs, the output data is coming as expected and there is no > error in that. My concern is, why is it not giving these messages every > time I do a spark-submit but only at times. Also, the stack trace does not > point to any specific point in my line of code. Full stack trace is as > follows. Please let me know if you need any other information > > > WARN 2016-01-28 19:08:24 org.apache.spark.scheduler.TaskSetManager: Lost > task 188.0 in stage 16.0 (TID 637, 10.2.1.82): > java.io.InvalidClassException: org.apache.spark.sql.types.TimestampType$; > unable to create instance > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1788) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at scala.collection.immutable.$colon$colon.readObject(List.scala:362) > > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at scala.collection.immutable.$colon$colon.readObject(List.scala:362) > > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at scala.collection.immutable.$colon$colon.readObject(List.scala:362) > > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at scala.collection.immutable.$colon$colon.readObject(List.scala:362) > > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at scala.collection.immutable.$colon$colon.readObject(List.scala:362) > > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69) > > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95) > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) > > at org.apache.spark.scheduler.Task.run(Task.scala:70) > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.reflect.InvocationTargetException > > at > sun.reflect.GeneratedSerializationConstructorAccessor89.newInstance(Unknown > Source) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > > at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:967) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1785) > > ... 109 more > > Caused by: java.lang.UnsupportedOperationException: tail of empty list > > at scala.collection.immutable.Nil$.tail(List.scala:339) > > at scala.collection.immutable.Nil$.tail(List.scala:334) > > at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172) > > at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1491) > > at scala.reflect.internal.Types$NoArgsTypeRef.typeParams(Types.scala:2144) > > at > scala.reflect.internal.Types$TypeRef.initializedTypeParams(Types.scala:2408) > > at > scala.reflect.internal.Types$TypeRef.typeParamsMatchArgs(Types.scala:2409) > > at > scala.reflect.internal.Types$AliasTypeRef$class.dealias(Types.scala:2232) > > at scala.reflect.internal.Types$TypeRef$$anon$3.dealias(Types.scala:2539) > > at > scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1259) > > at > scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:202) > > at > scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:65) > > at org.apache.spark.sql.types.AtomicType.<init>(DataType.scala:96) > > at org.apache.spark.sql.types.TimestampType.<init>(TimestampType.scala:37) >
