While running a my Spark Application over 1.3.0 with Scala 2.10.0 i
encountered
15/04/21 09:13:21 ERROR executor.Executor: Exception in task 7.0 in stage
2.0 (TID 28)
java.lang.UnsupportedOperationException: tail of empty list
at scala.collection.immutable.Nil$.tail(List.scala:339)
at scala.collection.immutable.Nil$.tail(List.scala:334)
at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172)
My app has quite a lot of reflection usage as i need to build avro schema.
I was told on stack overflow that this error is because reflection is not
thread safe in scala 2.10 and was advised to use scala 2.11
Hence I moved to scala 2.11 and moved all spark maven dependencies to 2.11
(spark-avro is still 2.10 as 2.11 is not available). I ran into this
15/04/21 09:13:21 ERROR executor.Executor: Exception in task 7.0 in stage
2.0 (TID 28)
java.lang.UnsupportedOperationException: tail of empty list
at scala.collection.immutable.Nil$.tail(List.scala:339)
at scala.collection.immutable.Nil$.tail(List.scala:334)
at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172)
Code:
args.map {
option =>
{
val pairs = option.split("=")
arguments += pairs(0) -> pairs(1)
}
}
Error is at ->. Does Spark use scala libs 2.10 at runtime ? Any suggestions
for fix ?
--
Deepak