Hi,
I have Terasort being executed on spark1.4.1 with hadoop 2.7 for a
datasize of 1TB. When i change my os user from spark1 to hduser, i am
observing below exception. Please let me know what is wrong here. I tried
to update scala-2.10 to 2.11 and compiled Terasort scala code using sbt.
But nothing is helping here. I had created the spark1 (spark1 to spark4)
user for a multi-tenancy run.
2015-10-13 03:42:54,840 [sparkDriver-akka.actor.default-dispatcher-2] INFO
org.apache.spark.scheduler.TaskSetManager - Starting task 199.0 in stage
0.0 (TID 199, 9.37.251.65, ANY, 1544 bytes)
2015-10-13 03:42:54,843 [task-result-getter-2] WARN
org.apache.spark.scheduler.TaskSetManager - Lost task 173.0 in stage 0.0
(TID 173, 9.37.251.65): java.io.InvalidClassException:
scala.reflect.ClassTag$$anon$1; local class incompatible: stream classdesc
serialVersionUID = -4937928798201944954, local class serialVersionUID =
-8102093212602380348
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Any help in this regard would be of great help.
Regards
Shreeharsha GN
