Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

2014-11-26 Thread lokeshkumar 
Hi Sean

Thanks for reply,
We upgraded our spark cluster from 1.1.0 to 1.2.0.
And we also thought that this issue might be due to mis matching spark jar
versions.
But we double checked and re installed our app completely in a new system
with spark-1.2.0 distro, but still no result.
Facing the same problem.

This does not happen when master is set to 'local[*]'.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-latest-1-2-0-build-ClassCastException-from-B-to-SerializableWritable-tp19824p19864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

2014-11-26 Thread Sean Owen 
I'll take a wild guess that you have mismatching versions of Spark at
play. Your cluster has one build and you're accidentally including
another version.

I think this code path has changed recently (
https://github.com/apache/spark/commit/7e63bb49c526c3f872619ae14e4b5273f4c535e9#diff-83eb37f7b0ebed3c14ccb7bff0d577c2
?) but shouldn't cause this kind of problem per se, I think. Maybe
double-check the version mismatch possibility first.

On Wed, Nov 26, 2014 at 2:42 AM, lokeshkumar  wrote:
> Hello forum,
>
> We are using spark distro built from the source of latest 1.2.0 tag.
> And we are facing the below issue, while trying to act upon the JavaRDD
> instance, the stacktrace is given below.
> Can anyone please let me know, what can be wrong here?
>
> java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable
> at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:138)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.RDD.take(RDD.scala:1060)
> at 
> org.apache.spark.api.java.JavaRDDLike$class.take(JavaRDDLike.scala:419)
> at org.apache.spark.api.java.JavaRDD.take(JavaRDD.scala:32)
> at
> com.dataken.common.chores.InformationDataLoadChore.run(InformationDataLoadChore.java:69)
> at com.dataken.common.pipeline.DatakenTask.start(DatakenTask.java:110)
> at
> com.dataken.tasks.objectcentricprocessor.ObjectCentricProcessTask.execute(ObjectCentricProcessTask.java:99)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2014-11-26 08:07:38,454 ERROR [DefaultQuartzScheduler_Worker-2]
> org.quartz.core.ErrorLogger
> Job (report_report.report_report threw an exception.
>
> org.quartz.SchedulerException: Job threw an unhandled exception. [See nested
> exception: java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable]
> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable
> at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:138)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
> at org.apache.spark.rdd.RDD.take(RDD.scala:1060)
> at 
> org.apache.spark.api.java.JavaRDDLike$class.take(JavaRDDLike.scala:419)
> at org.apache.spark.api.java.JavaRDD.take(JavaRDD.scala:32)
> at
> com.dataken.common.chores.InformationDataLoadChore.run(InformationDataLoadChore.java:69)
> at com.dataken.common.pipeline.DatakenTask.start(DatakenTask.java:110)
> at
> com.dataken.tasks.objectcentricprocessor.ObjectCentricProcessTask.execute(ObjectCentricProcessTask.java:99)
> at org.quartz.core.JobRunS

Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

2014-11-26 Thread lokeshkumar 
The above issue happens while trying to do the below activity on JavaRDD
(calling take() on rdd)

JavaRDD loadedRDD = sc.textFile(...);
String[] tokens = loadedRDD.take(1).get(0).split(",");





--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-latest-1-2-0-build-ClassCastException-from-B-to-SerializableWritable-tp19824p19859.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org