[ https://issues.apache.org/jira/browse/SPARK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-20252. ------------------------------- Resolution: Duplicate This is basically a limitation of how the shell and classloaders work. Simpler variants on this work as do compiled apps. (Also it's not necessarily going to work to stop and restart the context.) > java.lang.ClassNotFoundException: $line22.$read$$iwC$$iwC$movie_row > ------------------------------------------------------------------- > > Key: SPARK-20252 > URL: https://issues.apache.org/jira/browse/SPARK-20252 > Project: Spark > Issue Type: Bug > Components: Spark Shell > Affects Versions: 1.6.3 > Environment: Datastax DSE dual node SPARK cluster > [cqlsh 5.0.1 | Cassandra 3.0.12.1586 | DSE 5.0.7 | CQL spec 3.4.0 | Native > protocol v4] > Reporter: Peter Mead > > After starting a spark shell using DSE -u xxxxxxxx -p xxxxxxxxx spark > scala> case class movie_row (actor: String, character_name: String, video_id: > java.util.UUID, video_year: Int, title: String) > defined class movie_row > scala> val vids=sc.cassandraTable("hcl","videos_by_actor") > vids: > com.datastax.spark.connector.rdd.CassandraTableScanRDD[com.datastax.spark.connector.CassandraRow] > = CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:15 > scala> val vids=sc.cassandraTable[movie_row]("hcl","videos_by_actor") > vids: com.datastax.spark.connector.rdd.CassandraTableScanRDD[movie_row] = > CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15 > scala> vids.count > res0: Long = 114961 > Works OK!! > BUT if the spark context is stopped and recreated THEN: > scala> sc.stop() > scala> import org.apache.spark.SparkContext, org.apache.spark.SparkContext._, > org.apache.spark.SparkConf > import org.apache.spark.SparkContext > import org.apache.spark.SparkContext._ > import org.apache.spark.SparkConf > scala> :paste > // Entering paste mode (ctrl-D to finish) > val conf = new SparkConf(true) > .set("spark.cassandra.connection.host", "redacted") > .set("spark.cassandra.auth.username", "redacted") > .set("spark.cassandra.auth.password", "redacted") > // Exiting paste mode, now interpreting. > conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@e207342 > scala> val sc = new SparkContext("spark://192.168.1.84:7077", "pjm", conf) > sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@12b8b7c8 > scala> case class movie_row (actor: String, character_name: String, video_id: > java.util.UUID, video_year: Int, title: String) > defined class movie_row > scala> val vids=sc.cassandraTable[movie_row]("hcl","videos_by_actor") > vids: com.datastax.spark.connector.rdd.CassandraTableScanRDD[movie_row] = > CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:15 > scala> vids.count > [Stage 0:> (0 + 2) / > 2]WARN 2017-04-07 12:52:03,277 org.apache.spark.scheduler.TaskSetManager: > Lost task 0.0 in stage 0.0 (TID 0, cassandra183): > java.lang.ClassNotFoundException: $line22.$read$$iwC$$iwC$movie_row > FAILS!! > I have been unable to get this to work from a remote SPARK shell! -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org