Spark-cassandra-connector 1.0.0-rc5: java.io.NotSerializableException

[Shing Hing Man]

Hi,

My version of Spark is 1.0.2.
I am trying to use Spark-cassandra-connector to execute an update csql 
statement inside 
an CassandraConnector(conf).withSessionDo  block :

CassandraConnector(conf).withSessionDo {
      session =>
        {
          myRdd.foreach {
            case (ip, values) =>             
              session.execute( {An csql update statement})
          }

        }

    }
    
 The above code is in the main method of the driver object. 
 When I run the above code (in local mode), I get an exception :
 
  java.io.NotSerializableException: 
com.datastax.spark.connector.cql.SessionProxy
  
 Exception in thread "main" org.apache.spark.SparkException: Job aborted due to 
stage failure: Task not serializable: java.io.NotSerializableException: 
com.datastax.spark.connector.cql.SessionProxy
    at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
    at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
    at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
    at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
    at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:772)
    at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:906)
    at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply(DAGScheduler.scala:903)
    at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply(DAGScheduler.scala:903)
  
  
Is there a way to use an RDD inside an   CassandraConnector(conf).withSessionDo 
 block  ?

Thanks in advance for any assistance !
 
Shing

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org