You should not use SparkContext or RDD directly in your closures. Could you show the codes of "method1"? Maybe you only needs join or something else. E.g.,
val cassandraRDD = sc.cassandraTable("keySpace", "tableName") reRDD.join(cassandraRDD).map(....).saveAsTextFile(outputDir) On Tue, Jan 19, 2016 at 4:12 AM, Ricardo Paiva <ricardo.pa...@corp.globo.com > wrote: > Did you try SparkContext.getOrCreate() ? > > You don't need to pass the sparkContext to the map function, you can > retrieve it from the SparkContext singleton. > > Regards, > > Ricardo > > > On Mon, Jan 18, 2016 at 6:29 PM, gpatcham [via Apache Spark User List] > <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=26006&i=0>> wrote: > >> Hi, >> >> I have a use case where I need to pass sparkcontext in map function >> >> reRDD.map(row =>method1(row,sc)).saveAsTextFile(outputDir) >> >> Method1 needs spark context to query cassandra. But I see below error >> >> java.io.NotSerializableException: org.apache.spark.SparkContext >> >> Is there a way we can fix this ? >> >> Thanks >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/using-spark-context-in-map-funciton-TASk-not-serilizable-error-tp25998.html >> To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=26006&i=1> >> To unsubscribe from Apache Spark User List, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > > -- > Ricardo Paiva > Big Data > *globo.com* <http://www.globo.com> > > ------------------------------ > View this message in context: Re: using spark context in map funciton > TASk not serilizable error > <http://apache-spark-user-list.1001560.n3.nabble.com/using-spark-context-in-map-funciton-TASk-not-serilizable-error-tp25998p26006.html> > > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >