using sparkContext from within a map function (from spark streaming app)

Hi, We are designing a solution which pulls file paths from Kafka and for the current stage just counts the lines in each of these files. When running the code it fails on: Exception in thread main org.apache.spark.SparkException: Task not serializable at

Re: using sparkContext from within a map function (from spark streaming app)

Yes, you can never use the SparkContext inside a remote function. It is on the driver only. On Sun, Mar 8, 2015 at 4:22 PM, Daniel Haviv daniel.ha...@veracity-group.com wrote: Hi, We are designing a solution which pulls file paths from Kafka and for the current stage just counts the lines in