hi: appendix are the full error message I try to modify dataframe row in non-sql way and get Task not serializable,my test procedure like follows: 1.val df=cc.sql(select * from t1) 2.def function1 (iterator: Iterator[Row]):Iterator[Row]={ var list=scala.collection.mutable.ListBuffer[Row]() while (iterator.hasNext) { var r=iterator.next if(r.getAs[String]("col1").toString.equalsIgnoreCase(r.getAs[String]("col2").toString)) list+=r} list.iterator } 3.df.mapPartitions(r=>function1 (r)) I also apply function1 to dataframe generated from sqlContext work fine, so i believe carboncontext is refering some outer variables that are not serializable.
sunerhan1...@sina.com