I want to call a function for batches of elements from an rdd val javaClass:org.apache.spark.api.java.function.Function[Seq[String],Unit] = new JavaClass() rdd.mapPartitions(_.grouped(5)).foreach(javaClass)
1.This worked fine in spark 0.9.1 , when we upgrade to spark 1.0.2 , Function changed from class to interface and we get : type mismatch; found : org.apache.spark.api.java.function.Function[Seq[String],Unit] required: Seq[String] => Unit We are using Java 1.7 We use that class for one time initialization method call on each executor and for batch processing . 2. Previously on hadoop by RecordWriter.close() we get a callback method for every executor that processed map/reduce operations. We would like this in spark too , is it possible? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-functionality-similar-to-hadoop-s-RecordWriter-close-method-tp13795.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
