Hi, I have a file called out with random numbers where each number in on one line in the file. I am loading the complete file into a RDD and I want to create partitions with the help of coalesce function. This is my code snippet. import scala.math.Ordered import org.apache.spark.rdd.CoalescedRDD import org.apache.spark.api.java.JavaPairRDD import org.apache.spark.api.java.JavaSparkContext import org.apache.spark.api.java.function.Function import org.apache.spark.api.java.function.PairFunction
val dRDD = sc.textFile("hdfs://master:54310/out",10) val keyval=dRDD.coalesce(100,true).mapPartitionsWithIndex{(ind,iter) => iter.map(x => process(ind,x.trim().split(' ').map(_.toDouble),q,m,r))} However, I am getting this error. I tried looking at various other links but I always got this error. Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.coalesce(IZ)Lorg/apache/spark/rdd/RDD; at SimpleApp$.main(SimpleApp.scala:432) at SimpleApp.main(SimpleApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) A follow up, is there any way I can access the elements in a RDD with the help of an index and create partitions? Like for example I have a RDD with values like 1,2,3,...,100. I would like to create partitions which look like this: part1: 1,2,3..,10 part2: 8,9,10,...,20 part3: 18,19,20,...,30 and so on... Thanks and regards, Madhura -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Need-help-with-coalesce-tp10243.html Sent from the Apache Spark User List mailing list archive at Nabble.com.