If you don't really care about the batchedDegree, but rather just want to
do operations over some set of elements rather than one at a time, then
just use mapPartitions().
Otherwise, if you really do want certain sized batches and you are able to
relax the constraints slightly, is to construct these batches within each
partition. For instance:
val batchedRDD = rdd.mapPartitions { iter: Iterator[Int] =>
new Iterator[Array[Int]] {
def hasNext: Boolean = iter.hasNext
def next(): Array[Int] = {
iter.take(batchedDegree).toArray
}
}
}
This function is efficient in that it does not load the entire partition
into memory, just enough to construct each batch. However, there will be
one smaller batch at the end of each partition (rather than just one over
the entire dataset).
On Sat, Jul 12, 2014 at 6:03 PM, Parthus <peng.wei....@gmail.com> wrote:
> Hi there,
>
> I have a bunch of data in a RDD, which I processed it one by one
> previously.
> For example, there was a RDD denoted by "data: RDD[Object]" and then I
> processed it using "data.map(...)". However, I got a new requirement to
> process the data in a patched way. It means that I need to convert the RDD
> from RDD[Object] to RDD[Array[Object]] and then process it, which is to
> fill
> out this function: def convert2array(inputs: RDD[Object], batchedDegree:
> Int): RDD[Array[Object]] = {...}.
>
> I hope that after the conversion, each element of the new RDD is an array
> of
> the previous RDD elements. The parameter "batchedDegree" specifies how many
> elements are batched together. For example, if I have 210 elements in the
> previous RDD, the result of the conversion functions should be a RDD with 3
> elements. Each element is an array, and the first two arrays contains 1~100
> and 101~200 elements. The third element contains 201~210 elements.
>
> I was wondering if anybody could help me complete this scala function with
> an efficient way. Thanks a lot.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Convert-from-RDD-Object-to-RDD-Array-Object-tp9530.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
