And if you can relax your constraints even further to only require
RDD[List[Int]], then it's even simpler:
rdd.mapPartitions(_.grouped(batchedDegree))
On Sat, Jul 12, 2014 at 6:26 PM, Aaron Davidson <ilike...@gmail.com> wrote:
> If you don't really care about the batchedDegree, but rather just want to
> do operations over some set of elements rather than one at a time, then
> just use mapPartitions().
>
> Otherwise, if you really do want certain sized batches and you are able to
> relax the constraints slightly, is to construct these batches within each
> partition. For instance:
>
> val batchedRDD = rdd.mapPartitions { iter: Iterator[Int] =>
> new Iterator[Array[Int]] {
> def hasNext: Boolean = iter.hasNext
> def next(): Array[Int] = {
> iter.take(batchedDegree).toArray
> }
> }
> }
>
> This function is efficient in that it does not load the entire partition
> into memory, just enough to construct each batch. However, there will be
> one smaller batch at the end of each partition (rather than just one over
> the entire dataset).
>
>
>
> On Sat, Jul 12, 2014 at 6:03 PM, Parthus <peng.wei....@gmail.com> wrote:
>
>> Hi there,
>>
>> I have a bunch of data in a RDD, which I processed it one by one
>> previously.
>> For example, there was a RDD denoted by "data: RDD[Object]" and then I
>> processed it using "data.map(...)". However, I got a new requirement to
>> process the data in a patched way. It means that I need to convert the RDD
>> from RDD[Object] to RDD[Array[Object]] and then process it, which is to
>> fill
>> out this function: def convert2array(inputs: RDD[Object], batchedDegree:
>> Int): RDD[Array[Object]] = {...}.
>>
>> I hope that after the conversion, each element of the new RDD is an array
>> of
>> the previous RDD elements. The parameter "batchedDegree" specifies how
>> many
>> elements are batched together. For example, if I have 210 elements in the
>> previous RDD, the result of the conversion functions should be a RDD with
>> 3
>> elements. Each element is an array, and the first two arrays contains
>> 1~100
>> and 101~200 elements. The third element contains 201~210 elements.
>>
>> I was wondering if anybody could help me complete this scala function with
>> an efficient way. Thanks a lot.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Convert-from-RDD-Object-to-RDD-Array-Object-tp9530.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
