ect: "dynamically" sort a large collection?
Hey,
I need to scan a large "key-value" collection as below:
1) sort it on an attribute of “value”
2) scan it one by one, from element with largest value
2.1) if the current element matches a pre-defined condition, its value will be
Hey,
I need to scan a large "key-value" collection as below:
1) sort it on an attribute of “value”
2) scan it one by one, from element with largest value
2.1) if the current element matches a pre-defined condition, its value will be
reduced and the element will be inserted back to collection.
rtedRdd.flatMapValues( v =>
> If (v % 2 == 0) {
> Some(v / 2)
> } else {
> None
> }
> )
>
> Then you need to sort again.
>
> -adrian
>
> From: Yifan LI
> Date: Monday, October 12, 2015 at 1:03 PM
> To: spark users
> Subject: "dynamically" sor
.@adobe.com>> wrote:
>>
>> I think you’re looking for the flatMap (or flatMapValues) operator – you can
>> do something like
>>
>> sortedRdd.flatMapValues( v =>
>> If (v % 2 == 0) {
>> Some(v / 2)
>> } else {
>> None
>> }
>> )
>>
>> T
