I'm not sure what your use case is, but perhaps you could use mapPartitions
to reduce across the individual partitions and apply your filtering. Then
you can finish with a reduceByKey.

On Thu, Feb 19, 2015 at 9:21 AM, Debasish Das <[email protected]>
wrote:

> Hi,
>
> Before I send out the keys for network shuffle, in reduceByKey after map +
> combine are done, I would like to filter the keys based on some threshold...
>
> Is there a way to get the key, value after map+combine stages so that I
> can run a filter on the keys ?
>
> Thanks.
> Deb
>



-- 
Daniel Siegmann, Software Developer
Velos
Accelerating Machine Learning

54 W 40th St, New York, NY 10018
E: [email protected] W: www.velos.io

Reply via email to