The problem is that your new mapped values may be in the wrong partition, according to your partitioner. Look for methods that have a preservesPartitioning flag, which is a way to indicate that you know the partitioning remains correct. (Like, you partition by keys and didn't change the keys in mapping)
On Mon, Feb 22, 2016 at 6:21 PM, Brian London <brianmlon...@gmail.com> wrote: > It appears that when a custom partitioner is applied in a groupBy operation, > it is not propagated through subsequent non-shuffle operations. Is this > intentional? Is there any way to carry custom partitioning through maps? > > I've uploaded a gist that exhibits the behavior. > https://gist.github.com/BrianLondon/c3c3355d1971971f3ec6 --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
