Would be great if you can share the piece of code happening inside your mapPartition, I'm assuming you are creating/handling a lot of Complex objects and hence it slows down the performance. Here's a link <http://spark.apache.org/docs/latest/tuning.html> to performance tuning if you haven't seen it already.
Thanks Best Regards On Wed, Dec 31, 2014 at 8:45 AM, Raghavendra Pandey < [email protected]> wrote: > I have a spark app that involves series of mapPartition operations and > then a keyBy operation. I have measured the time inside mapPartition > function block. These blocks take trivial time. Still the application takes > way too much time and even sparkUI shows that much time. > So i was wondering where does it take time and how can I reduce this. > > Thanks > Raghavendra >
