Each RDD is composed of multiple blocks known as partitions, when you apply transformation over it, then it can grow in size depending on the operation (as the # objects/references increase) and that is probably the reason why you are seeing increased number of partitions.
I don't think increased number of partitions will cause any performance decrease as it helps to evenly distribute the tasks across machines and per core. If you don't want more partitions, then you can do a .repartition over the transformed RDD. Custom partitioner can improve the performance depending on the usecase that you are having. Thanks Best Regards On Fri, Mar 27, 2015 at 4:39 PM, sayantini <sayantiniba...@gmail.com> wrote: > In our application where we load our historical data in 40 partitioned > RDDs (no. of available cores X 2) and we have not implemented any custom > partitioner. > > After applying transformations on these RDDs intermediate RDDs are created > which have partitions greater than 40 and sometimes partitions are going up > till 300. > > 1. Is Spark intelligent enough to manage the partitions of RDD? Please > suggest why there is an increase in the no. of partitions? > > 2. We suspect that increasing the no. of partitions is causing decrease in > performance. > > 3. If we create a custom Partitioner will it improve our performance? > > > > Thanks, > > Sayantini >
