[ https://issues.apache.org/jira/browse/SPARK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
SaintBacchus updated SPARK-6464: -------------------------------- Attachment: screenshot-1.png > Add a new transformation of rdd named processCoalesce which was particularly > to deal with the small and cached rdd > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-6464 > URL: https://issues.apache.org/jira/browse/SPARK-6464 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.3.0 > Reporter: SaintBacchus > Attachments: screenshot-1.png > > > Nowadays, the transformation *coalesce* was always used to expand or reduce > the number of the partition in order to gain a good performance. > But *coalesce* can't make sure that the child partition will be executed in > the same executor as the parent partition. And this will lead to have a large > network transfer. > In some scenario such as I mentioned in the title +small and cached rdd+, we > want to coalesce all the partition in the same executor into one partition > and make sure the child partition will be executed in this executor. It can > avoid network transfer and reduce the scheduler of the Tasks and also can > reused the cpu core to do other job. > In this scenario, our performance had improved 20% than before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org