[ https://issues.apache.org/jira/browse/SPARK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
SaintBacchus updated SPARK-6464: -------------------------------- Description: Nowadays, the transformation *coalesce* was always used to expand or reduce the number of the partition in order to gain a good performance. But *coalesce* can't make sure that the child partition will be executed in the same executor as the parent partition. And this will lead to have a large network transfer. In some scenario such as I metioned in the title > Add a new transformation of rdd named processCoalesce which was particularly > to deal with the small and cached rdd > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-6464 > URL: https://issues.apache.org/jira/browse/SPARK-6464 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.3.0 > Reporter: SaintBacchus > > Nowadays, the transformation *coalesce* was always used to expand or reduce > the number of the partition in order to gain a good performance. > But *coalesce* can't make sure that the child partition will be executed in > the same executor as the parent partition. And this will lead to have a large > network transfer. > In some scenario such as I metioned in the title -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org