[ https://issues.apache.org/jira/browse/SPARK-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-4716. --------------------------------- Resolution: Incomplete > Avoid shuffle when all-to-all operation has single input and output partition > ----------------------------------------------------------------------------- > > Key: SPARK-4716 > URL: https://issues.apache.org/jira/browse/SPARK-4716 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Sandy Ryza > Priority: Major > Labels: bulk-closed > > I encountered an application that performs joins on a bunch of small RDDs, > unions the results, and then performs larger aggregations across them. Many > of these small RDDs fit in a single partition. For these operations with > only a single partition, there's no reason to write data to disk and then > fetch it over a socket. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org