[jira] [Assigned] (SPARK-7884) Allow Spark shuffle APIs to be more customizable
[ https://issues.apache.org/jira/browse/SPARK-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7884: --- Assignee: Apache Spark Allow Spark shuffle APIs to be more customizable Key: SPARK-7884 URL: https://issues.apache.org/jira/browse/SPARK-7884 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Matt Massie Assignee: Apache Spark The current Spark shuffle has some hard-coded assumptions about how shuffle managers will read and write data. The FileShuffleBlockResolver.forMapTask method creates disk writers by calling BlockManager.getDiskWriter. This forces all shuffle managers to store data using the DiskBlockObjectWriter which read/write data as record-oriented (preventing column-orient record writing). The BlockStoreShuffleFetcher.fetch method relies on the ShuffleBlockFetcherIterator that assumes shuffle data is written using the BlockManager.getDiskWriter method and doesn't allow for customization. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-7884) Allow Spark shuffle APIs to be more customizable
[ https://issues.apache.org/jira/browse/SPARK-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7884: --- Assignee: (was: Apache Spark) Allow Spark shuffle APIs to be more customizable Key: SPARK-7884 URL: https://issues.apache.org/jira/browse/SPARK-7884 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Matt Massie The current Spark shuffle has some hard-coded assumptions about how shuffle managers will read and write data. The FileShuffleBlockResolver.forMapTask method creates disk writers by calling BlockManager.getDiskWriter. This forces all shuffle managers to store data using the DiskBlockObjectWriter which read/write data as record-oriented (preventing column-orient record writing). The BlockStoreShuffleFetcher.fetch method relies on the ShuffleBlockFetcherIterator that assumes shuffle data is written using the BlockManager.getDiskWriter method and doesn't allow for customization. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
