[ https://issues.apache.org/jira/browse/SPARK-15690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332960#comment-15332960 ]
Joseph Fourny edited comment on SPARK-15690 at 6/16/16 3:00 AM: ---------------------------------------------------------------- I am trying to develop single-node clusters on large servers (30+ CPU cores) with 2-3 TB or RAM. Our use cases involve small to medium size datasets, but with a huge amount of concurrent jobs (shared, multi-tenant environments). Efficiency and sub-second response times are the primary requirements. This shuffle between stages is the current bottleneck. Writing anything to disk is just a waste of time if all computations are done in the same JVM (or a small set of JVMs on the same machine). We tried using RAMFS to avoid disk I/O, but still a lot of CPU time is spent in compression and serialization. I would rather not hack my way out of this one. Is it wishful thinking to have this officially supported? was (Author: josephfourny): +1 on this. I am trying to develop single-node clusters on large servers (30+ CPU cores) with 2-3 TB or RAM. Our use cases involve small to medium size datasets, but with a huge amount of concurrent jobs (shared, multi-tenant environments). Efficiency and sub-second response times are the primary requirements. This shuffle between stages is the current bottleneck. Writing anything to disk is just a waste of time if all computations are done in the same JVM (or a small set of JVMs on the same machine). We tried using RAMFS to avoid disk I/O, but still a lot of CPU time is spent in compression and serialization. I would rather not hack my way out of this one. Is it wishful thinking to have this officially supported? > Fast single-node (single-process) in-memory shuffle > --------------------------------------------------- > > Key: SPARK-15690 > URL: https://issues.apache.org/jira/browse/SPARK-15690 > Project: Spark > Issue Type: New Feature > Components: Shuffle, SQL > Reporter: Reynold Xin > > Spark's current shuffle implementation sorts all intermediate data by their > partition id, and then write the data to disk. This is not a big bottleneck > because the network throughput on commodity clusters tend to be low. However, > an increasing number of Spark users are using the system to process data on a > single-node. When in a single node operating against intermediate data that > fits in memory, the existing shuffle code path can become a big bottleneck. > The goal of this ticket is to change Spark so it can use in-memory radix sort > to do data shuffling on a single node, and still gracefully fallback to disk > if the data size does not fit in memory. Given the number of partitions is > usually small (say less than 256), it'd require only a single pass do to the > radix sort with pretty decent CPU efficiency. > Note that there have been many in-memory shuffle attempts in the past. This > ticket has a smaller scope (single-process), and aims to actually > productionize this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org