[ https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563839#comment-15563839 ]
Mike Dusenberry edited comment on SPARK-4630 at 10/10/16 11:10 PM: ------------------------------------------------------------------- It would be really nice to revisit this issue, perhaps even updated to just focus on {{DataSet}} given the current direction of the project. Basically, I would be really interested in smart DataSet/DataFrame/RDD repartitioning that automatically decides the proper number of partitions based on characteristics of the data (i.e. width) and the cluster. From my experience, the outcome of a "wrong" number of partitions is frequent OOM errors, 2GB partition limits (although that's been lifted in 2.x, it's still a perf issue), and if you save a DataFrame/DataSet to, say, Parquet format with too few partitions, the individual compressed files may be too large to read later on (say if a partition is 2GB, but there isn't enough executor memory to open that when working with the files later on with a different Spark setup -- think perhaps a batch preprocessing cluster vs. production serving cluster). was (Author: mwdus...@us.ibm.com): It would be really nice to revisit this issue, perhaps even updated to just focus on {{DataSet}} given the current direction of the project. Basically, I would be really interested in Smart DataSet/DataFrame/RDD repartitioning that automatically decides the proper number of partitions based on characteristics of the data (i.e. width) and the cluster. From my experience, the outcome of a "wrong" number of partitions is frequent OOM errors, 2GB partition limits (although that's been lifted in 2.x, it's still a perf issue), and if you save a DataFrame/DataSet to, say, Parquet format with too few partitions, the individual compressed files may be too large to read later on (say if a partition is 2GB, but there isn't enough executor memory to open that when working with the files later on with a different Spark setup -- think perhaps a batch preprocessing cluster vs. production serving cluster). > Dynamically determine optimal number of partitions > -------------------------------------------------- > > Key: SPARK-4630 > URL: https://issues.apache.org/jira/browse/SPARK-4630 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: Kostas Sakellis > Assignee: Kostas Sakellis > > Partition sizes play a big part in how fast stages execute during a Spark > job. There is a direct relationship between the size of partitions to the > number of tasks - larger partitions, fewer tasks. For better performance, > Spark has a sweet spot for how large partitions should be that get executed > by a task. If partitions are too small, then the user pays a disproportionate > cost in scheduling overhead. If the partitions are too large, then task > execution slows down due to gc pressure and spilling to disk. > To increase performance of jobs, users often hand optimize the number(size) > of partitions that the next stage gets. Factors that come into play are: > Incoming partition sizes from previous stage > number of available executors > available memory per executor (taking into account > spark.shuffle.memoryFraction) > Spark has access to this data and so should be able to automatically do the > partition sizing for the user. This feature can be turned off/on with a > configuration option. > To make this happen, we propose modifying the DAGScheduler to take into > account partition sizes upon stage completion. Before scheduling the next > stage, the scheduler can examine the sizes of the partitions and determine > the appropriate number tasks to create. Since this change requires > non-trivial modifications to the DAGScheduler, a detailed design doc will be > attached before proceeding with the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org