[ https://issues.apache.org/jira/browse/SPARK-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207885#comment-14207885 ]
Sean Owen commented on SPARK-4341: ---------------------------------- So I think some of this is already done by Spark. For example, the number of partitions is determined in the same way that Hadoop does, and carries through a pipeline of transformations. Some of this is not necessarily the right thing to do. For example I could be running several transformations at once, and trying to match each's parallelism to the number of executors may be inefficient, not only because it may mean making partitions that are excessively small or large, but because it may require a shuffle, which is expensive. Finally I think the issue of resource usage is better dealt with by increasing/decreasing the number of executors dynamically in response to demand or load, and there is already work in progress on those. So maybe that covers what you are thinking of already. > Spark need to set num-executors automatically > --------------------------------------------- > > Key: SPARK-4341 > URL: https://issues.apache.org/jira/browse/SPARK-4341 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.1.0 > Reporter: Hong Shen > > The mapreduce job can set maptask automaticlly, but in spark, we have to set > num-executors, executor memory and cores. It's difficult for users to set > these args, especially for the users want to use spark sql. So when user > havn't set num-executors, spark should set num-executors automatically > accroding to the input partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org