[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-10-09 Thread Michael Park (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197402#comment-16197402 ] Michael Park commented on SPARK-20589: -- Pardon my ignorance of the inner workings of task

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-09-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161561#comment-16161561 ] Apache Spark commented on SPARK-20589: -- User 'dhruve' has created a pull request for this issue:

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-09-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157424#comment-16157424 ] Apache Spark commented on SPARK-20589: -- User 'dhruve' has created a pull request for this issue:

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144025#comment-16144025 ] Apache Spark commented on SPARK-20589: -- User 'dhruve' has created a pull request for this issue:

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129380#comment-16129380 ] Imran Rashid commented on SPARK-20589: -- Its pretty much the same thing whether you're trying to

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-16 Thread Amit Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129289#comment-16129289 ] Amit Kumar commented on SPARK-20589: As you said, adding job boundary via code will be much easier

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129198#comment-16129198 ] Imran Rashid commented on SPARK-20589: -- the proposed solution will only let you control concurrency

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127942#comment-16127942 ] Thomas Graves commented on SPARK-20589: --- Note that this type of option is also already supported in

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-15 Thread Amit Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127930#comment-16127930 ] Amit Kumar commented on SPARK-20589: [~imranr] Like you said, we could restrict the number of

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127911#comment-16127911 ] Imran Rashid commented on SPARK-20589: -- couldn't you just limit the number of executors of your

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-04 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114628#comment-16114628 ] Dhruve Ashar commented on SPARK-20589: -- I have a patch for this and would like to have some

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-08-04 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114626#comment-16114626 ] Dhruve Ashar commented on SPARK-20589: -- Spark defines stages based on the shuffle dependencies and

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-06-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049228#comment-16049228 ] Thomas Graves commented on SPARK-20589: --- [~Robin Shao] can you please clarify your comment? What

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-06-08 Thread Fei Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043877#comment-16043877 ] Fei Shao commented on SPARK-20589: -- Tasks are assigned to executors. If we set the number of executors

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-24 Thread Mark Nelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023769#comment-16023769 ] Mark Nelson commented on SPARK-20589: - I would find this very useful. We're currently using coalesce

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-15 Thread Amit Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011024#comment-16011024 ] Amit Kumar commented on SPARK-20589: I can probably give more reference. This originally arose from

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995650#comment-15995650 ] Thomas Graves commented on SPARK-20589: --- thanks for the suggestions [~mridulm80]. Note just for

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995550#comment-15995550 ] Mridul Muralidharan commented on SPARK-20589: - coalasce with shuffle=false might be a