[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083250#comment-17083250 ]
Thomas Graves commented on SPARK-31437: --------------------------------------- This is something I wanted to eventually do but I have not started working on this as I wanted to keep it basic for the first implementation to make it easier to review and think about. I agree this feature would be nice, but it adds a fair bit of complexity. I think there are other parts of the feature that aren't yet implemented that are more important. I'm actually surprised someone is using this already, so glad to see that. Feel free to work on this if you would like but I want to make sure whatever we do it is optional and isn't to invasive. once you start doing this, you definitely have the potential to waste a lot of resources when you place things badly. We had discussed some of this in one of the docs or prs, but I can't remember which one. For instance you can place a cpu based task onto an executors with GPUs and waste that expensive GPU resource the entire time. Note that right now everything is tracked by the resource profile id, so if you start putting tasks on on executors with different resource profiles, that tracking needs to be enhanced to handle it. Similarly in the allocation manager, it has to handle it differently as well because right now its very straight forward - which was intentional to make initial implementation and reviewing easier. > Try assigning tasks to existing executors by which required resources in > ResourceProfile are satisfied > ------------------------------------------------------------------------------------------------------ > > Key: SPARK-31437 > URL: https://issues.apache.org/jira/browse/SPARK-31437 > Project: Spark > Issue Type: Improvement > Components: Scheduler > Affects Versions: 3.1.0 > Reporter: Hongze Zhang > Priority: Major > > By the change in [PR|https://github.com/apache/spark/pull/27773] of > SPARK-29154, submitted tasks are scheduled onto executors only if resource > profile IDs strictly match. As a result Spark always starts new executors for > customized ResourceProfiles. > This limitation makes working with process-local jobs unfriendly. E.g. Task > cores has been increased from 1 to 4 in a new stage, and executor has 8 > slots, it is expected that 2 new tasks can be run on the existing executor > but Spark starts new executors for new ResourceProfile. The behavior is > unnecessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org