[
https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083250#comment-17083250
]
Thomas Graves commented on SPARK-31437:
---------------------------------------
This is something I wanted to eventually do but I have not started working on
this as I wanted to keep it basic for the first implementation to make it
easier to review and think about. I agree this feature would be nice, but it
adds a fair bit of complexity. I think there are other parts of the feature
that aren't yet implemented that are more important. I'm actually surprised
someone is using this already, so glad to see that.
Feel free to work on this if you would like but I want to make sure whatever we
do it is optional and isn't to invasive. once you start doing this, you
definitely have the potential to waste a lot of resources when you place things
badly. We had discussed some of this in one of the docs or prs, but I can't
remember which one. For instance you can place a cpu based task onto an
executors with GPUs and waste that expensive GPU resource the entire time.
Note that right now everything is tracked by the resource profile id, so if you
start putting tasks on on executors with different resource profiles, that
tracking needs to be enhanced to handle it. Similarly in the allocation
manager, it has to handle it differently as well because right now its very
straight forward - which was intentional to make initial implementation and
reviewing easier.
> Try assigning tasks to existing executors by which required resources in
> ResourceProfile are satisfied
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-31437
> URL: https://issues.apache.org/jira/browse/SPARK-31437
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 3.1.0
> Reporter: Hongze Zhang
> Priority: Major
>
> By the change in [PR|https://github.com/apache/spark/pull/27773] of
> SPARK-29154, submitted tasks are scheduled onto executors only if resource
> profile IDs strictly match. As a result Spark always starts new executors for
> customized ResourceProfiles.
> This limitation makes working with process-local jobs unfriendly. E.g. Task
> cores has been increased from 1 to 4 in a new stage, and executor has 8
> slots, it is expected that 2 new tasks can be run on the existing executor
> but Spark starts new executors for new ResourceProfile. The behavior is
> unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
