Thomas Graves created SPARK-29149:
-------------------------------------
Summary: Update YARN cluster manager For Stage Level Scheduling
Key: SPARK-29149
URL: https://issues.apache.org/jira/browse/SPARK-29149
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.0.0
Reporter: Thomas Graves
For Stage Level Scheduling, we need to update the YARN allocator to handle
requesting executors for multiple ResourceProfiles.
* The container requests have to be updated to be based on a number of
containers per ResourceProfile. This is a larger change then you might expect
because on YARN you can’t ask for different container sizes within the same
YARN container priority. So we will have to ask for containers at different
priorities to be able to get different container sizes. Other YARN applications
like Tez handle this now so it shouldn’t be a big deal just a matter of mapping
stages to different priorities.
* The allocation response from YARN has to match the containers to a resource
profile.
* We need to launch the container with additional parameters so the executor
knows its resource profile to report back to the ExecutorMonitor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
