[
https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783550#comment-16783550
]
Xingbo Jiang commented on SPARK-27005:
--------------------------------------
I updated the above document, so the Spark internal shall always consider
accelerator in general, and GPU is one instance of generic accelerators.
Since TaskSetManager manages the pending tasks for each stage attempt, we shall
update it to provide a pending task that have GPU requirements when possible.
The weight between accelerators vs locality is subtle, in general we want tasks
that require accelerators get scheduled earlier, so I'd propose that for each
locality level, we create a queue to store pending tasks that require
accelerators and the corresponding locality. Inside the `dequeueTask()`
function, we shall first search all the task queues with accelerator
requirements, if that doesn't generate a candidate, we continue to search for
task with only locality preferences.
> Design sketch: Accelerator-aware scheduling
> -------------------------------------------
>
> Key: SPARK-27005
> URL: https://issues.apache.org/jira/browse/SPARK-27005
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Xingbo Jiang
> Priority: Major
>
> This task is to outline a design sketch for the accelerator-aware scheduling
> SPIP discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
