[
https://issues.apache.org/jira/browse/SPARK-58126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Neuenroth updated SPARK-58126:
----------------------------------------
Description:
Currently the algorithm that orders the scheduler pools (FAIR mode) or task sets
(FIFO mode) is hard-wired: Pool selects either FairSchedulingAlgorithm or
FIFOSchedulingAlgorithm based solely on spark.scheduler.mode. There is no
supported way to customize the ordering — e.g. to implement priority-based
pool ordering, deadline-aware ordering, or business-specific SLAs — without
patching Spark.
The top-level pool ordering is governed by the comparator installed on the root
pool, and the root pool is created by TaskSchedulerImpl, where the SparkConf is
available. Rather than exposing Spark's internal, mutable Schedulable hierarchy
as an extension point, the ordering is customized through a standard
java.util.Comparator over an immutable snapshot of a schedulable's
ordering-relevant properties.
Proposal
--------
Introduce a configuration property:
spark.scheduler.rootPool.comparator.class
When set to the fully qualified name of a class implementing
java.util.Comparator<org.apache.spark.scheduler.SchedulableInfo> (with a no-arg
constructor), TaskSchedulerImpl instantiates it (reflectively, via
Utils.classForName) and wraps it to order the root pool; each Schedulable is
captured as an immutable SchedulableInfo snapshot before being compared. When
unset, behavior is unchanged: the algorithm is derived from spark.scheduler.mode
(FAIR/FIFO).
The only type exposed to user code is a new @DeveloperApi value type,
SchedulableInfo: an immutable snapshot of a schedulable's ordering-relevant
properties (schedulingMode, weight, minShare, runningTasks, priority, stageId,
name). Spark's internal Schedulable and SchedulingAlgorithm stay private[spark],
so the live, mutable scheduler internals remain hidden and free to evolve.
This mirrors the existing pluggable-class pattern used for spark.serializer and
spark.shuffle.manager. It is also the only practical customization path on
managed platforms (Databricks, EMR) where the user does not create the
SparkContext and therefore cannot call a runtime setter.
Scope
Additive, opt-in, no behavior change when the property is unset.
Only the root pool ordering is configurable in this change. Per-pool custom
comparators (via the fair scheduler XML) could be a follow-up.
An alternative runtime API (a setter on SparkContext) was prototyped but
rejected in favor of the config-based approach, which is more idiomatic and
works on managed platforms.
was:
Currently the algorithm that orders the scheduler pools (FAIR mode) or task sets
(FIFO mode) is hard-wired: Pool selects either FairSchedulingAlgorithm or
FIFOSchedulingAlgorithm based solely on spark.scheduler.mode. There is no
supported way to customize the ordering — e.g. to implement priority-based
pool ordering, deadline-aware ordering, or business-specific SLAs — without
patching Spark.
SchedulingAlgorithm is a small, well-defined trait
(comparator(s1: Schedulable, s2: Schedulable): Boolean), which makes it a
natural extension point. The top-level pool ordering is governed by the
comparator installed on the root pool, and the root pool is created by
TaskSchedulerImpl, where the SparkConf is available.
Proposal
--------
Introduce a configuration property:
spark.scheduler.rootPool.algorithm.class
When set to the fully qualified name of a class implementing
org.apache.spark.scheduler.SchedulingAlgorithm (with a no-arg constructor),
TaskSchedulerImpl instantiates it (reflectively, via Utils.classForName) and
uses it to order the root pool. When unset, behavior is unchanged: the
algorithm is derived from spark.scheduler.mode (FAIR/FIFO).
To make this a supported extension point, SchedulingAlgorithm and Schedulable
are annotated @DeveloperApi. Schedulable exposes only the read-only scheduling
properties (weight, minShare, runningTasks, priority, stageId, name,
schedulingMode) as public API; structural/mutating members remain
private[spark].
This mirrors the existing pluggable-class pattern used for spark.serializer and
spark.shuffle.manager. It is also the only practical customization path on
managed platforms (Databricks, EMR) where the user does not create the
SparkContext and therefore cannot call a runtime setter.
Scope
-----
- Additive, opt-in, no behavior change when the property is unset.
- Only the root pool ordering is configurable in this change. Per-pool custom
algorithms (via the fair scheduler XML) could be a follow-up.
An alternative runtime API (SparkContext.setRootPoolSchedulingAlgorithm) was
prototyped but rejected in favor of the config-based approach, which is more
idiomatic and works on managed platforms.
> Support a pluggable root pool comparator via configuration
> ----------------------------------------------------------
>
> Key: SPARK-58126
> URL: https://issues.apache.org/jira/browse/SPARK-58126
> Project: Spark
> Issue Type: New Feature
> Components: Scheduler
> Affects Versions: 5.0.0
> Reporter: Christian Neuenroth
> Priority: Major
> Labels: pull-request-available
>
> Currently the algorithm that orders the scheduler pools (FAIR mode) or task
> sets
> (FIFO mode) is hard-wired: Pool selects either FairSchedulingAlgorithm or
> FIFOSchedulingAlgorithm based solely on spark.scheduler.mode. There is no
> supported way to customize the ordering — e.g. to implement priority-based
> pool ordering, deadline-aware ordering, or business-specific SLAs — without
> patching Spark.
> The top-level pool ordering is governed by the comparator installed on the
> root
> pool, and the root pool is created by TaskSchedulerImpl, where the SparkConf
> is
> available. Rather than exposing Spark's internal, mutable Schedulable
> hierarchy
> as an extension point, the ordering is customized through a standard
> java.util.Comparator over an immutable snapshot of a schedulable's
> ordering-relevant properties.
> Proposal
> --------
> Introduce a configuration property:
> spark.scheduler.rootPool.comparator.class
> When set to the fully qualified name of a class implementing
> java.util.Comparator<org.apache.spark.scheduler.SchedulableInfo> (with a
> no-arg
> constructor), TaskSchedulerImpl instantiates it (reflectively, via
> Utils.classForName) and wraps it to order the root pool; each Schedulable is
> captured as an immutable SchedulableInfo snapshot before being compared. When
> unset, behavior is unchanged: the algorithm is derived from
> spark.scheduler.mode
> (FAIR/FIFO).
> The only type exposed to user code is a new @DeveloperApi value type,
> SchedulableInfo: an immutable snapshot of a schedulable's ordering-relevant
> properties (schedulingMode, weight, minShare, runningTasks, priority, stageId,
> name). Spark's internal Schedulable and SchedulingAlgorithm stay
> private[spark],
> so the live, mutable scheduler internals remain hidden and free to evolve.
> This mirrors the existing pluggable-class pattern used for spark.serializer
> and
> spark.shuffle.manager. It is also the only practical customization path on
> managed platforms (Databricks, EMR) where the user does not create the
> SparkContext and therefore cannot call a runtime setter.
> Scope
> Additive, opt-in, no behavior change when the property is unset.
> Only the root pool ordering is configurable in this change. Per-pool custom
> comparators (via the fair scheduler XML) could be a follow-up.
> An alternative runtime API (a setter on SparkContext) was prototyped but
> rejected in favor of the config-based approach, which is more idiomatic and
> works on managed platforms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
