Hi all, I'd like to propose a small, additive extension point in the scheduler and get feedback before this moves forward.
Problem ------- The ordering of scheduler pools (FAIR mode) / task sets (FIFO mode) is hard-wired to FairSchedulingAlgorithm / FIFOSchedulingAlgorithm, chosen solely from spark.scheduler.mode. There is no supported way to customize pool ordering (e.g. priority-based, deadline-aware, or SLA-driven ordering) without patching Spark. Proposal -------- Add a configuration property: spark.scheduler.rootPool.algorithm.class = <fqcn of a SchedulingAlgorithm> When set, TaskSchedulerImpl instantiates the class (no-arg constructor, via Utils.classForName) and uses it to order the root pool. When unset, behavior is unchanged: the algorithm is derived from spark.scheduler.mode (FAIR/FIFO). This follows the same pluggable-class pattern already used for spark.serializer and spark.shuffle.manager. To make this a supported extension point, SchedulingAlgorithm and Schedulable are annotated @DeveloperApi. Schedulable exposes only its read-only scheduling properties (weight, minShare, runningTasks, priority, stageId, name, schedulingMode) as public API; structural/mutating members remain private[spark]. Why config-based (not a runtime setter) --------------------------------------- A runtime SparkContext.setRootPoolSchedulingAlgorithm(...) was prototyped but rejected: on managed platforms (Databricks, EMR, ...) the user does not create the SparkContext, so a config-based approach is the only one that works there, and it is more idiomatic for Spark. Scope ----- Opt-in, no behavior change when unset. Only the root-pool ordering is configurable in this change; per-pool custom algorithms (via the fair scheduler XML) could be a follow-up. JIRA: https://issues.apache.org/jira/browse/SPARK-58126 PR: https://github.com/apache/spark/pull/57257 The PR includes the implementation and unit tests (core PoolSuite, 16/16 passing; scalastyle clean). Questions for the list: 1. Is @DeveloperApi the right stability level here, or would you prefer this stays fully internal / warrants a SPIP? 2. Config name: spark.scheduler.rootPool.algorithm.class — any preference? 3. Would there be interest in a follow-up for per-pool custom algorithms via the fair scheduler XML? Thanks, Christian Internal
