ChrisMuki opened a new pull request, #57257:
URL: https://github.com/apache/spark/pull/57257
### What changes were proposed in this pull request?
This PR adds a configuration property
`spark.scheduler.rootPool.algorithm.class`
that lets users plug in a custom `SchedulingAlgorithm` to order the
scheduler's
root pool.
- `Pool` now takes the `SchedulingAlgorithm` as an optional constructor
parameter and gains a companion factory
`Pool.schedulingAlgorithmFor(mode)`.
All existing call-sites are unchanged (a secondary constructor derives the
algorithm from the scheduling mode as before).
- `TaskSchedulerImpl` resolves the root pool algorithm from configuration
(reflectively via `Utils.classForName`), falling back to the mode-derived
algorithm when the property is unset.
- `SchedulingAlgorithm` and `Schedulable` are annotated `@DeveloperApi`.
`Schedulable` exposes only the read-only scheduling properties publicly;
structural/mutating members are tightened to `private[spark]`.
- `PoolSuite` gains tests for direct injection and config-based override in
both
FAIR and FIFO mode.
### Why are the changes needed?
The pool/task-set ordering is currently hard-wired to
`FairSchedulingAlgorithm`
or `FIFOSchedulingAlgorithm`, selected only by `spark.scheduler.mode`. Users
who
need custom ordering (priority-based, deadline-aware, SLA-driven) have no
supported extension point and must patch Spark. This follows the established
pluggable-class pattern (`spark.serializer`, `spark.shuffle.manager`) and is
the
only practical customization path on managed platforms where the user does
not
create the `SparkContext`.
### Does this PR introduce _any_ user-facing change?
Yes, additive and opt-in:
- New config `spark.scheduler.rootPool.algorithm.class` (defaults to unset,
so
there is no behavior change).
- `SchedulingAlgorithm` and `Schedulable` are now `@DeveloperApi`.
### How was this patch tested?
New and existing unit tests in `core` `PoolSuite`
(`./build/sbt "core/testOnly org.apache.spark.scheduler.PoolSuite"`, 16/16
passing), plus `core/scalastyle` and `core/Test/scalastyle` (0 errors). New
tests cover direct algorithm injection into a `Pool` and config-based
root-pool
override in both FAIR and FIFO mode.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
