ChrisMuki commented on PR #57257: URL: https://github.com/apache/spark/pull/57257#issuecomment-4974070769
Some additional context for reviewers: This adds an opt-in extension point for the scheduler's pool ordering. Today the comparator is hard-wired to `FairSchedulingAlgorithm` / `FIFOSchedulingAlgorithm` based solely on `spark.scheduler.mode`, with no supported way to customize pool ordering (priority-based, deadline-aware, SLA-driven, ...) short of patching Spark. The change follows the existing pluggable-class pattern (`spark.serializer`, `spark.shuffle.manager`): a new optional config `spark.scheduler.rootPool.algorithm.class` names a `SchedulingAlgorithm` implementation that `TaskSchedulerImpl` installs on the root pool. When unset, behavior is unchanged. It is also the only practical customization path on managed platforms (Databricks, EMR, ...) where the user does not create the `SparkContext`, so a runtime setter would not work. Because this exposes a new extension point, `SchedulingAlgorithm` and `Schedulable` are annotated `@DeveloperApi` (with `Schedulable`'s structural/mutating members kept `private[spark]`). I'd like to confirm on the dev list whether `@DeveloperApi` is the right stability level here or whether a SPIP is preferred — I'll start a `[DISCUSS]` thread and link it here. JIRA: https://issues.apache.org/jira/browse/SPARK-58126 cc reviewers familiar with the scheduler — feedback on the config name and the API surface is very welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
