ump45nose opened a new pull request, #2643: URL: https://github.com/apache/datafusion-ballista/pull/2643
# Which issue does this PR close? Closes #2596. # Rationale for this change `QueryStageScheduler::on_receive` was one ~490-line function: a `rest-api` event-log pre-pass that matched on the same variants as the main dispatch (repeating the `event_log_graph` / `append_final` / `finish_job` sequence for all three terminal events), followed by a match whose arms inlined everything, including the spawned planning task for `JobQueued` and the nested no-executor grace-period task for `ExecutorLost`. # What changes are included in this PR? A refactor of `ballista/scheduler/src/scheduler_server/query_stage_scheduler.rs` along the lines suggested in the issue: - `on_receive` is now just the event-log call, the timing wrapper and a call to `dispatch`, which matches on the event and calls one method per non-trivial arm: `on_job_queued`, `on_job_submitted`, `on_job_planning_failed`, `on_job_finished`, `on_job_running_failed`, `on_job_updated`, `on_job_cancel`, `on_task_updating` and `on_executor_lost`. The one-liners (`ReviveOffers`, `CancelTasks`, `JobDataClean`) stay inline. - The `rest-api` pre-pass moved into `record_event_log(&event)`, and the three terminal events share `finish_event_log(log, job_id, |graph| ...)`. The existing comments about `finish_job` and `.running` files moved onto that helper. - The grace-period timer moved into `schedule_no_executor_check`, and the loop that fails the still-running jobs into a free `fail_jobs_without_executors` function. The subscriber notification on planning failure is now `notify_planning_failed`. Logic, log messages and error propagation are unchanged. The only small difference: when `queue_job` fails, `on_receive` used to `return Ok(())` before the slow-event `[METRICS]` warning; now that branch returns from `on_job_queued`, so the timing check still runs for it. `cargo test -p ballista-scheduler --lib` passes (474 passed, 2 ignored), and `cargo clippy -p ballista-scheduler --all-targets -- -D warnings` (default features, with `rest-api`) and `cargo clippy -p ballista-scheduler --no-default-features --lib --tests -- -D warnings` are both clean; `cargo fmt --all` applied. # Are there any user-facing changes? No. --- This PR was prepared with AI assistance; I reviewed the change and ran the checks above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
