andygrove opened a new pull request, #2260: URL: https://github.com/apache/datafusion-ballista/pull/2260
# Which issue does this PR close? Part of #1923. Second of four PRs splitting up #1925. > **Stacked on #2256.** GitHub shows both branches until that one merges, so review the last commit here (`feat(history): add the event-log schema, writer, and reader`). It needs the `ballista-api-types` crate #2256 introduces. # Rationale for this change #1923 asks for a Spark-History-Server equivalent: the scheduler records a durable log of each job, and a standalone server replays those logs and serves the same `/api/*` responses so the existing TUI can browse completed jobs with no scheduler running. This PR adds the format and the machinery to read and write it, and nothing else. Nothing in the scheduler calls it yet, so it can be reviewed purely on its own terms: is the on-disk schema right, and does the writer behave under load? Those are the two questions worth isolating. The schema is durable and cross-machine, so it is the part that is expensive to change later. And the writer sits next to the scheduler's hot path, so its backpressure behaviour matters more than its line count suggests. # What changes are included in this PR? **New `ballista-history` crate**, depending on `ballista-api-types` plus serde, tokio and log. `event.rs` — the versioned JSONL schema. `JobStart`, `StageStart`, `StageEnd` and `TaskEnd` form an incremental timeline. The terminal `JobEnd` embeds the finished `JobResponse` and `QueryStagesResponse`. That last choice is the important one. Storing the built responses means replay re-serves exactly what the scheduler produced, rather than re-deriving it from stored state. Byte-identity then falls out of there being one definition and one place that populates it, instead of two implementations that have to keep agreeing. The consequence, worth knowing up front, is that anything not captured at write time cannot be recovered later. Identifiers are fixed-width `u32` rather than `usize`, since a log written by a 64-bit scheduler has to mean the same thing to any reader. `writer.rs` — an async buffered `EventLogWriter`. All file I/O happens on a background task, so the scheduler's event loop never waits on disk. One append-mode handle per job is held for the life of the process. The backpressure split is deliberate. `append` never blocks and drops the event if the queue is full, because losing a progress record is better than stalling scheduling. `append_final` instead waits for capacity, because a job whose `JobEnd` was dropped is invisible to the history server entirely. `finish_job` then flushes and closes the handle, ordered after the terminal event by the single-consumer channel. `reader.rs` — folds a completed log back into the payload a server would serve. A file is "completed" once it contains a `JobEnd`. Malformed lines are skipped rather than treated as fatal, so a log truncated by a crash still yields its job if the terminal record survived. **`JobConfig` returns to `ballista-api-types`.** I removed it in #2256 as dead; the `JobEnd` record is its real consumer. **`TaskEnd` names a task, not a partition.** Under the multi-partition task model a task owns a slice of partitions, and `TaskStatus` carries `task_id` rather than `partition_id`. #1925 predates that change and recorded a `partition` field that no longer has a source. **Release tooling.** `ballista-history` is registered in `dev/update_ballista_versions.py`, the publish order, and the crate dependency graph. # Are there any user-facing changes? No. The crate is new and nothing depends on it yet. No scheduler code changes, no configuration, no API changes. Verified locally: `cargo test -p ballista-history` passes (5 tests), `cargo check -p ballista-scheduler` is unaffected, clippy is clean for both crates with `--all-features -D warnings`, and fmt/taplo are clean. # Follow-ups 3. Scheduler event-log wiring behind a new `--event-log-dir` flag, off by default. 4. The history server binary, its docs, and the byte-identical-JSON end-to-end test. Both are already written and forward-ported; they are waiting on this landing rather than on being figured out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
