andygrove opened a new pull request, #2265:
URL: https://github.com/apache/datafusion-ballista/pull/2265
# Which issue does this PR close?
Closes #1923. Last of four PRs splitting up #1925, after #2256 (merged),
#2260 and #2264.
> **Stacked on #2264.** GitHub shows the whole chain until the earlier ones
merge, so review the last commit here (`feat(scheduler): add the history
server`).
# Rationale for this change
#2264 makes the scheduler write durable per-job logs, but nothing reads
them. This serves them.
The scheduler forgets a job shortly after it finishes: completed jobs are
cleaned up after `finished_job_state_clean_up_interval_seconds`, and everything
is gone on restart. With this, the TUI can point at a history server and browse
completed jobs with no scheduler running at all.
`user-personas.md` lists "a history server / UI" among the things Persona 2
depends on, so this fills in a guarantee already written down.
# What changes are included in this PR?
**`ballista-history-server --event-log-dir <dir>`** loads every completed
log at startup and serves the same `/api/*` paths as the live scheduler:
`jobs`, `job/{id}`, `job/{id}/stages`, `job/{id}/config`, `job/{id}/dot`.
Corrupt or unreadable logs are logged and skipped rather than failing
startup, so one file truncated by a crash cannot hide every other job.
`/api/executors` returns empty and `/api/state` a static payload, since there
is no cluster behind a history server and the TUI calls both at startup.
**A user-guide page** covering both flags, the endpoints, and the
operational caveats.
## How the payloads are served
The stored `/api/job/{id}` and `/api/job/{id}/stages` payloads are relayed
as raw JSON rather than deserialized and re-serialized. Clients get the exact
bytes the live scheduler produced, and a later change to the REST types cannot
make an existing log unservable, which is the compatibility property #2260 is
built around.
The job list is the one endpoint that needs structure, since it omits the
plan fields. It is rebuilt from the frozen `JobIndex` rather than by editing a
stored payload, so that path never parses a response it would mostly discard.
## The end-to-end guarantee
`history_store_serves_byte_identical_json_to_live_scheduler` was held out of
#2264 because it needs `HistoryStore`. It lands here. It emits a real `JobEnd`
through the real async writer, loads it back through `HistoryStore::load`, and
compares the served bytes against what the live builders produce for the same
graph.
Storing raw JSON made it a stronger test than it was in #1925: it is now a
literal byte comparison rather than a structural one.
# Are there any user-facing changes?
Yes, all additive:
- New `ballista-history-server` binary.
- The TUI can point at it with `--host` / `--port` to browse completed jobs
with no live scheduler.
No API changes, and nothing about a cluster that has not enabled event
logging.
Verified locally: `cargo test -p ballista-scheduler --lib` passes (337
tests), clippy is clean with `--all-features -D warnings`, fmt, taplo and
prettier are clean, and the `--no-default-features` check still passes.
I also ran the binary against the checked-in `schema-v1.eventlog` fixture
and curled every endpoint. It loads the job, serves the list, the job, the
stages (relaying the multi-partition `partition_id` array verbatim) and the
config, and 404s on an unknown job.
# Known limitations
Documented on the user-guide page rather than left to be discovered:
- **Local filesystem only**, and logs are read once at startup. Restart to
pick up newly finished jobs.
- **Disk is not reclaimed automatically.** Logs accumulate until pruned.
- **Plans are rendered once, at write time**, so `?plan_format=` has no
effect against a history server.
- **Only the terminal record is served.** The per-task timeline is captured
so a future UI can show a job progressing, but nothing reads it yet.
# Not yet verified
I have run this against hand-written and fixture logs, not against a log
produced by a live cluster end to end. The emission path is covered by unit
tests in #2264, but the two halves have not been exercised together on a real
cluster. Worth doing before this lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
