ForVic opened a new pull request, #59696: URL: https://github.com/apache/spark/pull/59696
### What changes were proposed in this pull request? This is a continuation of #54878, which was closed by the stale bot. It is rebased on the current master and addresses the review comments from that PR. When a Spark application completes, it writes out its AppStatus — the materialized state built by the `AppStatusListener` — as protobuf files to a shared path. When the History Server loads that application, it restores the UI from that snapshot instead of replaying the event log through a `ReplayListenerBus` and `AppStatusListener`. If there is no usable snapshot, the History Server replays the event log as before. - `spark.history.snapshot.enabled` / `spark.history.snapshot.path` must be set both for applications and for the History Server. Applications only write snapshots when `spark.eventLog.enabled` is true, since the History Server discovers applications through their event logs. - A snapshot is a data directory of length-delimited protobuf records (one file per KVStore entity type, including SQL, streaming, Thrift server and Connect UI data) plus a versioned manifest. The manifest is written under a temporary name and renamed into place, so readers only ever see a complete snapshot. - Snapshots are written with the same file permissions as event logs, and the application environment is redacted the same way it is in the event log. - Corrupt snapshots are deleted by the History Server and the UI falls back to replay. Snapshots that fail for other reasons (transient storage errors, or a snapshot written with a different schema/store version) are kept. - When the History Server cleaner is enabled, snapshots are deleted together with their event logs. Snapshots of unknown applications older than `spark.history.fs.cleaner.maxAge` are also deleted. Changes since #54878, addressing the review comments: - Log invalid snapshot imports as warnings, and log the duration when a snapshot write fails. - Validate record sizes against the remaining bytes in an entity file before allocating, so a corrupt size prefix fails as an invalid snapshot instead of an `OutOfMemoryError`. - Add `spark.history.snapshot.numWriteThreads` (default 1) to write entity files in parallel. - Turn `HistorySnapshotStore` into a class that reuses a single Hadoop `Configuration` and `FileSystem`. - Add snapshot retention tied to the event log cleaner, plus cleanup of orphaned snapshots. Other changes found while preparing this PR: - Redact the application environment and use event log file permissions for snapshots. - Reject manifests that reference paths outside of their snapshot directory. - Record the AppStatus store version in the manifest, and only delete snapshots that are actually corrupt. - Publish manifests atomically via rename, and clean up data directories left behind by failed writes. - Include Thrift server and Connect UI data in snapshots, and flush live SQL executions before the snapshot is written. - Remove the unconditional flush at application end, which changed behavior even when snapshots were disabled. - Remove the snapshot write metrics, since the driver metrics system is stopped before the status store is closed and they would never be reported. - Use structured logging, the `4.4.0` config version, and document the configs in `monitoring.md`. ### Why are the changes needed? The Spark History Server can be slow to load applications with large event logs, due to the deserialization overhead of replaying them. Replay repeats the exact work the driver already did while serving the live UI from the `AppStatusListener`'s state in the KVStore. ### Does this PR introduce _any_ user-facing change? Yes. It adds three configs, `spark.history.snapshot.enabled`, `spark.history.snapshot.path` and `spark.history.snapshot.numWriteThreads`, which are disabled by default. When enabled, the History Server UI loads faster for completed applications. We saw p90 History Server load times improve by about 10x and p99 by about 30x, with larger gains for larger event logs. ### How was this patch tested? New unit tests in `FsHistoryProviderSuite`, `AppStatusListenerSuite`, `HistoryServerDiskManagerSuite` and `SQLAppStatusListenerSuite`. They cover snapshot round trips, corrupt and incompatible snapshots, path validation, redaction, parallel writes and cleaner retention. An earlier version of this change also ran internally at scale across multiple History Server instances for several days. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: GPT-5.4, Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
