luoyuqinlaura opened a new pull request, #59964: URL: https://github.com/apache/spark/pull/59964
### What changes were proposed in this pull request? In `SQLAppStatusListener.onJobStart`, change `update(exec, force = true)` to `update(exec, force = live)`. ### Why are the changes needed? `onJobStart` unconditionally forced a synchronous KVStore write on every job-start event, even during History Server replay (`live = false`). During replay `liveUpdatePeriodNs` is -1, so this forced write is the only source of intermediate writes. Each write serializes the execution's growing job/stage state, so an execution with J job starts costs O(J^2) to replay. Local History Server replay of two real production event logs: - 615.7 MB: 531s -> 101s (5.3x) - 1010 MB: 833.7s -> 284.5s (2.93x) N (Job Starts) | live = true (Before) | live = false (After) | Speedup -- | -- | -- | -- 1,000 | 185 ms | 1 ms | ~185× 2,000 | 370 ms | 1 ms | ~370× 5,000 | 2,226 ms | 1 ms | ~2,226× 10,000 | 9,370 ms | 2 ms | ~4,685× ### Does this PR introduce _any_ user-facing change? No. Live UI behavior is unchanged. During replay the final write on execution end is preserved, and never-completed executions are still persisted by the `kvstore.onFlush` hook. ### How was this patch tested? Added a unit test in `SQLAppStatusListenerSuite` that fails without the fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
