Yuqin Luo created SPARK-60756:
---------------------------------
Summary: SQLAppStatusListener.onJobStart forces synchronous
kvstore write during history replay, causing O(J^2) replay cost
Key: SPARK-60756
URL: https://issues.apache.org/jira/browse/SPARK-60756
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.6
Reporter: Yuqin Luo
SQLAppStatusListener.onJobStart unconditionally calls update(exec, force =
true), forcing a synchronous KVStore write on every job-start event, even
during History Server replay (live = false).
During replay, liveUpdatePeriodNs is -1, so this forced write is the only
source of intermediate writes. Each write serializes the execution's growing
job/stage state, so an execution with J job-start events costs O(J^2) to replay.
Proposed fix: change update(exec, force = true) to update(exec, force = live)
in onJobStart. Live UI behavior is unchanged. During replay, the final write on
execution end (onExecutionEnd) is preserved, and executions that never complete
are still persisted by the existing kvstore.onFlush hook.
Measured with local History Server replay of two real production event logs
(patched vs. unpatched):
- 615.7 MB event log: 531s -> 101s (5.3x faster)
- 1010 MB event log: 833.7s -> 284.5s (2.93x faster)
The final replayed data is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
