cxzl25 opened a new pull request, #55578:
URL: https://github.com/apache/spark/pull/55578
### What changes were proposed in this pull request?
When `mergeApplicationListing()` successfully parses a completed event log,
it now proactively
deletes any existing disk store for the app if the app's UI is not currently
tracked in
`activeUIs`. Concretely, the following lines are added after the
`invalidateUI()` call in
`doMergeApplicationListingInternal`:
```scala
if (app.attempts.head.info.completed) {
val hasActiveUI = synchronized { activeUIs.contains((appId, attemptId)) }
if (!hasActiveUI) {
diskManager.foreach(_.release(appId, attemptId, delete = true))
}
}
```
### Why are the changes needed?
There is a race condition between `ApplicationCache`'s LRU eviction and
`FsHistoryProvider.invalidateUI()` that causes the History Server to serve
stale UI data
after an in-progress app completes.
### Does this PR introduce _any_ user-facing change?
Yes. After this fix, users who access the History Server UI for an
application that completed after its UI was evicted from the `ApplicationCache`
will see the fully-completed application data (all jobs, stages, and the final
application end event), instead of a stale snapshot from when the UI was last
loaded while the app was still in progress.
### How was this patch tested?
Add test to FsHistoryProviderSuite
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
