Marcelo Vanzin created SPARK-20664:
--------------------------------------
Summary: Remove stale applications from SHS listing
Key: SPARK-20664
URL: https://issues.apache.org/jira/browse/SPARK-20664
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Marcelo Vanzin
See spec in parent issue (SPARK-18085) for more details.
This task is actually not explicit in the spec, and it's also an issue with the
current SHS. But having the SHS persist listing data makes it worse.
Basically, the SHS currently does not detect when files are deleted from the
event log directory manually; so those applications are still listed, and
trying to see their UI will either show the UI (if it's loaded) or an error (if
it's not).
With the new SHS, that also means that data is leaked in the disk stores used
to persist listing and UI data, making the problem worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
