[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726671#comment-15726671 ]
Thomas Graves commented on SPARK-18733: --------------------------------------- yes looks like a dup but I'm not sure on current solution. max age could easily be set to say 1 week and a job could easily run longer then 1 week. I'll post comment on PR about that though. > Spark history server file cleaner excludes in-progress files > ------------------------------------------------------------ > > Key: SPARK-18733 > URL: https://issues.apache.org/jira/browse/SPARK-18733 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.0.2 > Reporter: Ergin Seyfe > > When we restart history server, it does spend a lot of time to load/replay > incomplete applications which mean the inprogress log files in the log folder. > We have already enabled "spark.history.fs.cleaner.enabled" but seems like > it's skipping the inprogress files. > I checked the log folder and saw that there are many old orphan files. > Probably files left over due to spark-driver failures or OOMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org