[ https://issues.apache.org/jira/browse/SPARK-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin updated SPARK-8617: ---------------------------------- Target Version/s: (was: 1.5.0) > Handle history files better > --------------------------- > > Key: SPARK-8617 > URL: https://issues.apache.org/jira/browse/SPARK-8617 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.4.0 > Reporter: Thomas Graves > > Right now there are many conditions in which history files get left around > with the .inprogress extension. The cleaner doesn't remove these because it > can't distinguish between something running and left over abandoned files. > There are cases its really hard to not leave these around (application > killed/crashes/etc). But we should still try to handle the history files > better. > > For instance we could first write them to the .sparkStaging directory and > then moving to the history file location before the application finishes. > That way you only get completed history files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org