Thomas Graves created SPARK-6751: ------------------------------------ Summary: Spark History Server support multiple application attempts Key: SPARK-6751 URL: https://issues.apache.org/jira/browse/SPARK-6751 Project: Spark Issue Type: Improvement Components: Spark Core, YARN Affects Versions: 1.3.0 Reporter: Thomas Graves
Spark on Yarn supports running multiple application attempts (configurable number) in case the first (or second..) attempts fail. The Spark History server only supports one history file though. Under the default configs it keeps the first attempts history file. You can set the undocumented config spark.eventLog.overwrite to allow the follow on attempts to overwrite the first attempts history file. Note that in spark 1.2 not having the overwrite config set causes any following attempts to actually fail to run, in spark 1.3 they run and you just see a warning at the end of the attempts. It would be really nice to have an option that keeps all the attempts history files. This way a user can go back and look at each one individually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org