[ https://issues.apache.org/jira/browse/SPARK-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061536#comment-14061536 ]
Masayoshi TSUZUKI commented on SPARK-2458: ------------------------------------------ History server uses FsHistoryProvider as ApplicationHistoryProvider by default. {code:title=HistoryServer.scala|borderStyle=solid} val providerName = conf.getOption("spark.history.provider") .getOrElse(classOf[FsHistoryProvider].getName()) val provider = Class.forName(providerName) .getConstructor(classOf[SparkConf]) .newInstance(conf) .asInstanceOf[ApplicationHistoryProvider] {code} While FsHistoryServer continuously check for new log directories, it filters out the directory which doesn't contain APPLICATION_COMPLETE file. {code:title=FsHistoryProvider.scala|borderStyle=solid} val logInfos = logDirs.filter { dir => fs.isFile(new Path(dir.getPath(), EventLoggingListener.APPLICATION_COMPLETE)) } {code} > Make failed application log visible on History Server > ----------------------------------------------------- > > Key: SPARK-2458 > URL: https://issues.apache.org/jira/browse/SPARK-2458 > Project: Spark > Issue Type: Improvement > Components: Web UI > Affects Versions: 1.0.0 > Reporter: Masayoshi TSUZUKI > > History server is very helpful for debugging application correctness & > performance after the application finished. However, when the application > failed, the link is not listed on the hisotry server UI and history can't be > viewed. > It would be very useful if we can check the history of failed application. -- This message was sent by Atlassian JIRA (v6.2#6252)